Power Query - Loop through unknown number of Tables and append them all - solved

Anonymous
2018-07-17T09:29:18+00:00

Hello,

I have an Excel spreadsheet with a lot of columns but I want to save it in csv format with line breaks to put certain columns on different lines.   This is because I can upload different record types into my database but each record type needs its own line in csv with a header type as the first field on that line.

eg this in Excel:

col1   col2   col3   col4   col5   col6   col7   col8   col9   col10

01        b        c        d       e        02         g       h        03         j

01        r         s         t       u        02         v       w        03         x

becomes this in csv:

01, b, c, d, e

02, g, h

03, j

01, r, s, t, u

02, v, w

03, x              

Is this possible?  Easy?  

thanks

Louise

<The thread has been moved to the correct category by forum moderator>

Microsoft 365 and Office | Excel | For home | Windows

Locked Question. This question was migrated from the Microsoft Support Community. You can vote on whether it's helpful, but you can't add comments or replies or follow the question.

0 comments No comments
Answer accepted by question author
Lz365 38,201 Reputation points Volunteer Moderator
2018-08-01T12:23:56+00:00

This is a very simplified version of this thread. We have an Input Table that looks like:

where each Group of colored columns needs to be transformed in a separate Table (so 3 in this case) and at the end we want the 3 Tables to be Appended (Combined according to Power Query M language)

Challenge

The number of Groups may vary (i.e. from 1 to say 10) from time to time. So we need a query that is flexible enough to handle this variation

Note

In the above example the 1st Group has 3 columns, the 2nd has 4 and the last has 5. To properly Append/Combine Tables they all must have the same number of columns. That aspect of the problem is not detailed in the below Power Query M snipet.

How To

Power Quey M language doesn't offer an easy way to loop. This example uses the List.Generate function to do that the appropriate number of times to build each Table and then Append/Combine them

let

    ...

    GroupsList = Record.ToList(Table.FirstN(Source, 1){0}),

    NbGroups = List.Count(List.Distinct(GroupsList)),

    // Loop (actually NbGroups of times) to create a Table for each "Group"

    // At the end we get a List that contains Table(s)

    The next Step will Append/Combine all Table(s)

    TablesToAppend = List.Generate(

         ()=> [i = -1, j = 0, OutputTable = #table({"Col","Index","Index2"},{})],

            each [i] < NbGroups,

                each

                    [

                        j = [j] +1,

                        ...,              // step1 to build the OutputTable

                        ...,              // step2 to build the OutputTable

                        TableIdx = ...,   // step3 to build the OutputTable

                        OutputTable = Table.AddIndexColumn(TableIdx, "Index2", j, 1),

                        i = [i] +1

                    ],

                each [OutputTable]

                                  ),

    // Append all tables from above List

    TablesAppended = Table.Combine(List.Transform(TablesToAppend, each (_))),

    ...

in

    ...

Was this answer helpful?

2 people found this answer helpful.
0 comments No comments

59 additional answers

Sort by: Newest
  1. Lz365 38,201 Reputation points Volunteer Moderator
    2018-08-01T11:56:14+00:00

    Hi again Louise

    My first real success with no help and List.Generate on something quite complex compared to what I did until now with that challenging function.

    I am about to post a very simplified version (w/o any of your data of course) of the implemented solution.

    So this can help someone else sooner or later I will appreciate you do 2 things after I post that summary:

    1. Change the title of that post with something like: Power Query - Loop through unknown number of Tables and append them all
    2. Mark as Answer the summary I'm about to post

    Thanks very much in advance

    Was this answer helpful?

    0 comments No comments
  2. Anonymous
    2018-08-01T09:21:50+00:00

    Oh WOW!  Totally different and no need to update the code.  Well done!!  I can crack on with the real data now.  Thanks again.  :0)

    Was this answer helpful?

    0 comments No comments
  3. Lz365 38,201 Reputation points Volunteer Moderator
    2018-08-01T09:03:39+00:00

    (went to Biarritz a long time ago. Liked it)

    Here you go. Sounds good to me, please confirm. Query documented, any question let me know

    Was this answer helpful?

    0 comments No comments
  4. Anonymous
    2018-08-01T08:36:38+00:00

    Morning,

    Holiday sounds great, Biarritz is nice if you are anywhere near there, and dynamic looping sounds intriguing.  ;0)

    Obviously I'm not sharing 'real' data with you but I did have a go at adding a couple of additional records types, even one in the middle of what I'd already provided, so here is a set with 6 record types.  Hope that helps.

    https://www.dropbox.com/s/u0kwb5xj84zt74z/Copy%20of%20QUERY\_TO\_PREP\_CSV%20-%20V101.xlsm?dl=0

    Was this answer helpful?

    0 comments No comments