Power Query - Loop through unknown number of Tables and append them all - solved

Anonymous
2018-07-17T09:29:18+00:00

Hello,

I have an Excel spreadsheet with a lot of columns but I want to save it in csv format with line breaks to put certain columns on different lines.   This is because I can upload different record types into my database but each record type needs its own line in csv with a header type as the first field on that line.

eg this in Excel:

col1   col2   col3   col4   col5   col6   col7   col8   col9   col10

01        b        c        d       e        02         g       h        03         j

01        r         s         t       u        02         v       w        03         x

becomes this in csv:

01, b, c, d, e

02, g, h

03, j

01, r, s, t, u

02, v, w

03, x              

Is this possible?  Easy?  

thanks

Louise

<The thread has been moved to the correct category by forum moderator>

Microsoft 365 and Office | Excel | For home | Windows

Locked Question. This question was migrated from the Microsoft Support Community. You can vote on whether it's helpful, but you can't add comments or replies or follow the question.

0 comments No comments
Answer accepted by question author
Lz365 38,201 Reputation points Volunteer Moderator
2018-08-01T12:23:56+00:00

This is a very simplified version of this thread. We have an Input Table that looks like:

where each Group of colored columns needs to be transformed in a separate Table (so 3 in this case) and at the end we want the 3 Tables to be Appended (Combined according to Power Query M language)

Challenge

The number of Groups may vary (i.e. from 1 to say 10) from time to time. So we need a query that is flexible enough to handle this variation

Note

In the above example the 1st Group has 3 columns, the 2nd has 4 and the last has 5. To properly Append/Combine Tables they all must have the same number of columns. That aspect of the problem is not detailed in the below Power Query M snipet.

How To

Power Quey M language doesn't offer an easy way to loop. This example uses the List.Generate function to do that the appropriate number of times to build each Table and then Append/Combine them

let

    ...

    GroupsList = Record.ToList(Table.FirstN(Source, 1){0}),

    NbGroups = List.Count(List.Distinct(GroupsList)),

    // Loop (actually NbGroups of times) to create a Table for each "Group"

    // At the end we get a List that contains Table(s)

    The next Step will Append/Combine all Table(s)

    TablesToAppend = List.Generate(

         ()=> [i = -1, j = 0, OutputTable = #table({"Col","Index","Index2"},{})],

            each [i] < NbGroups,

                each

                    [

                        j = [j] +1,

                        ...,              // step1 to build the OutputTable

                        ...,              // step2 to build the OutputTable

                        TableIdx = ...,   // step3 to build the OutputTable

                        OutputTable = Table.AddIndexColumn(TableIdx, "Index2", j, 1),

                        i = [i] +1

                    ],

                each [OutputTable]

                                  ),

    // Append all tables from above List

    TablesAppended = Table.Combine(List.Transform(TablesToAppend, each (_))),

    ...

in

    ...

Was this answer helpful?

2 people found this answer helpful.
0 comments No comments

59 additional answers

Sort by: Newest
  1. Lz365 38,201 Reputation points Volunteer Moderator
    2018-07-17T17:42:46+00:00

    Louise,

    Made good progress but there's a problem with your CSV. If I split it in columns using Text to Columns I get the following:

    Last Name is in column M instead of G. My query is based on your sample and logically produces the same output. So ???

    Was this answer helpful?

    0 comments No comments
  2. Anonymous
    2018-07-17T14:21:37+00:00

    Yes.  The two files contain the same data.   The Excel version is how I'm working on preparing the data as it's easier to use the functionality of Excel like sorting and filtering.  The csv version is how it needs to be laid out for the upload to work.  

    I did try to upload it as one huge long string of one line per person (which of course is easy, just save as csv) but it didn't like that!

    Was this answer helpful?

    0 comments No comments
  3. Lz365 38,201 Reputation points Volunteer Moderator
    2018-07-17T14:14:56+00:00

    Not sure I understand everything yet. Just to be sure: is the CSV you uploaded representative of the expected output?

    Was this answer helpful?

    0 comments No comments
  4. Anonymous
    2018-07-17T13:54:48+00:00

    Hello,

    I have a more realistic example.  Basically this is staff data so each employee has a record on the database.  With other databases I've used there would be separate APIs to upload (eg) personal data, address data, job data, etc, but this database allows everything to be uploaded together so long as each record type is on a new line in the csv file.  Hence my issue.

    This file is my Excel spreadsheet where I'm working on all the data in one place.

    https://www.dropbox.com/s/oy8izs24u1jaiiy/LR%20csv%20query%20in%20Excel.xlsx?dl=0

    I've left the header rows in to show you what data I'm dealing with but in reality they would be removed, however, the columns without data would stay otherwise the system would think surname goes in 10PERDET column 2.  Therefore the record type columns are important as the tell the API that data following 00EMPLOYEE is the new record, then 10PERDET data goes into the People table, then 15ADDDET goes into the address table, etc.

    If you open this file in Notepad you'll see I've put in the line breaks at the change in record type indicator but I've done this manually and I'll have 120 staff records with I'm not sure yet how many record types so I don't want to be pressing return 500-odd times!

    https://www.dropbox.com/s/wrxylvqijh89g05/LR%20csv%20query%20in%20csv.csv?dl=0

    I hope that makes more sense and thanks for looking at this.  ;0)

    Was this answer helpful?

    0 comments No comments