Power Query - Loop through unknown number of Tables and append them all - solved

Anonymous
2018-07-17T09:29:18+00:00

Hello,

I have an Excel spreadsheet with a lot of columns but I want to save it in csv format with line breaks to put certain columns on different lines.   This is because I can upload different record types into my database but each record type needs its own line in csv with a header type as the first field on that line.

eg this in Excel:

col1   col2   col3   col4   col5   col6   col7   col8   col9   col10

01        b        c        d       e        02         g       h        03         j

01        r         s         t       u        02         v       w        03         x

becomes this in csv:

01, b, c, d, e

02, g, h

03, j

01, r, s, t, u

02, v, w

03, x              

Is this possible?  Easy?  

thanks

Louise

<The thread has been moved to the correct category by forum moderator>

Microsoft 365 and Office | Excel | For home | Windows

Locked Question. This question was migrated from the Microsoft Support Community. You can vote on whether it's helpful, but you can't add comments or replies or follow the question.

0 comments No comments
Answer accepted by question author
Lz365 38,201 Reputation points Volunteer Moderator
2018-08-01T12:23:56+00:00

This is a very simplified version of this thread. We have an Input Table that looks like:

where each Group of colored columns needs to be transformed in a separate Table (so 3 in this case) and at the end we want the 3 Tables to be Appended (Combined according to Power Query M language)

Challenge

The number of Groups may vary (i.e. from 1 to say 10) from time to time. So we need a query that is flexible enough to handle this variation

Note

In the above example the 1st Group has 3 columns, the 2nd has 4 and the last has 5. To properly Append/Combine Tables they all must have the same number of columns. That aspect of the problem is not detailed in the below Power Query M snipet.

How To

Power Quey M language doesn't offer an easy way to loop. This example uses the List.Generate function to do that the appropriate number of times to build each Table and then Append/Combine them

let

    ...

    GroupsList = Record.ToList(Table.FirstN(Source, 1){0}),

    NbGroups = List.Count(List.Distinct(GroupsList)),

    // Loop (actually NbGroups of times) to create a Table for each "Group"

    // At the end we get a List that contains Table(s)

    The next Step will Append/Combine all Table(s)

    TablesToAppend = List.Generate(

         ()=> [i = -1, j = 0, OutputTable = #table({"Col","Index","Index2"},{})],

            each [i] < NbGroups,

                each

                    [

                        j = [j] +1,

                        ...,              // step1 to build the OutputTable

                        ...,              // step2 to build the OutputTable

                        TableIdx = ...,   // step3 to build the OutputTable

                        OutputTable = Table.AddIndexColumn(TableIdx, "Index2", j, 1),

                        i = [i] +1

                    ],

                each [OutputTable]

                                  ),

    // Append all tables from above List

    TablesAppended = Table.Combine(List.Transform(TablesToAppend, each (_))),

    ...

in

    ...

Was this answer helpful?

2 people found this answer helpful.
0 comments No comments

59 additional answers

Sort by: Newest
  1. Lz365 38,201 Reputation points Volunteer Moderator
    2018-07-26T10:10:48+00:00

    Just used another PC (running Windows 8.1) as a Server. Made 2 tests:

    1. Input file on the Server and Output file on my local drive => No issue
    2. Input AND Output files on the Server  => No issue

    Was this answer helpful?

    0 comments No comments
  2. Lz365 38,201 Reputation points Volunteer Moderator
    2018-07-26T09:52:31+00:00

    I see now ;-). I am not 100% sure but this seems to be a classic problem of Source unavailability (I don't have a server here to try replicating).

    You said earlier: I've saved both docs on my drive and you can see from the screen shot… and the Path to the INPUT file starts with **X:**.... Drive X: is rarely a local drive on a PC. More often it's a virtual drive mapped to a Server/OtherPC. And of course if - for whatever reason - the server isn't available you don't have access to its resources (files, printers…).

    To ensure we are not facing a Server/Resource unavailability issue I suggest you copy the INPUT & OUTPUT files to a drive that you're sure exist on your PC and try again

    Was this answer helpful?

    0 comments No comments
  3. Anonymous
    2018-07-26T09:32:04+00:00

    Oh don't know why but by screen shot didn't capture the error message!!  I've redone it:

    https://www.dropbox.com/s/d3mu6g69469e3w5/Output%20error%20message.docx?dl=0

    Was this answer helpful?

    0 comments No comments
  4. Lz365 38,201 Reputation points Volunteer Moderator
    2018-07-26T08:58:58+00:00

    Good morning Louise

    Just opened your Word doc. OK, it shows the Input file path as X:.... but I Don't see any error message anywhere on your screenshot :-((

    Without an error message I Don't see I can guide you further. So, could you do the same thing as yesterday with your Input file located on X:\…., click on the Refresh Query… button and drop me in a way or another the error message you get

    Was this answer helpful?

    0 comments No comments