How to get stack traces working with kubernetes containerd using RKE2?

Kraus, Alois 0 Reputation points
2026-01-22T07:08:08.78+00:00

When profiling a Windows container which is using Host Process Isolation (this means it is running under System account with full access to host registry, file system and other processes). The only part which comes from the docker image is the mount point c:\hpc where the container file system data for the executables are located.

In theory I should be able to profile these container from the host as is but this works only for some dlls but not all of them.

The dlls which get proper image id data can be retrieved from host during merge step are reachable from containerd snapshot:

C:\var\lib\rancher\rke2\agent\containerd\io.containerd.snapshotter.v1.windows\snapshots\413\Files\product\bin\boost date_time-mt.dll 

While all other dlls are only accesible via its Vhd path:

\Device\VhdHardDisk{8cd8f3e4-93f5-47ce-83b2-78bd58522c39}\product\bin\product.dll 

The "normal" approach to successfully add image id events is to do the merge step on host and all containers like

xperf -stop "NT Kernel Logger" -stop UserModeLogger
xperf -merge kernel.etl user.etl mergedOnHost.etl

foreach(containerId in containers)
{
   kubectl exec containerId -i -t -- cmd /C xperf merge c:\temp\mergedOnhost.etl c:\temp\mergedinContainer.etl & del c:\temp\mergedOnhost.etl  & move c:\temp\mergedinContainer.etl  c:\temp\mergedOnHost.etl 
}

This will also properly generate NGEN pdbs which works because the image roots are stored in host registry and c:\windows\assembly... anyway. Just the path resoluton to c:\hpc\product\bin which is the path used as Native Image root. Dialing in into each container makes the path accesible and ngen createpbd calls will succeed.

But the Vhd paths are not resolved in this case. Is this a known limitation of the merge code inside containers. Who would be responsible to fix?

I have tried to inject image id events on my own by relogging an etl but for some reason I was not able to inject anything. I was using TraceEvent library from PerfView.

   internal unsafe void Inject()
   {
       using (var relogger = new ETWReloggerTraceEventSource(myInputEtlFile, myOutputEtlFile) { OutputUsesCompressedFormat = false })
       {
           bool bRelogged = false;
           relogger.AllEvents += (obj) =>
           {
               var time = GetDateTimeForUTCToBeUsedForWriteEvent(relogger, 100);
               relogger.WriteEvent(obj);
               Guid ImageIdGuid = new Guid("B3E675D7-2554-4f18-830B-2762732560DE");
               if (obj.TaskGuid == ImageIdGuid && (int) obj.Opcode == 64 && !bRelogged)
               {
                   bRelogged = true;
                   var ed = new _EVENT_DESCRIPTOR
                   {
                       Id = 0,   
                       Task = 0,
                       Opcode = 64, // Image ID Version but any other opcode does also not show up in relogged etl file
                       Version = 0,
                       Channel = 0x0,
                       Level = 0,
                       Keyword = 0,
                   };
                   for (int i = 1; i < 2000; i++)
                   {
                        relogger.WriteEvent(ImageIdGuid, ref ed, time, obj.ProcessID, 0, obj.ThreadID, obj.ActivityID,
                       new object[] {
                       1234,
                       0x1000,
                       "origFileName.dll"+i,
                       "file description",
                       "file version string"+i,
                       "bin file version string",
                       "ver_language string",
                       "product name",
                       "company Name",
                       "product version",
                       "File Id",
                       "Program id"
                           }
                       );
                   }
              }
               
           };
           relogger.Process();
       }
   }
        private DateTime GetDateTimeForUTCToBeUsedForWriteEvent(ETWReloggerTraceEventSource relogger, double relativeTimeToStartInMS)
        {
            Type type = typeof(TraceEventSource);
            MethodInfo UTCDateTimeToQPCMethode = type.GetMethod("RelativeMSecToQPC", BindingFlags.NonPublic | BindingFlags.Instance);
            var relativeTimeToStart = relativeTimeToStartInMS;
            var modTime = (long)UTCDateTimeToQPCMethode.Invoke(relogger, new object[] { relativeTimeToStart });
            var newTime = DateTime.FromFileTimeUtc(modTime);
            return newTime;
        }

When I inject events they do not show up in PerfView as I would expect them. So something is wrong, but I am not sure why.

Windows development | Windows Performance Toolkit
0 comments No comments

3 answers

Sort by: Most helpful
  1. Kraus, Alois 0 Reputation points
    2026-01-22T11:53:40.67+00:00

    Hi Gary,

    I have checked to emit existing events. PerfView can dump the Image ID events where 64 is an existing one:

    <Event MSec=  "6333.3967" PID="12964" PName=   "xperf" TID="11608" EventName="ImageID/FileVersion"
      TimeStamp="01/12/26 13:48:26.971825" ID="Illegal" Version="0" Keywords="0x00000000" TimeStampQPC="168,815,739,072" QPCTime="0.100us"
      Level="Always" ProviderName="KernelTraceControl" ProviderGuid="28ad2447-105b-4fe2-9599-e59b2aa9a634" ClassicProvider="True" ProcessorNumber="6"
      Opcode="64" TaskGuid="b3e675d7-2554-4f18-830b-2762732560de" Channel="0" PointerSize="8"
      CPU="6" EventIndex="2300" TemplateType="FileVersionTraceData">
      <PrettyPrint>
        <Event MSec=  "6333.3967" PID="12964" PName=   "xperf" TID="11608" EventName="ImageID/FileVersion" ImageSize="0x00079000" TimeDateStamp="0xFFFFFFFFB1EF349B" OrigFileName="MpOAV.dll" FileDescription="IOfficeAntiVirus Module" FileVersion="4.18.2104.5 (WinBuild.160101.0800)" BinFileVersion="4.18.2104.5" VerLanguage="1033" ProductName="Microsoft® Windows® Operating System" CompanyName="Microsoft Corporation" ProductVersion="4.18.2104.5" FileId="" ProgramId=""/>
      </PrettyPrint>
      <Payload Length="326">
           0:   0 90  7  0 9b 34 ef b1 | 4d  0 70  0 4f  0 41  0   .....4.. M.p.O.A.
          10:  56  0 2e  0 64  0 6c  0 | 6c  0  0  0 49  0 4f  0   V...d.l. l...I.O.
          20:  66  0 66  0 69  0 63  0 | 65  0 41  0 6e  0 74  0   f.f.i.c. e.A.n.t.
          30:  69  0 56  0 69  0 72  0 | 75  0 73  0 20  0 4d  0   i.V.i.r. u.s. .M.
          40:  6f  0 64  0 75  0 6c  0 | 65  0  0  0 34  0 2e  0   o.d.u.l. e...4...
          50:  31  0 38  0 2e  0 32  0 | 31  0 30  0 34  0 2e  0   1.8...2. 1.0.4...
          60:  35  0 20  0 28  0 57  0 | 69  0 6e  0 42  0 75  0   5. .(.W. i.n.B.u.
          70:  69  0 6c  0 64  0 2e  0 | 31  0 36  0 30  0 31  0   i.l.d... 1.6.0.1.
          80:  30  0 31  0 2e  0 30  0 | 38  0 30  0 30  0 29  0   0.1...0. 8.0.0.).
          90:   0  0 34  0 2e  0 31  0 | 38  0 2e  0 32  0 31  0   ..4...1. 8...2.1.
          a0:  30  0 34  0 2e  0 35  0 |  0  0 31  0 30  0 33  0   0.4...5. ..1.0.3.
          b0:  33  0  0  0 4d  0 69  0 | 63  0 72  0 6f  0 73  0   3...M.i. c.r.o.s.
          c0:  6f  0 66  0 74  0 ae  0 | 20  0 57  0 69  0 6e  0   o.f.t...  .W.i.n.
          d0:  64  0 6f  0 77  0 73  0 | ae  0 20  0 4f  0 70  0   d.o.w.s. .. .O.p.
          e0:  65  0 72  0 61  0 74  0 | 69  0 6e  0 67  0 20  0   e.r.a.t. i.n.g. .
          f0:  53  0 79  0 73  0 74  0 | 65  0 6d  0  0  0 4d  0   S.y.s.t. e.m...M.
         100:  69  0 63  0 72  0 6f  0 | 73  0 6f  0 66  0 74  0   i.c.r.o. s.o.f.t.
         110:  20  0 43  0 6f  0 72  0 | 70  0 6f  0 72  0 61  0    .C.o.r. p.o.r.a.
         120:  74  0 69  0 6f  0 6e  0 |  0  0 34  0 2e  0 31  0   t.i.o.n. ..4...1.
         130:  38  0 2e  0 32  0 31  0 | 30  0 34  0 2e  0 35  0   8...2.1. 0.4...5.
         140:   0  0  0  0  0  0       |                           ......
      </Payload>
    </Event>
    

    I do never see any event emitted in the rewritten etl file, although I am sure in the input file such events exist. I did write 2000 events to easily find them again later in the file but the event counts did not increase accordingly.

    WPA is bad at showing events because the Generic View table only shows events it has not already parsed for another table.

    On never knows if some events do not exist, or they are hidden because some predefined table did "eat" them. This is true especially for Image ID events. I guess I will need to ask the WPA team directly.

    Was this answer helpful?

    0 comments No comments

  2. Gary Nebbett 6,536 Reputation points
    2026-01-22T11:06:29.9266667+00:00

    Hello Alois,

    Below is a snapshot from the WPA Trace Statistics (looking at a merged trace).

    User's image

    The event from the "Image" provider come from the "raw" ETL and the events from the ImageId provider are injected during the merge process.

    Your code is looking for what WPA calls "ImageId: FileVersion" events (opcode 64 / 0x40) and, if found, then injecting more events of the same type. However, since there are no "ImageId: FileVersion" events in the input ETL then none will be added to the output ETL. The "Image" provider Load and Rundown events are what you need to look for (they contain the file path of the image).

    Of the events fabricated and injected during a "merge", it is the "DbgId: RSDS" (opcode 36 / 0x24) events that are most useful for symbol loading, since they contain PDB signature and age information.

    Gary

    Was this answer helpful?


  3. AI answer

    2026-01-22T07:08:18.6066667+00:00

    To profile Windows containers using Host Process Isolation with RKE2 and containerd, you may encounter limitations regarding the resolution of VHD paths during the merge step. This is primarily because the merge process relies on the accessibility of DLLs and their paths, which can differ based on how the container is set up and the underlying storage mechanisms used by containerd.

    In your case, DLLs accessible via the containerd snapshot can be merged successfully, but those only reachable through VHD paths may not be resolved correctly. This behavior could be attributed to how the merge code handles paths within the container environment, particularly with respect to VHDs.

    As for the injection of image ID events, if they are not appearing in PerfView as expected, it could be due to several factors:

    • Ensure that the event descriptors and their properties are correctly defined and match the expected format.
    • Verify that the event writing logic is functioning as intended and that the events are being processed correctly by the relogger.
    • Check if there are any permission issues or other constraints that might prevent the events from being logged or displayed.

    If this is a known limitation, it would typically require investigation from the maintainers of the container runtime or the relevant tooling, as they would be responsible for addressing such issues in the merge functionality.

    For further troubleshooting, consider reviewing the logs generated during the profiling and merging processes to identify any specific errors or warnings that could provide more insight into the issue.

    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.