Slow wired upload speed vs Linux on same hardware

Hello @mensa84 et al.,

I think that I understand at an abstract level what is going wrong. In the trace there are 127 retransmissions: 94 fast retransmits and 33 normal retransmits. Each retransmission causes the "congestion window" to close a little (and, therefore, the throughput to be reduced). Most of the "fast retransmissions" are "spurious" (were not actually needed). WireShark "Expert Information" shows:

Both "fast retransmission" and "spurious retransmission" are mentioned. Here is a more detailed view of what is happening:

At 22:18:14.967353 the byte range 172967:174387 is transmitted for the first time.

Between 22:18:14.997584 and 22:18:14.997586 (30.231 milliseconds after the above and within a 2 microsecond period), 12 acknowledgements appear in the event trace (they have probably been "collected" by a lower level and then "indicated" as a group).

At this point in the trace, one estimate of "Round Trip Time" (RackMinRtt) is 18.994 milliseconds. Either the three occurrences of 172967 as the next "unacknowledged" sequence number (simple duplicated acknowledgement threshold) or a combination of this with "recent acknowledgement" ("rack") timings suggest that a "fast retransmission" would be apposite. This happens before the next packet is examined a few nanoseconds later, showing that the "missing" byte range has been acknowledged.

These images of the Microsoft-Windows-TCPIP trace data show that the packets arrive as a group and that, subsequently, a fast retransmission occurs:

It is the high degree of "out-of-order" delivery of TCP segments (i.e. segments that are not received in the same order/sequence as which they were sent in) which perhaps limits the number of users who experience this behaviour.

I don't yet fully understand what can be done to influence this behaviour. Setting (in the registry) TcpMaxDupAcks to 3 might help (see https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/description-tcp-features) but, beyond that; I don't currently have any suggestions.

Another question is whether this behaviour has any impact on one's typical workload. It certainly substantially slows "streaming" of data to a server (as happens when posting data to a web server - this is what many "speed tests" measure), but it might have less (or no) impact when uploading data to a file share using SMB encapsulated in a VPN connection (in a "working from home" scenario) - the command/response nature of such transfers (and perhaps the fact that the "top level" protocol is Encapsulating Security Payload (ESP) rather than TCP) might slow down all devices/OSs to approximately the same level.

Gary

I am receiving a very similar (actually looks identical) issue on my Windows 10 with both my Ethernet and WiFi connections on my PC. However, If i use linux (via bootable USB), the issue doesn't persist. the issue didn't exist before but around 2 weeks i started noticing it. It became a major issue when i was streaming to my youtube channel and my upload speed massively tanked. Usually I stream at 10k bitrate but this time it was struggling to send even 2k.

Hello @mensa84 ,

I need to do some more checking to verify whether this is always or mostly the case, but I think that there are hints of the cause of the problem in the trace data. Here is a short extract:

At some time before this extract, the 1420 bytes between 172967:174387 were sent from your system. In the extract above, we can see that the bytes up to 172967 were acknowledged and, initially, some bytes after 174387 were acknowledged with a "selective acknowledgement" (sack) - a warning that the bytes between 172967:174387 may have been lost.

A little later (at 14.997585, highlighted in blue) the bytes between 172967:174387 are finally acknowledged.

Now, very oddly, 100 microseconds later (at 14.997685, highlighted in yellow), your system retransmits the bytes between 172967:174387. If the timings in the trace are accurate and inferences drawn from the trace data match the Windows TCP/IP implementation's understanding of the state of affairs, there is no reason to retransmit the data.

The retransmission of the data is not, in itself, a major problem but the fact that a retransmission was "necessary" causes the "congestion window" to be reduced. The congestion window is dynamically updated throughout the transfer but never increases to anywhere near the size needed to get the full "nominal" throughput of the link.

My suspicion is that there is some "receive offloading" taking place - if that is the case and is the cause of the slow performance, we might be able to disable it. I will do some more analysis to increase my confidence level that this is the problem.

Gary

Hello,

thanks for the explanation!
I did this now, here is the SlowUp.etl zipped: https://drive.google.com/file/d/1rt_jvuq-zr4XPu-OV4bth8iLYMJY08i_/view?usp=sharing

Do you see anything?

Hello @mensa84 ,

Open a PowerShell window running as administrator. This image shows one way of doing that (right mouse clicking over the Windows PowerShell icon and choosing "Run as Administrator"):

Now get ready to run the speed test (but don't start it yet) and then paste the following four commands into the PowerShell window:

 New-NetEventSession -LocalFilePath $Env:TEMP\SlowUp.etl -Name SlowUp  
 Add-NetEventPacketCaptureProvider -TruncationLength 100 -Level 255 -SessionName SlowUp  
 Add-NetEventProvider -Name "Microsoft-Windows-TCPIP" -Level 255 -SessionName SlowUp  
 Start-NetEventSession -Name SlowUp  

Now start the speed test. As soon as it ends, paste the following two commands into the PowerShell window:

 Stop-NetEventSession -Name SlowUp  
 Remove-NetEventSession  

You should now have a file containing the trace data (%TEMP%\SlowUp.etl, i.e. a file named SlowUp.etl in your temporary files directory).

The file will be quite large, and the best way of sharing it is by copying it to a network storage service, such as Microsoft OneDrive, Google Drive, Dropbox, etc., and posting a link to the file.

Gary