Slow wired upload speed vs Linux on same hardware

Dawid Oosthuizen 26

Intel® Ethernet Controller X550-AT2, 10G network interface on ASRock Rack ROMED8-2T with AMD EPYC 7232P processor.
Windows 10 Pro for Workstations, 2004.
Latest Windows updates and Intel drivers installed as of 09/09/2020.

This machine is a dual boot, the above Windows version, and Ubuntu 20.04.
When doing a speed test, I get good performance from Ubuntu, but very poor uploads from Windows. This is on the same machine, the exact same hardware.

The WAN link is 1000Mbps down, 50Mbps up.

This is the Windows speedtest result:

This is the Ubuntu speedtest result:

I have tried to tweak the adapter's advanced driver settings in Windows, such as disabling LSO, etc. No luck, performance remains poor.

I've also noticed it on another PC running Windows 10 Pro, and a laptop running Windows 10 Pro for Workstations, both give the same poor upload performance. Whereas my other Ubuntu 20.04 Server machine, and also my phone connected via Wi-Fi, is getting good upload speeds.

I have even taken the Windows laptop and plugged it straight into my incoming WAN connection (bypassing router), and it still gets poor upload speeds.

Incidentally, when the speed test is running, I can see that the upload looks bursty on Windows, like it is only getting chunks of data here and there, while in Linux and on Android it looks the same as the download, the graph is drawn at a consistent high rate and with consistent high values.

franco d'esaro 25,181 Reputation points Volunteer Moderator

2020-09-09T12:47:10.917+00:00

Hello,
Firewall / Antivirus in use on Windows machines?
Try running and maintaining a Clean Boot on Windows machines:
https://support.microsoft.com/en-us/help/929135/how-to-perform-a-clean-boot-in-windows
Dawid Oosthuizen 26 Reputation points

2020-09-10T01:19:34.907+00:00
Thanks for the suggestions, unfortunately no luck:

I'm using Microsoft's Windows Defender, not a third party product. And disabling Defender (virus protection and firewall) had no effect.

Also did a clean boot (only MS services enabled), no effect.

I've added some more info in my original post, FWIW.
mensa84 6 Reputation points

2020-11-12T00:22:07.743+00:00

Did you find a solution? I habe exactly the same problem :(
Guery Illanes 1 Reputation point

2021-07-08T23:42:19.677+00:00

Hi,
Try with PDANET software. Must be installed in laptop with Windows and also in mobile. It works fine with this issue.

Guery
Guery Illanes 1 Reputation point

2021-07-08T23:42:45.823+00:00

Hi,
Try with PDANET software. Must be installed in laptop with Windows and also in mobile. It works fine with this issue.

Guery
@Live MiaoSKY 0 Reputation points

2025-03-31T01:57:03.21+00:00

This problem has been bothering me for a long time, and I finally found out that it was a problem with speedtest.net. It limits the user-agent of edge. If you use the user-agent of edge, the upload speed is only about 5mbps.

After I used the extension to change it to the user-agent of chrome, the upload speed went up.... Fuck speedtest.net!!!

这个问题让我头疼很久, 最后发现是speedtest.net的问题. 它限制了edge的user-agent, 如果你使用edge的user-agent, 上传速度就只有5mbps左右.

在我使用扩展插件修改成chrome的user-agent以后上传速度就上去了.... 真是fuck speedtest.net !!!

50 answers

Egoist 1 Reputation point

2020-11-28T13:11:20.32+00:00

@GaryNebbett-6715

Thanks for the clarification and all the effort to help me understand the root of this issue, that was really informative as it's an area of my interest.

I will try to report the problem to the people who manage the network, hopefully it can get fixed. Although i still wonder what Clear Linux is doing (i suspect they have a different Congestion Control Algorithm maybe?)

Edit: It appears to be using BBR Congestion Control, but i am not sure if that's what is giving it the advantage.

https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-congestion-control-02.pdf

Thanks again for your time
Was this answer helpful?

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Gary Nebbett 6,536

Hello @Egoist ,

Sorry, I was too quick to cast aspersions on the "Large Send Offload" functionality. After longer consideration, all of the problems seem to arise outside of Windows.

Let's look at some of the interesting behaviour.

11:39:17.343617 10.132.84.7.58616 > 24.220.62.52.8080: P 63:124(61) ack 76 win 1026 (DF)  
11:39:17.343635 truncated-ip 0  
11:39:17.343654 truncated-ip 0  
11:39:17.343662 truncated-ip 0  
11:39:17.343670 10.132.84.7.58616 > 24.220.62.52.8080: . 12412:13872(1460) ack 76 win 1026 (DF)  
11:39:17.348628 24.220.62.52.8080 > 10.132.84.7.58616: . ack 63 win 251 <nop,nop,sack 4220:5680> (DF)  
11:39:17.349441 24.220.62.52.8080 > 10.132.84.7.58616: . ack 63 win 274 <nop,nop,sack 7140:8316 4220:5680> (DF)  
11:39:17.349442 24.220.62.52.8080 > 10.132.84.7.58616: . ack 63 win 297 <nop,nop,sack 9776:11236 7140:8316 4220:5680> (DF)  
11:39:17.349442 24.220.62.52.8080 > 10.132.84.7.58616: . ack 63 win 320 <nop,nop,sack 9776:12412 7140:8316 4220:5680> (DF)  
11:39:17.349442 24.220.62.52.8080 > 10.132.84.7.58616: . ack 63 win 343 <nop,nop,sack 9776:13872 7140:8316 4220:5680> (DF)  
11:39:17.349604 24.220.62.52.8080 > 10.132.84.7.58616: . ack 124 win 343 <nop,nop,sack 9776:13872 7140:8316 4220:5680> (DF)

First, 61 bytes are sent, then 3 large segments (each carrying 4096 bytes - which are sent as two 1460 byte packets and one 1176 byte packet by the network adapter) and then a 1460 byte packet.

Those first 61 bytes are first acknowledged in the sixth ack response - long after later packets have been acknowledged with a "sack" (selective acknowledgement). The subsequent acks also indicate that packets are being received by the server "out of order".

"Out of order" reception typically results from packet loss plus retransmission, but no retransmission is taking place at this time and some packets seem to be "overtaking" others as they cross the network.

The frequent packet loss quickly reduces the "congestion window" which starts at 10 * MSS and soon falls to just one MSS:

SeqNo = 648939448, BytesSent = 0, CWnd = 14600  
SeqNo = 648939449, BytesSent = 0, CWnd = 14600  
SeqNo = 648939449, BytesSent = 62, CWnd = 14600  
SeqNo = 648939511, BytesSent = 61, CWnd = 14662  
SeqNo = 648939572, BytesSent = 4096, CWnd = 14662  
SeqNo = 648943668, BytesSent = 4096, CWnd = 14662  
SeqNo = 648947764, BytesSent = 4096, CWnd = 14662  
SeqNo = 648951860, BytesSent = 2313, CWnd = 14662  
SeqNo = 648953320, BytesSent = 6815, CWnd = 17643  
SeqNo = 648942492, BytesSent = 1460, CWnd = 1460  
SeqNo = 648942492, BytesSent = 1460, CWnd = 1460  
SeqNo = 648960135, BytesSent = 2920, CWnd = 2920  
SeqNo = 648945128, BytesSent = 2920, CWnd = 2920  
SeqNo = 648948048, BytesSent = 1460, CWnd = 4380  
SeqNo = 648949508, BytesSent = 2922, CWnd = 4382  
SeqNo = 648953320, BytesSent = 4383, CWnd = 4383  
SeqNo = 648953320, BytesSent = 1460, CWnd = 1460  
SeqNo = 648953320, BytesSent = 1460, CWnd = 1460  
SeqNo = 648953320, BytesSent = 1460, CWnd = 1460  
SeqNo = 648963055, BytesSent = 4380, CWnd = 4380  
SeqNo = 648963055, BytesSent = 1460, CWnd = 1460  
SeqNo = 648967435, BytesSent = 2920, CWnd = 2920  
SeqNo = 648964515, BytesSent = 2920, CWnd = 2920  
SeqNo = 648967435, BytesSent = 1460, CWnd = 4380  
SeqNo = 648964515, BytesSent = 1460, CWnd = 1460  
SeqNo = 648970355, BytesSent = 2920, CWnd = 3066  
SeqNo = 648968895, BytesSent = 3066, CWnd = 3066  
SeqNo = 648971815, BytesSent = 1460, CWnd = 4380  
SeqNo = 648968895, BytesSent = 1460, CWnd = 1460  
SeqNo = 648973275, BytesSent = 4526, CWnd = 4526  
SeqNo = 648977655, BytesSent = 1607, CWnd = 4527  
SeqNo = 648974735, BytesSent = 1460, CWnd = 1460  
SeqNo = 648974735, BytesSent = 1460, CWnd = 1460  
SeqNo = 648974735, BytesSent = 1460, CWnd = 1460  
SeqNo = 648979115, BytesSent = 4380, CWnd = 4380  
SeqNo = 648983495, BytesSent = 4383, CWnd = 4383  
SeqNo = 648987875, BytesSent = 4386, CWnd = 4386  
SeqNo = 648992255, BytesSent = 1467, CWnd = 4387  
SeqNo = 648993715, BytesSent = 1861, CWnd = 4781  
SeqNo = 648990795, BytesSent = 1460, CWnd = 1460  
SeqNo = 648995175, BytesSent = 2920, CWnd = 2920  
SeqNo = 648992255, BytesSent = 4380, CWnd = 4380  
SeqNo = 648998095, BytesSent = 4381, CWnd = 4381  
SeqNo = 649002475, BytesSent = 4384, CWnd = 4384  
SeqNo = 649006855, BytesSent = 1977, CWnd = 4897  
SeqNo = 649003935, BytesSent = 1460, CWnd = 1460  
SeqNo = 649008315, BytesSent = 3427, CWnd = 3427  
SeqNo = 649011235, BytesSent = 3429, CWnd = 3429  
SeqNo = 649014155, BytesSent = 4041, CWnd = 4041  
SeqNo = 649017075, BytesSent = 4954, CWnd = 4954  
SeqNo = 649021455, BytesSent = 5867, CWnd = 5867  
SeqNo = 649027295, BytesSent = 2215, CWnd = 6595  
SeqNo = 649029510, BytesSent = 1460, CWnd = 6595  
SeqNo = 649030970, BytesSent = 1460, CWnd = 6595  
SeqNo = 649022915, BytesSent = 1460, CWnd = 5840  
SeqNo = 649032430, BytesSent = 1460, CWnd = 5840  
SeqNo = 649033890, BytesSent = 1460, CWnd = 5840  
SeqNo = 649035350, BytesSent = 1460, CWnd = 5840  
SeqNo = 649036810, BytesSent = 1460, CWnd = 5840  
SeqNo = 649038270, BytesSent = 1460, CWnd = 5840  
SeqNo = 649039730, BytesSent = 1460, CWnd = 5840  
SeqNo = 649022915, BytesSent = 1460, CWnd = 1460  
SeqNo = 649041190, BytesSent = 948, CWnd = 5548  
SeqNo = 649042139, BytesSent = 0, CWnd = 5549

The impact that this has on performance can be seen clearly here:

11:39:18.271541 24.220.62.52.8080 > 10.132.84.7.58616: . ack 8600 win 594 <nop,nop,sack 7140:8316 15332:23607 9776:13872> (DF)  
11:39:18.275841 24.220.62.52.8080 > 10.132.84.7.58616: . ack 13872 win 616 <nop,nop,sack 9776:10060 15332:23607> (DF)  
11:39:18.275916 10.132.84.7.58616 > 24.220.62.52.8080: . 13872:15332(1460) ack 76 win 1026 (DF)  
11:39:18.587669 10.132.84.7.58616 > 24.220.62.52.8080: . 13872:15332(1460) ack 76 win 1026 (DF)  
11:39:19.196868 10.132.84.7.58616 > 24.220.62.52.8080: . 13872:15332(1460) ack 76 win 1026 (DF)  
11:39:20.399855 10.132.84.7.58616 > 24.220.62.52.8080: . 13872:15332(1460) ack 76 win 1026 (DF)  
11:39:20.406807 24.220.62.52.8080 > 10.132.84.7.58616: . ack 23607 win 639 (DF)

Windows is trying to fill the gap at 13872:15332 and can only send one MSS at a time because the congestion window is now that size (i.e. Windows has to wait until this data is acked before it can send any more). In fact it re-sends this packet 4 times at ever increasing intervals until, after two seconds, it is finally acknowledged.

The Windows system always seems to be the victim of the network behaviour without itself having put a foot wrong - there appears to be nothing that can be done from the Windows side to improve the situation.

Gary

0 comments

Egoist 1

Hey @GaryNebbett-6715 ,
Thanks for analyzing the etl file.

I tried playing around with the LSO settings on the NIC, but it doesn't appear to make change to the performance. I disabled it for now.

What's still weird for me is how Clear Linux have the same features enabled as they are on Ubuntu, that's including all offload settings. But somehow Clear Linux tend to have higher speed.

Features for enp4s0:  
rx-checksumming: on  
tx-checksumming: on  
	tx-checksum-ipv4: off [fixed]  
	tx-checksum-ip-generic: on  
	tx-checksum-ipv6: off [fixed]  
	tx-checksum-fcoe-crc: off [fixed]  
	tx-checksum-sctp: off [fixed]  
scatter-gather: on  
	tx-scatter-gather: on  
	tx-scatter-gather-fraglist: off [fixed]  
tcp-segmentation-offload: on  
	tx-tcp-segmentation: on  
	tx-tcp-ecn-segmentation: off [fixed]  
	tx-tcp-mangleid-segmentation: off  
	tx-tcp6-segmentation: on  
generic-segmentation-offload: on  
generic-receive-offload: on  
large-receive-offload: off [fixed]  
rx-vlan-offload: on  
tx-vlan-offload: on  
ntuple-filters: off [fixed]  
receive-hashing: on  
highdma: on [fixed]  
rx-vlan-filter: on [fixed]  
vlan-challenged: off [fixed]  
tx-lockless: off [fixed]  
netns-local: off [fixed]  
tx-gso-robust: off [fixed]  
tx-fcoe-segmentation: off [fixed]  
tx-gre-segmentation: off [fixed]  
tx-gre-csum-segmentation: off [fixed]  
tx-ipxip4-segmentation: off [fixed]  
tx-ipxip6-segmentation: off [fixed]  
tx-udp_tnl-segmentation: off [fixed]  
tx-udp_tnl-csum-segmentation: off [fixed]  
tx-gso-partial: off [fixed]  
tx-tunnel-remcsum-segmentation: off [fixed]  
tx-sctp-segmentation: off [fixed]  
tx-esp-segmentation: off [fixed]  
tx-udp-segmentation: off [fixed]  
tx-gso-list: off [fixed]  
fcoe-mtu: off [fixed]  
tx-nocache-copy: off  
loopback: off [fixed]  
rx-fcs: off  
rx-all: off  
tx-vlan-stag-hw-insert: off [fixed]  
rx-vlan-stag-hw-parse: off [fixed]  
rx-vlan-stag-filter: off [fixed]  
l2-fwd-offload: off [fixed]  
hw-tc-offload: off [fixed]  
esp-hw-offload: off [fixed]  
esp-tx-csum-hw-offload: off [fixed]  
rx-udp_tunnel-port-offload: off [fixed]  
tls-hw-tx-offload: off [fixed]  
tls-hw-rx-offload: off [fixed]  
rx-gro-hw: off [fixed]  
tls-hw-record: off [fixed]  
rx-gro-list: off  
macsec-hw-offload: off [fixed]

That's on both systems.

0 comments

Gary Nebbett 6,536

Hello @Egoist ,

Focusing on that upload with 68 data bearing segments (your second TCP connection summary), a tcpdump format of the network traffic looks like this:

11:39:17.322590 10.132.84.7.58616 > 24.220.62.52.8080: S 648939448:648939448(0) win 64240 <mss 1460,nop,wscale 8,nop,nop,sackOK> (DF)  
11:39:17.327362 24.220.62.52.8080 > 10.132.84.7.58616: S 3512509149:3512509149(0) ack 648939449 win 29200 <mss 1460,nop,nop,sackOK,nop,wscale 7> (DF)  
11:39:17.327387 10.132.84.7.58616 > 24.220.62.52.8080: . ack 1 win 1026 (DF)  
11:39:17.338006 10.132.84.7.58616 > 24.220.62.52.8080: P 1:63(62) ack 1 win 1026 (DF)  
11:39:17.342646 24.220.62.52.8080 > 10.132.84.7.58616: . ack 63 win 229 (DF)  
11:39:17.343033 24.220.62.52.8080 > 10.132.84.7.58616: P 1:76(75) ack 63 win 229 (DF)  
11:39:17.343617 10.132.84.7.58616 > 24.220.62.52.8080: P 63:124(61) ack 76 win 1026 (DF)  
11:39:17.343635 truncated-ip 0  
11:39:17.343654 truncated-ip 0  
11:39:17.343662 truncated-ip 0  
11:39:17.343670 10.132.84.7.58616 > 24.220.62.52.8080: . 12412:13872(1460) ack 76 win 1026 (DF)  
11:39:17.348628 24.220.62.52.8080 > 10.132.84.7.58616: . ack 63 win 251 <nop,nop,sack 4220:5680> (DF)  
11:39:17.349441 24.220.62.52.8080 > 10.132.84.7.58616: . ack 63 win 274 <nop,nop,sack 7140:8316 4220:5680> (DF)  
11:39:17.349442 24.220.62.52.8080 > 10.132.84.7.58616: . ack 63 win 297 <nop,nop,sack 9776:11236 7140:8316 4220:5680> (DF)  
11:39:17.349442 24.220.62.52.8080 > 10.132.84.7.58616: . ack 63 win 320 <nop,nop,sack 9776:12412 7140:8316 4220:5680> (DF)  
11:39:17.349442 24.220.62.52.8080 > 10.132.84.7.58616: . ack 63 win 343 <nop,nop,sack 9776:13872 7140:8316 4220:5680> (DF)  
11:39:17.349604 24.220.62.52.8080 > 10.132.84.7.58616: . ack 124 win 343 <nop,nop,sack 9776:13872 7140:8316 4220:5680> (DF)  
11:39:17.349605 24.220.62.52.8080 > 10.132.84.7.58616: . ack 1584 win 365 <nop,nop,sack 9776:13872 7140:8316 4220:5680> (DF)  
11:39:17.349605 24.220.62.52.8080 > 10.132.84.7.58616: . ack 3044 win 388 <nop,nop,sack 9776:13872 7140:8316 4220:5680> (DF)  
11:39:17.349631 10.132.84.7.58616 > 24.220.62.52.8080: . 13872:15332(1460) ack 76 win 1026 (DF)  
11:39:17.349631 10.132.84.7.58616 > 24.220.62.52.8080: P 15332:16792(1460) ack 76 win 1026 (DF)  
11:39:17.349631 10.132.84.7.58616 > 24.220.62.52.8080: . 16792:18252(1460) ack 76 win 1026 (DF)  
11:39:17.349632 10.132.84.7.58616 > 24.220.62.52.8080: . 18252:19712(1460) ack 76 win 1026 (DF)  
11:39:17.349632 10.132.84.7.58616 > 24.220.62.52.8080: P 19712:20687(975) ack 76 win 1026 (DF)  
11:39:17.354336 24.220.62.52.8080 > 10.132.84.7.58616: . ack 3044 win 411 <nop,nop,sack 15332:16792 9776:13872 7140:8316 4220:5680> (DF)  
11:39:17.354337 24.220.62.52.8080 > 10.132.84.7.58616: . ack 3044 win 434 <nop,nop,sack 15332:18252 9776:13872 7140:8316 4220:5680> (DF)  
11:39:17.354337 24.220.62.52.8080 > 10.132.84.7.58616: . ack 3044 win 457 <nop,nop,sack 15332:19712 9776:13872 7140:8316 4220:5680> (DF)  
11:39:17.354337 24.220.62.52.8080 > 10.132.84.7.58616: . ack 3044 win 480 <nop,nop,sack 15332:20687 9776:13872 7140:8316 4220:5680> (DF)  
11:39:17.650492 10.132.84.7.58616 > 24.220.62.52.8080: P 3044:4504(1460) ack 76 win 1026 (DF)  
11:39:18.259648 10.132.84.7.58616 > 24.220.62.52.8080: P 3044:4504(1460) ack 76 win 1026 (DF)  
11:39:18.265090 24.220.62.52.8080 > 10.132.84.7.58616: . ack 5680 win 502 <nop,nop,sack 4220:4504 15332:20687 9776:13872 7140:8316> (DF)  
11:39:18.265157 10.132.84.7.58616 > 24.220.62.52.8080: . 20687:22147(1460) ack 76 win 1026 (DF)  
11:39:18.265158 10.132.84.7.58616 > 24.220.62.52.8080: . 22147:23607(1460) ack 76 win 1026 (DF)  
11:39:18.265263 10.132.84.7.58616 > 24.220.62.52.8080: . 5680:7140(1460) ack 76 win 1026 (DF)  
11:39:18.265264 10.132.84.7.58616 > 24.220.62.52.8080: P 7140:8600(1460) ack 76 win 1026 (DF)  
11:39:18.269824 24.220.62.52.8080 > 10.132.84.7.58616: . ack 5680 win 525 <nop,nop,sack 15332:22147 9776:13872 7140:8316> (DF)  
11:39:18.269826 24.220.62.52.8080 > 10.132.84.7.58616: . ack 5680 win 548 <nop,nop,sack 15332:23607 9776:13872 7140:8316> (DF)  
11:39:18.269895 10.132.84.7.58616 > 24.220.62.52.8080: . 8600:10060(1460) ack 76 win 1026 (DF)  
11:39:18.271539 24.220.62.52.8080 > 10.132.84.7.58616: . ack 8316 win 571 <nop,nop,sack 15332:23607 9776:13872> (DF)  
11:39:18.271541 24.220.62.52.8080 > 10.132.84.7.58616: . ack 8600 win 594 <nop,nop,sack 7140:8316 15332:23607 9776:13872> (DF)  
11:39:18.275841 24.220.62.52.8080 > 10.132.84.7.58616: . ack 13872 win 616 <nop,nop,sack 9776:10060 15332:23607> (DF)  
11:39:18.275916 10.132.84.7.58616 > 24.220.62.52.8080: . 13872:15332(1460) ack 76 win 1026 (DF)  
11:39:18.587669 10.132.84.7.58616 > 24.220.62.52.8080: . 13872:15332(1460) ack 76 win 1026 (DF)  
11:39:19.196868 10.132.84.7.58616 > 24.220.62.52.8080: . 13872:15332(1460) ack 76 win 1026 (DF)  
11:39:20.399855 10.132.84.7.58616 > 24.220.62.52.8080: . 13872:15332(1460) ack 76 win 1026 (DF)  
11:39:20.406807 24.220.62.52.8080 > 10.132.84.7.58616: . ack 23607 win 639 (DF)  
11:39:20.406887 truncated-ip 0  
11:39:20.411644 24.220.62.52.8080 > 10.132.84.7.58616: . ack 23607 win 662 <nop,nop,sack 26527:27987> (DF)  
11:39:20.712193 10.132.84.7.58616 > 24.220.62.52.8080: P 23607:25067(1460) ack 76 win 1026 (DF)  
11:39:20.717302 24.220.62.52.8080 > 10.132.84.7.58616: . ack 25067 win 685 <nop,nop,sack 26527:27987> (DF)  
11:39:20.717317 10.132.84.7.58616 > 24.220.62.52.8080: P 27987:29447(1460) ack 76 win 1026 (DF)  
11:39:20.717317 10.132.84.7.58616 > 24.220.62.52.8080: . 29447:30907(1460) ack 76 win 1026 (DF)  
11:39:20.717355 10.132.84.7.58616 > 24.220.62.52.8080: . 25067:26527(1460) ack 76 win 1026 (DF)  
11:39:20.721989 24.220.62.52.8080 > 10.132.84.7.58616: . ack 25067 win 708 <nop,nop,sack 26527:29447> (DF)  
11:39:21.024691 10.132.84.7.58616 > 24.220.62.52.8080: . 25067:26527(1460) ack 76 win 1026 (DF)  
11:39:21.030430 24.220.62.52.8080 > 10.132.84.7.58616: . ack 29447 win 730 (DF)  
11:39:21.030485 10.132.84.7.58616 > 24.220.62.52.8080: . 30907:32367(1460) ack 76 win 1026 (DF)  
11:39:21.030486 10.132.84.7.58616 > 24.220.62.52.8080: P 32367:33827(1460) ack 76 win 1026 (DF)  
11:39:21.030621 truncated-ip 0  
11:39:21.035249 24.220.62.52.8080 > 10.132.84.7.58616: . ack 29447 win 753 <nop,nop,sack 30907:32367> (DF)  
11:39:21.035251 24.220.62.52.8080 > 10.132.84.7.58616: . ack 29447 win 776 <nop,nop,sack 30907:33827> (DF)  
11:39:21.337015 10.132.84.7.58616 > 24.220.62.52.8080: . 29447:30907(1460) ack 76 win 1026 (DF)  
11:39:21.342664 24.220.62.52.8080 > 10.132.84.7.58616: . ack 33827 win 799 (DF)  
11:39:21.342685 truncated-ip 0  
11:39:21.347635 24.220.62.52.8080 > 10.132.84.7.58616: . ack 33827 win 822 <nop,nop,sack 36747:38207> (DF)  
11:39:21.347636 24.220.62.52.8080 > 10.132.84.7.58616: . ack 35287 win 845 <nop,nop,sack 36747:38207> (DF)  
11:39:21.347657 10.132.84.7.58616 > 24.220.62.52.8080: . 38207:39667(1460) ack 76 win 1026 (DF)  
11:39:21.352332 24.220.62.52.8080 > 10.132.84.7.58616: . ack 35287 win 867 <nop,nop,sack 36747:39667> (DF)  
11:39:21.649516 10.132.84.7.58616 > 24.220.62.52.8080: . 35287:36747(1460) ack 76 win 1026 (DF)  
11:39:22.258745 10.132.84.7.58616 > 24.220.62.52.8080: . 35287:36747(1460) ack 76 win 1026 (DF)  
11:39:23.461618 10.132.84.7.58616 > 24.220.62.52.8080: . 35287:36747(1460) ack 76 win 1026 (DF)  
11:39:23.467358 24.220.62.52.8080 > 10.132.84.7.58616: . ack 39667 win 890 (DF)  
11:39:23.467442 truncated-ip 0  
11:39:23.473059 24.220.62.52.8080 > 10.132.84.7.58616: . ack 41127 win 913 (DF)  
11:39:23.473061 24.220.62.52.8080 > 10.132.84.7.58616: . ack 42587 win 936 (DF)  
11:39:23.473062 24.220.62.52.8080 > 10.132.84.7.58616: . ack 44047 win 959 (DF)  
11:39:23.473162 truncated-ip 0  
11:39:23.478092 24.220.62.52.8080 > 10.132.84.7.58616: . ack 45507 win 981 (DF)  
11:39:23.478094 24.220.62.52.8080 > 10.132.84.7.58616: . ack 46967 win 1004 (DF)  
11:39:23.478095 24.220.62.52.8080 > 10.132.84.7.58616: . ack 48427 win 1027 (DF)  
11:39:23.478196 truncated-ip 0  
11:39:23.482649 24.220.62.52.8080 > 10.132.84.7.58616: . ack 49887 win 1050 (DF)  
11:39:23.482661 10.132.84.7.58616 > 24.220.62.52.8080: P 52807:54267(1460) ack 76 win 1026 (DF)  
11:39:23.482757 24.220.62.52.8080 > 10.132.84.7.58616: . ack 51347 win 1073 (DF)  
11:39:23.482767 10.132.84.7.58616 > 24.220.62.52.8080: . 54267:55727(1460) ack 76 win 1026 (DF)  
11:39:23.487275 24.220.62.52.8080 > 10.132.84.7.58616: . ack 51347 win 1095 <nop,nop,sack 54267:55727> (DF)  
11:39:23.789637 10.132.84.7.58616 > 24.220.62.52.8080: . 51347:52807(1460) ack 76 win 1026 (DF)  
11:39:23.794344 24.220.62.52.8080 > 10.132.84.7.58616: . ack 52807 win 1118 <nop,nop,sack 54267:55727> (DF)  
11:39:23.794408 10.132.84.7.58616 > 24.220.62.52.8080: . 55727:57187(1460) ack 76 win 1026 (DF)  
11:39:23.794409 10.132.84.7.58616 > 24.220.62.52.8080: P 57187:58647(1460) ack 76 win 1026 (DF)  
11:39:23.799075 24.220.62.52.8080 > 10.132.84.7.58616: . ack 52807 win 1141 <nop,nop,sack 54267:57187> (DF)  
11:39:23.799077 24.220.62.52.8080 > 10.132.84.7.58616: . ack 52807 win 1164 <nop,nop,sack 54267:58647> (DF)  
11:39:23.799142 10.132.84.7.58616 > 24.220.62.52.8080: P 52807:54267(1460) ack 76 win 1026 (DF)  
11:39:23.803807 24.220.62.52.8080 > 10.132.84.7.58616: . ack 58647 win 1187 (DF)  
11:39:23.803885 truncated-ip 0  
11:39:23.808462 24.220.62.52.8080 > 10.132.84.7.58616: . ack 60107 win 1210 (DF)  
11:39:23.808464 24.220.62.52.8080 > 10.132.84.7.58616: . ack 61567 win 1232 (DF)  
11:39:23.808465 24.220.62.52.8080 > 10.132.84.7.58616: . ack 63027 win 1255 (DF)  
11:39:23.808560 truncated-ip 0  
11:39:23.813176 24.220.62.52.8080 > 10.132.84.7.58616: . ack 64487 win 1278 (DF)  
11:39:23.813178 24.220.62.52.8080 > 10.132.84.7.58616: . ack 64487 win 1301 <nop,nop,sack 65947:67407> (DF)  
11:39:23.813244 10.132.84.7.58616 > 24.220.62.52.8080: . 67407:68867(1460) ack 76 win 1026 (DF)  
11:39:23.817858 24.220.62.52.8080 > 10.132.84.7.58616: . ack 64487 win 1324 <nop,nop,sack 65947:68867> (DF)  
11:39:24.117727 10.132.84.7.58616 > 24.220.62.52.8080: P 64487:65947(1460) ack 76 win 1026 (DF)  
11:39:24.122438 24.220.62.52.8080 > 10.132.84.7.58616: . ack 68867 win 1346 (DF)  
11:39:24.122518 truncated-ip 0  
11:39:24.127207 24.220.62.52.8080 > 10.132.84.7.58616: . ack 70327 win 1369 (DF)  
11:39:24.127209 24.220.62.52.8080 > 10.132.84.7.58616: . ack 71787 win 1392 (DF)  
11:39:24.127295 truncated-ip 0  
11:39:24.132061 24.220.62.52.8080 > 10.132.84.7.58616: . ack 73247 win 1415 (DF)  
11:39:24.132063 24.220.62.52.8080 > 10.132.84.7.58616: . ack 74707 win 1432 (DF)  
11:39:24.132153 truncated-ip 0  
11:39:24.136834 24.220.62.52.8080 > 10.132.84.7.58616: . ack 76167 win 1432 (DF)  
11:39:24.136836 24.220.62.52.8080 > 10.132.84.7.58616: . ack 77627 win 1432 (DF)  
11:39:24.136910 truncated-ip 0  
11:39:24.141681 24.220.62.52.8080 > 10.132.84.7.58616: . ack 79087 win 1432 (DF)  
11:39:24.141683 24.220.62.52.8080 > 10.132.84.7.58616: . ack 80547 win 1432 (DF)  
11:39:24.141684 24.220.62.52.8080 > 10.132.84.7.58616: . ack 82007 win 1432 (DF)  
11:39:24.141779 truncated-ip 0  
11:39:24.146350 24.220.62.52.8080 > 10.132.84.7.58616: . ack 83467 win 1432 (DF)  
11:39:24.146352 24.220.62.52.8080 > 10.132.84.7.58616: . ack 83467 win 1432 <nop,nop,sack 84927:86387> (DF)  
11:39:24.146353 24.220.62.52.8080 > 10.132.84.7.58616: . ack 83467 win 1432 <nop,nop,sack 84927:87847> (DF)  
11:39:24.146429 10.132.84.7.58616 > 24.220.62.52.8080: . 87847:89307(1460) ack 76 win 1026 (DF)  
11:39:24.146431 10.132.84.7.58616 > 24.220.62.52.8080: P 89307:90062(755) ack 76 win 1026 (DF)  
11:39:24.146478 10.132.84.7.58616 > 24.220.62.52.8080: . 90062:91522(1460) ack 76 win 1026 (DF)  
11:39:24.146498 10.132.84.7.58616 > 24.220.62.52.8080: P 91522:92982(1460) ack 76 win 1026 (DF)  
11:39:24.151151 24.220.62.52.8080 > 10.132.84.7.58616: . ack 83467 win 1432 <nop,nop,sack 84927:89307> (DF)  
11:39:24.151153 24.220.62.52.8080 > 10.132.84.7.58616: . ack 83467 win 1432 <nop,nop,sack 84927:90062> (DF)  
11:39:24.151154 24.220.62.52.8080 > 10.132.84.7.58616: . ack 83467 win 1432 <nop,nop,sack 84927:91522> (DF)  
11:39:24.151155 24.220.62.52.8080 > 10.132.84.7.58616: . ack 83467 win 1432 <nop,nop,sack 84927:92982> (DF)  
11:39:24.151221 10.132.84.7.58616 > 24.220.62.52.8080: . 83467:84927(1460) ack 76 win 1026 (DF)  
11:39:24.151249 10.132.84.7.58616 > 24.220.62.52.8080: . 92982:94442(1460) ack 76 win 1026 (DF)  
11:39:24.151267 10.132.84.7.58616 > 24.220.62.52.8080: P 94442:95902(1460) ack 76 win 1026 (DF)  
11:39:24.151284 10.132.84.7.58616 > 24.220.62.52.8080: . 95902:97362(1460) ack 76 win 1026 (DF)  
11:39:24.155914 24.220.62.52.8080 > 10.132.84.7.58616: . ack 83467 win 1432 <nop,nop,sack 84927:94442> (DF)  
11:39:24.155916 24.220.62.52.8080 > 10.132.84.7.58616: . ack 83467 win 1432 <nop,nop,sack 84927:95902> (DF)  
11:39:24.155917 24.220.62.52.8080 > 10.132.84.7.58616: . ack 83467 win 1432 <nop,nop,sack 84927:97362> (DF)  
11:39:24.155978 10.132.84.7.58616 > 24.220.62.52.8080: P 97362:98822(1460) ack 76 win 1026 (DF)  
11:39:24.156004 10.132.84.7.58616 > 24.220.62.52.8080: . 98822:100282(1460) ack 76 win 1026 (DF)  
11:39:24.156022 10.132.84.7.58616 > 24.220.62.52.8080: P 100282:101742(1460) ack 76 win 1026 (DF)  
11:39:24.160678 24.220.62.52.8080 > 10.132.84.7.58616: . ack 83467 win 1432 <nop,nop,sack 84927:98822> (DF)  
11:39:24.160680 24.220.62.52.8080 > 10.132.84.7.58616: . ack 83467 win 1432 <nop,nop,sack 84927:100282> (DF)  
11:39:24.160681 24.220.62.52.8080 > 10.132.84.7.58616: . ack 83467 win 1432 <nop,nop,sack 84927:101742> (DF)  
11:39:24.461359 10.132.84.7.58616 > 24.220.62.52.8080: . 83467:84927(1460) ack 76 win 1026 (DF)  
11:39:24.466014 24.220.62.52.8080 > 10.132.84.7.58616: . ack 101742 win 1352 (DF)  
11:39:24.466110 10.132.84.7.58616 > 24.220.62.52.8080: FP 101742:102690(948) ack 76 win 1026 (DF)  
11:39:24.470981 24.220.62.52.8080 > 10.132.84.7.58616: F 76:76(0) ack 102691 win 1432 (DF)  
11:39:24.471048 10.132.84.7.58616 > 24.220.62.52.8080: . ack 77 win 1026 (DF)

Of interest here are the "truncated-ip 0" packets, the "sack" bearing "acks" and the "missing" data that is subsequently acknowledged without retransmission.

The "truncated-ip 0" packets are packets with an IP length of 0; in the corresponding TCPIP ETW data, these packets are accompanied by TcpLso (TCP Large Send Offload) events. The length of the IP data is conveyed to the network interface via other mechanisms (the IP checksum is also zero).

The packets created by the network interface from the large blob of IP data and then lost/dropped seem to be confusing the host (Windows) based congestion control algorithm.

There is much more interesting information/details that can be mined from the trace data, but the current "prime suspect" for the poor performance is the "Large Send Offload" functionality.

Gary

0 comments

Gary Nebbett 6,536 Reputation points

2020-11-27T16:30:23.2+00:00
Hello @Egoist ,

I am using a new "answer" to reply due to the very short limit on comment length.

By the "nominal" speed for upload, I mean the value specified by your ISP. I have a 250 Mbps download package and the ISP says that for all of its packages the upload speed is one tenth of the download speed - meaning 25 Mbps for me and I measure 24.6 Mbps (i.e. I can actually use the contracted speed).

Measuring 0.77 Mbps under Windows and 600+ Mbps under Clear Linux is surprising on two counts: the massive difference and the massive actual value obtainable with Clear Linux. What is your nominal download/upload speed, as mentioned by the ISP?

The first few packets of the network trace will show the TCP option conveying the MSS and window scaling values:
```
11:09:16.859537 192.168.0.10.63097 > 195.49.6.4.8080: S 2954490598:2954490598(0) win 64240 <mss 1460,nop,wscale 8,nop,nop,sackOK> (DF)  
11:09:16.874456 195.49.6.4.8080 > 192.168.0.10.63097: S 1089487300:1089487300(0) ack 2954490599 win 29200 <mss 1460,nop,nop,sackOK,nop,wscale 7> (DF)  
```
Later packets in the trace will show the actual IP packet sizes used.
```
11:09:16.964414 192.168.0.10.63097 > 195.49.6.4.8080: . 1906:3366(1460) ack 869 win 509 (DF)  
11:09:16.964416 192.168.0.10.63097 > 195.49.6.4.8080: . 3366:4826(1460) ack 869 win 509 (DF)  
11:09:16.964418 192.168.0.10.63097 > 195.49.6.4.8080: . 4826:6286(1460) ack 869 win 509 (DF)  
11:09:16.964419 192.168.0.10.63097 > 195.49.6.4.8080: . 6286:7746(1460) ack 869 win 509 (DF)  
11:09:16.964420 192.168.0.10.63097 > 195.49.6.4.8080: . 7746:9206(1460) ack 869 win 509 (DF)  
11:09:16.964421 192.168.0.10.63097 > 195.49.6.4.8080: . 9206:10666(1460) ack 869 win 509 (DF)  
```
In that second TCP Connection Summary, there were only 68 DataSegmentsOut, seemingly carrying 154,383 bytes - it should not be too much work to check these values by looking at the detailed trace data. Understanding mysteries like why the average output segment size is (or seems) so large will probably be essential to understand the performance behaviour.

Only once we understand the behaviour can we start the search for configuration settings that could potentially change the behaviour for the better.

Gary
Was this answer helpful?
Egoist 1 Reputation point

2020-11-27T18:02:59.71+00:00

Apologizes, I have been trying to get similar view as the ones you have for the network trace, but i don't seem to be able to find them. Is this using tcpdump or the same .etl network capture file?
What tool are you using to analyze the dump file?

I am using College network, which i believe would be 1 gigabit down/up, in which i don't have control of the router of course.

Since Clear Linux have no problem uploading big files, here is the etl file that i compressed if you are interested

view

Gary Nebbett 6,536 Reputation points

2020-11-27T19:55:17.07+00:00

Hello @Egoist ,

Using the PowerShell commands that I mentioned earlier, the .etl file contains all of the information. A good tool to analyze the mixed data in the trace file (raw network data plus additional ETW providers) would be Microsoft Message Analyzer, but unfortunately this tool has been "discontinued" by Microsoft.

There are tools that convert between common network capture formats (.pcap, .pcapng, .cap, .etl) - WireShark includes such a tool (not sure about .etl support). I can (and have) develop tools myself and normally use these, so I am not sure about details of openly available software. For some purposes, I find it most powerful to use command-line/text based tools (such as tcpdump) and, if necessary, I just convert the formats (rather than using better known GUI tools).

Thanks for providing the raw data. I have downloaded it and will provide feedback as quickly as possible (it is 21:00 here in Switzerland, so it make take until tomorrow).

Gary
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Slow wired upload speed vs Linux on same hardware

50 answers

Your answer