Slow wired upload speed vs Linux on same hardware

Dawid Oosthuizen 26

Intel® Ethernet Controller X550-AT2, 10G network interface on ASRock Rack ROMED8-2T with AMD EPYC 7232P processor.
Windows 10 Pro for Workstations, 2004.
Latest Windows updates and Intel drivers installed as of 09/09/2020.

This machine is a dual boot, the above Windows version, and Ubuntu 20.04.
When doing a speed test, I get good performance from Ubuntu, but very poor uploads from Windows. This is on the same machine, the exact same hardware.

The WAN link is 1000Mbps down, 50Mbps up.

This is the Windows speedtest result:

This is the Ubuntu speedtest result:

I have tried to tweak the adapter's advanced driver settings in Windows, such as disabling LSO, etc. No luck, performance remains poor.

I've also noticed it on another PC running Windows 10 Pro, and a laptop running Windows 10 Pro for Workstations, both give the same poor upload performance. Whereas my other Ubuntu 20.04 Server machine, and also my phone connected via Wi-Fi, is getting good upload speeds.

I have even taken the Windows laptop and plugged it straight into my incoming WAN connection (bypassing router), and it still gets poor upload speeds.

Incidentally, when the speed test is running, I can see that the upload looks bursty on Windows, like it is only getting chunks of data here and there, while in Linux and on Android it looks the same as the download, the graph is drawn at a consistent high rate and with consistent high values.

franco d'esaro 25,181 Reputation points Volunteer Moderator

2020-09-09T12:47:10.917+00:00

Hello,
Firewall / Antivirus in use on Windows machines?
Try running and maintaining a Clean Boot on Windows machines:
https://support.microsoft.com/en-us/help/929135/how-to-perform-a-clean-boot-in-windows
Dawid Oosthuizen 26 Reputation points

2020-09-10T01:19:34.907+00:00
Thanks for the suggestions, unfortunately no luck:

I'm using Microsoft's Windows Defender, not a third party product. And disabling Defender (virus protection and firewall) had no effect.

Also did a clean boot (only MS services enabled), no effect.

I've added some more info in my original post, FWIW.
mensa84 6 Reputation points

2020-11-12T00:22:07.743+00:00

Did you find a solution? I habe exactly the same problem :(
Guery Illanes 1 Reputation point

2021-07-08T23:42:19.677+00:00

Hi,
Try with PDANET software. Must be installed in laptop with Windows and also in mobile. It works fine with this issue.

Guery
Guery Illanes 1 Reputation point

2021-07-08T23:42:45.823+00:00

Hi,
Try with PDANET software. Must be installed in laptop with Windows and also in mobile. It works fine with this issue.

Guery
@Live MiaoSKY 0 Reputation points

2025-03-31T01:57:03.21+00:00

This problem has been bothering me for a long time, and I finally found out that it was a problem with speedtest.net. It limits the user-agent of edge. If you use the user-agent of edge, the upload speed is only about 5mbps.

After I used the extension to change it to the user-agent of chrome, the upload speed went up.... Fuck speedtest.net!!!

这个问题让我头疼很久, 最后发现是speedtest.net的问题. 它限制了edge的user-agent, 如果你使用edge的user-agent, 上传速度就只有5mbps左右.

在我使用扩展插件修改成chrome的user-agent以后上传速度就上去了.... 真是fuck speedtest.net !!!

50 answers

Gary Nebbett 6,536 Reputation points

2020-12-24T13:31:04.887+00:00

Hello @mensa84 ,

Thanks for trying.

I am coming to the opinion that there is no settable option to improve the situation - functionality seems to be missing from the Microsoft TCP/IP implementation.

The functionality in question is RACK ("Recent ACKnowledgment", The RACK-TLP loss detection algorithm for TCP, https://datatracker.ietf.org/doc/draft-ietf-tcpm-rack/?include_text=1). The RFC for this functionality is still in draft status - I had been looking at draft 14 and I just noticed that draft 15 was published 2 days ago (2020-12-22). RACK in the Windows TCP/IP implementation is mentioned in this c't article from 2016 (the date of the first draft): https://www.heise.de/select/ct/2016/17/1471609544700372

It would be useful to try to identify which device is responsible for the high degree of out-of-order delivery. When I trace the route to the speedtest server that you used, this is what I see:

tracert -w 200 84.116.34.253

Tracing route to 84.116.34.253 over a maximum of 30 hops

1 5 ms 3 ms 3 ms 192.168.0.1
2 * * * Request timed out.
3 10 ms 14 ms 13 ms riehen-sw1-po-1.gw.imp.ch [157.161.254.101]
4 24 ms 11 ms 10 ms bsl-mes-sw1-vlan-2140.gw.imp.ch [157.161.251.110]
5 12 ms 11 ms 13 ms bsl-dsp-sw1-vlan-2100.gw.imp.ch [157.161.251.42]
6 10 ms 10 ms 15 ms prt-imp-sw7-vlan-2302.gw.imp.ch [157.161.251.50]
7 11 ms 14 ms 26 ms prt-hea-sw1-vlan-2303.gw.imp.ch [157.161.251.53]
8 63 ms 48 ms 31 ms prt-cbl-sw1-vlan-2003.gw.imp.ch [157.161.251.9]
9 11 ms 11 ms 11 ms prt-cbl-core2-ve-3020.gw.imp.ch [157.161.254.153]
10 10 ms 11 ms 12 ms te0-0-2-1.nr11.b021978-0.bsl01.atlas.cogentco.com [149.6.34.41]
11 11 ms 10 ms 11 ms te0-0-2-3.rcr11.bsl01.atlas.cogentco.com [154.25.4.221]
12 13 ms 11 ms 14 ms te0-2-0-1.ccr51.zrh02.atlas.cogentco.com [130.117.2.145]
13 13 ms 12 ms 12 ms liberty.zrh01.atlas.cogentco.com [130.117.14.230]
14 29 ms 32 ms 30 ms ch-zrh03a-rc1-ae-9-0.aorta.net [84.116.134.141]
15 44 ms 43 ms 30 ms de-fra02a-rc1-ae-27-0.aorta.net [84.116.132.1]
16 28 ms 29 ms 28 ms at-vie01b-rc2-ae-3-0.aorta.net [84.116.136.117]
17 31 ms * 28 ms at-vie01b-rc1-ae-41-0.aorta.net [84.116.130.26]
18 29 ms 28 ms 30 ms at-vie01a-ra5-dc-ae-1-306.aorta.net [84.116.138.33]
19 28 ms 28 ms 27 ms 84.116.34.253

How many hops are their from your system to the first common hop that we share on the path to the speedtest server?

Gary
Was this answer helpful?

1 person found this answer helpful.
mensa84 6 Reputation points

2021-01-03T15:45:45.217+00:00

How may I find that out?
I don't remember which server I used on speedtest.net

My internet connection is through 5G if that matters?

Gary Nebbett 6,536 Reputation points

2021-01-03T16:13:24.69+00:00

Hello @mensa84 ,

The IP address of the speedtest sever that you used is the same that I used in the command "tracert -w 200 84.116.34.253" - that very same command is the one that you could use.

I suspect that the network component that causes the reordering is topologically (very) close to you, so this test does not help much - I only suggest trying it in the sense of "leaving no stone unturned" (i.e. trying everything that might possibly (even indirectly) help)...

Gary

mensa84 6 Reputation points

2021-01-03T16:31:10.527+00:00

Thanks, that's the output:

tracert -w 200 84.116.34.253

Routenverfolgung zu 84.116.34.253 über maximal 30 Hops

1 <1 ms <1 ms <1 ms unifi.localdomain [192.168.1.1]
2 3 ms 2 ms 29 ms 192.168.165.1
3 20 ms 41 ms 42 ms 172.20.162.37
4 23 ms 49 ms 45 ms 172.17.13.96
5 * * * Zeitüberschreitung der Anforderung.
6 27 ms 40 ms 40 ms 213.94.72.88
7 15 ms 71 ms 29 ms 195.3.114.193
8 22 ms 36 ms 18 ms lg4-9073.as8447.a1.net [195.3.64.141]
9 * * * Zeitüberschreitung der Anforderung.
10 50 ms 30 ms 46 ms at-vie05d-ri1-ae-1-0.aorta.net [213.46.184.61]
11 44 ms 32 ms 34 ms at-vie05d-rc1-ae-12-0.aorta.net [213.46.173.126]
12 66 ms 34 ms 78 ms at-vie01a-ra6-dc-ae-1-307.aorta.net [84.116.138.37]
13 58 ms 29 ms * 84.116.34.253
14 65 ms 51 ms 44 ms 84.116.34.253
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
mensa84 6 Reputation points

2020-12-24T13:01:43.367+00:00

Thx, but it did not change anything :(
Was this answer helpful?

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Gary Nebbett 6,536 Reputation points

2020-12-21T15:30:07.973+00:00

Hello @Moosa Mahsoom ,

That is interesting, the new graph of the upload connection looks like this:

It is (surprisingly) similar to the RTMP graph. One odd thing is that this time, the "congestion avoidance" algorithm is Compound TCP (CTCP) rather than CUBIC. Have you been experimenting with commands like "netsh int tcp set supplemental template=internet congestionprovider=ctcp"?

The reason for the very slow adjustment of the congestion window will need some more time to investigate, but it would be very helpful to know why the congestion provider seems to have changed...

Gary
Was this answer helpful?
Moosa Mahsoom 1 Reputation point

2020-12-21T15:32:16.493+00:00

As far as I can remember, i haven't changed anything related to the particular parameter. I did try changing things like the auto tuning level, max syn retransmissions.

Gary Nebbett 6,536 Reputation points

2020-12-21T15:38:00.567+00:00

Hello @Moosa Mahsoom ,

Can you please show the current output of the following two commands and/or the PowerShell cmdlet Get-NetTCPSetting?

netsh int tcp show global
netsh int tcp show supplemental

Gary

Moosa Mahsoom 1 Reputation point

2020-12-21T15:39:47.25+00:00

50052-tcp.txt

Have attached the output.

Moosa Mahsoom 1 Reputation point

2020-12-21T15:53:43.523+00:00

I ran reset

netsh int tcp reset

and collected the trace again with similar results. (Have confirmed that the congestion provider is now cubic)

Based on what you've said and my own testing across my home network. It looks like it's due to a lossy fibre (Perhaps a faulty transmitter??) channel to the ISP. Linux devices seems to be handling this situation better. I will contact the ISP and let them have a look at the connection I suppose.

https://1drv.ms/u/s!Ag3GVFPGmMEa9Umo6CBNuqrZZqpL?e=smyO9F

Gary Nebbett 6,536 Reputation points

2020-12-21T15:57:09.65+00:00

Hello @Moosa Mahsoom ,

That's odd. Comparing your output to mine, we differ for each template on the InitialRto, MaxSynRetransmissions and DynamicPortRange (not a big deal), but we also differ on the CongestionProvider in the "Internet" template - I have CUBIC and you have CTCP.

Could you try issuing the following PowerShell command and then checking whether it changes the output of Get-NetTCPSetting?

Set-NetTCPSetting -SettingName "Internet" -CongestionProvider CUBIC

If it does change the output, it might be worthwhile retrying the speed test (with network tracing, as before)...

Gary

Moosa Mahsoom 1 Reputation point

2020-12-21T15:59:59.04+00:00

@Gary Nebbett , I have already reset all the settings to the Windows Defaults. (can confirm now all the parameters are identical to what you've posted). But, the issue is persisting.

I have posted the Network Trace above in the same thread.

Gary Nebbett 6,536 Reputation points

2020-12-21T16:17:04.413+00:00

Hello @Moosa Mahsoom ,

There are a number of reasons why packets might be lost. Perhaps the most important is that packets are discarded by intermediate devices on the route to the target because they can't forward the incoming traffic as fast as it is arriving; if the devices just buffered all packets until they could send them, then they would require unlimited memory and packets could arrive at their target a very long time after they were sent. Many of the "congestion avoidance" algorithms work on the idea of sending as many packets as quickly as possible until they get feedback, via lost packet detection mechanisms, that they are sending data faster than it can be forwarded.

ISPs will also limit the throughput to the contractually agreed limit by the same mechanism (discarding packets when they arrive faster than contracted).

Another possibility is faulty cabling or hardware, but it is not the most likely cause...

Gary

Moosa Mahsoom 1 Reputation point

2020-12-21T16:19:19.047+00:00

I did try connecting directly to the device from my Internet Service Provider and the issue persisted. Thanks for the help. Definitely helped me understand the issue better. I'll let them have a look at the connection and let you know the results.

Gary Nebbett 6,536 Reputation points

2020-12-21T16:36:24.107+00:00

Hello @Moosa Mahsoom ,

Here is the graph for the CUBIC speed test - it is not good:

In one of my last messages to Egoist-7680 in this thread, I posted a graph comparing throughput to packet loss for CUBIC vs BBR. Your packet loss rate appears to be in the 5 to 10 percent range, which the graph shows is a bad place to be if one is using CUBIC.

Gary

Moosa Mahsoom 1 Reputation point

2020-12-21T16:38:02.953+00:00

Hi @Gary Nebbett , thanks for all the help. I just found something in my modem that could be the route cause. the Rx Power of the Laser Device has dropped by 10 dB. which could explain the packet loss.

Moosa Mahsoom 1 Reputation point

2020-12-22T07:16:49.227+00:00

@Gary Nebbett the issue has been resolved. One of the fibre boxes in my locality was faulty resulting in the packet losses. Thank You for your help.

Gary Nebbett 6,536 Reputation points

2020-12-22T09:05:37.363+00:00

Hello @Moosa Mahsoom ,

Thanks for letting me know. I normally think of packet loss due to hardware/cables outside off my home as a "fact of life" that I just have to accept. Good to know that sometimes something can be done.

Gary
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Gary Nebbett 6,536 Reputation points

2020-12-21T13:49:04.93+00:00

Hello @Moosa Mahsoom ,

The behaviour of your connection appears to be more similar to that of Egoist-7680. In this case, the path is "lossy" (i.e. packets are lost en route) and the "congestion avoidance" algorithm used by Windows ("CUBIC", in this case) does not match the throughput obtainable with other congestion avoidance algorithms used by some other systems (e.g. "BBR"). There are open questions about the network "fairness" of algorithms such as "BBR"; in any event, "CUBIC" is the best algorithm currently natively available under Windows.

Passing your connection through an additional device can only worsen the problem; it is the "end-user" device (Windows) that chooses the algorithm and makes the key decisions about windows sizes, retransmissions, etc..

I checked many of the retransmissions in the trace and they were all appropriate (the retransmitted data was not acknowledged until at least one round-trip time had elapsed since the retransmission) - this is the big difference to the behaviour observed by mensa84-6815, where retransmissions occur after the data has been acknowledged (but before the acknowledgement has been "processed").

Your trace data was of a "useful" network connection using a higher level protocol (Real-Time Messaging Protocol (RTMP) ) rather than a straightforward speed test. In RTMP, RTMP protocol interactions do occur and this has some impact on the perceived throughput. This is a graph of the throughput from client to server (the darker line is the data throughput, the lighter line is the receive window boundary):

Over the lifetime of the connection, the average throughput was 180 kilobytes/second or 1.5 megabits/second; however, near the end of the connection, a throughput of 615 kilobytes/second or 5 megabits/second was sustained for a short time. Since we truncated the capture length so that not much more than the TCP headers are captured, it is difficult to say what impact higher level (RTMP) protocol interactions had on the throughput.

Sorry, but I don't have any good ideas about what could be done to improve the situation.

Gary
Was this answer helpful?
Moosa Mahsoom 1 Reputation point

2020-12-21T14:01:06.357+00:00

Hi @Gary Nebbett , i collected a trace with SpeedTest this time. Could you check if the behavior matches with the OBS use case. I did try plugging my device directly to my ONT device. So, It's quite possible it's lossy in the fiber channel (I am not very well-versed with networking so sorry if I completely mess up my terms).

https://1drv.ms/u/s!Ag3GVFPGmMEa9Ugqzii9KwqqnX4B?e=NiEXpo

Moosa Mahsoom 1 Reputation point

2020-12-21T15:13:14.063+00:00

I connected a Windows 10 device directly to the ONT device. And the issue persisted. So, it's mostly an issue with the fibre channel being lossy.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Moosa Mahsoom 1 Reputation point

2020-12-20T21:12:51.853+00:00

@Gary Nebbett if I am having the same issue (I assume I am because it does look like a very similar case to me but I'll post my dumps tomorrow to confirm), this effects workloads such as streaming to YouTube (using OBS Studio)

I was streaming to YouTube in the morning and the bit rate completely drops. I can confirm the issue happens with all windows devices on my network.
Was this answer helpful?
Gary Nebbett 6,536 Reputation points

2020-12-20T21:31:23.137+00:00

Hello @Moosa Mahsoom ,

I will check your traces if you post them.

I hope however, that I posted enough information for people/companies who/that are comfortable analysing network traces so that they can recognize themselves if their problem probably has the same cause.

@anyone: If not then please let me know.

The big problem of finding a workaround still remains...

Gary

Moosa Mahsoom 1 Reputation point

2020-12-21T06:01:18.683+00:00

Hi Gary Nebbet,
I have provided the trace while running an instance of OBS.

You can see the bitrate on the bottom right which is significantly less than what i usually get (10k)

Attaching the SlowUp.etl -> https://1drv.ms/u/s!Ag3GVFPGmMEa9Udt5vaVpU-qD1k1?e=oFIaEq

Moosa Mahsoom 1 Reputation point

2020-12-21T06:03:32.723+00:00

I did an additional test which was I passed the connection through my Android phone via it's hotspot functionality and the issue persisted. Initially, I thought this could be because of the network configuration of my house since it comes through an unmanaged switch + router from the ONT. But, even when I connected directly to the ONT, the issue did exist. So, seems to be something related to how the ISP is handling uploads.

Moosa Mahsoom 1 Reputation point

2020-12-21T06:04:03.91+00:00

I would say all devices are affected since my Linux devices have also dropped in speed but Windows has had a bigger impact.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Slow wired upload speed vs Linux on same hardware

50 answers

Your answer