Driver r8169 causes very slow download speed due to ASPM

[Sriram R]

Running Ubuntu Server 21.04 (kernel 5.11.0-34-generic) and have run into a serious network issue where the internet download speed is around 25Mbps on a 150Mbps connection!

My setup:

    Lenovo Ideapad 300S with a wired Ethernet connection.
    Running Ubuntu Server 21.04 (kernel version 5.11.0-34-generic).
    Wired Ethernet connection with RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (r8169) driver.
    My contract with the ISP is for a 150Mbps download speed.

My problem:

The download speed is extremely low, around 25Mbps whereas my upload speed is 'normal' around 80-90Mbps. I use 'speedtest-cli' for measuring the bandwidth, by the way.

This problem does not happen on an Acer Aspire One laptop running the same kernel and connected to the same router.

After scouring through countless articles, I noticed that there were lot of issues with the r8169 driver and the RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller subsystem.

What eventually fixed the issue for me was setting the "pcie_aspm=off" kernel parameter during boot. But I strongly feel that is a 'workaround' and not a solution to the real problem, which is the r8169 driver has regression with newer kernels, hence filing this bug.

Thanks in advance for all your support! I'm a big fan of Ubuntu!!!

ProblemType: Bug
DistroRelease: Ubuntu 21.04
Package: linux-image-5.11.0-34-generic 5.11.0-34.36
ProcVersionSignature: Ubuntu 5.11.0-34.36-generic 5.11.22
Uname: Linux 5.11.0-34-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version k5.11.0-34-generic.
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu65.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/hwC0D2', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D10p', '/dev/snd/pcmC0D9p', '/dev/snd/pcmC0D8p', '/dev/snd/pcmC0D7p', '/dev/snd/pcmC0D3p', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Card0.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card0.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
CasperMD5CheckResult: pass
Date: Tue Sep 14 12:17:49 2021
InstallationDate: Installed on 2021-09-05 (9 days ago)
InstallationMedia: Ubuntu-Server 21.04 "Hirsute Hippo" - Release amd64 (20210421)
MachineType: LENOVO 80Q4
ProcEnviron:
 SHELL=/bin/bash
 LANG=en_US.UTF-8
 TERM=xterm-256color
 PATH=(custom, no user)
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.11.0-34-generic root=UUID=b3938b06-c0f7-4342-b99c-94888f030a99 ro pcie_aspm=off
RelatedPackageVersions:
 linux-restricted-modules-5.11.0-34-generic N/A
 linux-backports-modules-5.11.0-34-generic N/A
 linux-firmware 1.197.3
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/26/2018
dmi.bios.release: 1.39
dmi.bios.vendor: LENOVO
dmi.bios.version: D3CN39WW
dmi.board.asset.tag: No Asset Tag
dmi.board.name: Lenovo ideapad 300S-14ISK
dmi.board.vendor: LENOVO
dmi.board.version: SDK0J40709 WIN
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Lenovo ideapad 300S-14ISK
dmi.ec.firmware.release: 1.26
dmi.modalias: dmi:bvnLENOVO:bvrD3CN39WW:bd06/26/2018:br1.39:efr1.26:svnLENOVO:pn80Q4:pvrLenovoideapad300S-14ISK:skuLENOVO_MT_80Q4_BU_idea_FM_Lenovoideapad300S-14ISK:rvnLENOVO:rnLenovoideapad300S-14ISK:rvrSDK0J40709WIN:cvnLENOVO:ct10:cvrLenovoideapad300S-14ISK:
dmi.product.family: IDEAPAD
dmi.product.name: 80Q4
dmi.product.sku: LENOVO_MT_80Q4_BU_idea_FM_Lenovo ideapad 300S-14ISK
dmi.product.version: Lenovo ideapad 300S-14ISK
dmi.sys.vendor: LENOVO

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[AaronMa (mapengyu)]

Could you try upstream kenrel 5.14 and 5.15-rc1?

If no reproduce we should disable ASPM L1 in Ubuntu kernel too.

[Kai-Heng Feng (kaihengfeng)]

Can you please give OEM kernel a try? It keeps ASPM L1 enabled but disable ASPM L1.1/L1.2.

[Kai-Heng Feng (kaihengfeng)]

$ sudo apt install linux-oem-20.04c

[Sriram R (quarksrus)]

@kaihengfeng

apt install linux-oem-20.04c
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-oem-20.04c
E: Couldn't find any package by glob 'linux-oem-20.04c'

Any specific repo I need to have for this package?

[Sriram R (quarksrus)]

@mapengyu

5.14 and 5.15-rc1 kernels, for these upstream ones do I need to do an upgrade my system to satisfy dependencies or will these newer kernels install without any issues?

[Kai-Heng Feng (kaihengfeng)]

Please add "focal-updates" to /etc/apt/sources.list and try to install it again.

[Sriram R (quarksrus)]

@kaihengfeng added the following to /etc/apt/sources.list but still no luck!

deb http://archive.ubuntu.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ focal-updates main restricted universe multiverse

apt install linux-oem-20.04c
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-oem-20.04c
E: Couldn't find any package by glob 'linux-oem-20.04c'

[Kai-Heng Feng (kaihengfeng)]

$ sudo apt update
Before `apt install`

[Sriram R (quarksrus)]

Not home right now, but I'll try this first thing tomorrow and will let you know.

One more update, it seems the problem (slow download speed) returns if the server runs for ~10hrs or so!

[Sriram R (quarksrus)]

Ok, installed and booted off the oem kernel without the pcie_asm kernel parameter and the speedtest-cli results are promising.

uname -r
5.13.0-1012-oem

Output from 'speedtest-cli':

Retrieving speedtest.net configuration...
Testing from XXXXXX (XXXXXXXX)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Totalplay (Toluca) [249.28 km]: 16.321 ms
Testing download speed................................................................................
Download: 197.65 Mbit/s
Testing upload speed......................................................................................................
Upload: 105.19 Mbit/s

And output from 'ethtool -S enp3s0' doesn't show any rx_missed packets.

ethtool -S enp3s0
NIC statistics:
     tx_packets: 351099
     rx_packets: 445611
     tx_errors: 0
     rx_errors: 0
     rx_missed: 0
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 445185
     broadcast: 276
     multicast: 150
     tx_aborted: 0
     tx_underrun: 0

Output from 'ethtool enp3s0':

ethtool enp3s0
Settings for enp3s0:
        Supported ports: [ TP MII ]
        Supported link modes: 10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes: 10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes: 100baseT/Half 100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        master-slave cfg: preferred slave
        master-slave status: master
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: external
        MDI-X: Unknown
        Supports Wake-on: pumbg
        Wake-on: d
        Link detected: yes

Output from lspci -vnvn -s 03:00.0:

03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
        Subsystem: Lenovo RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [17aa:3835]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 17
        Region 0: I/O ports at d000 [size=256]
        Region 2: Memory at d1204000 (64-bit, non-prefetchable) [size=4K]
        Region 4: Memory at d1200000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: ...

[Sriram R (quarksrus)]

Hi,

Please let me know if you want me to test additional configs. Not sure if this bug will be addressed in the next release?

Thanks,
Sriram

[Alex Hung (alexhung)]

It seems a SRU was sent to Ubuntu kernel and the status is available @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1946433

So it seems a fix will land in impish later.

[Sriram R (quarksrus)]

Excellent news! Since both Hirsute and Impish EOL next year, would it be better to wait for JJ before doing the upgrade?