e1000e extremly slow

[hendrik]

[Impact]
e1000e on TGP PCH encounters a network speed issue that rx speed is below 1MB/s

[Fix]
Intel provides 2 patches to fix this
https://patchwork.ozlabs.org/project/intel-wired-lan/list/?series=263466

[Test]
Verified by our Dell and Lenovo platforms and the bug reporter.

[Where problems could occur]
The fix only applies to e1000e with TGP PCH, and add a flag "E1000_FFLT_DBG_DONT_GATE_WAKE_DMA_CLK". The impact should be minimized and should not lead to other regressions.

=============================

We have dell latitude 5420 & 5520 laptops with onboard intel ethernet controller.
Problem is that the ethernet port is extremely slow. about 100kbyte/s download throughput while on a gigabit connection. Upload seems to work just fine.

Also when I pxe boot the system it's also painfully slow.

But here comes the funny part. When I attach a usb memory stick the throughput of the networkcontroller is as expected.

I've used latest ubuntu 20.04.2 kernel 5.8.0-55
I've used latest ubuntu 20.04.2 kernel 5.10.0-1029-oem

Turns out this problem is not showing up when I manually install the 5.8.18 kernel.

Also when I create my own iso with the help of Cubic and install the 5.8.18 kernel in it the pxe boot is working as expected with high throughput

system: Dell latitude 5420 + 5520
nic: Ethernet Connection (13) I219-LM
ubuntu 20.04.2
module: e1000e

Please assist. Thank you

[hendrik (hendrikp)]

Issue also exists on 5.6.0-1032-oem. Which is certified.

https://certification.ubuntu.com/hardware/202010-28335

[hendrik (hendrikp)]

To whom is may concern. I found the workaround for this.

The trick is to set the boot kernel parameter "pcie_aspm=off" in '/etc/default/grub'

Like this:

GRUB_CMDLINE_LINUX_DEFAULT="splash pcie_aspm=off"

After that run;

update-grub

pcie_aspm is some sort of power management thingie which probably puts my networkcontroller to sleep or something.
And putting a usb memory stick in the laptop some how wakes it up.

[hendrik (hendrikp)]

Finally somebody else who is reporting the same.
https://askubuntu.com/questions/1344532/wired-network-extremely-slow/1345978#1345978

[Yann C (yannc)]

Hi, I'm the one who created the post on AskUbuntu.

In the certification, it appears that the Latitude 5420 is equipped with Intel I219-V ethernet cards, but this is not the case on ours, which are equipped with Intel I219-LM cards, maybe it's a clue?

Here is the kernels I've tested :
- 5.8.0-43-generic (fresh install from Ubuntu 20.04.2 iso)
- 5.8.0-55-generic (upgrade from Ubuntu 20.04.2 iso)
- 5.6.0-1032-oem (fresh install from Dell 20.04.1 iso)
- 5.10.0-1029-oem (upgrade from Dell 20.04.1 iso)
- 5.11.0-16-generic (fresh install from Ubuntu 21.04 iso)
- 5.11.0-18-generic (upgrade from Ubuntu 21.04 iso)

Also, I try to force the installation of the last e1000e drivers (v3.8.4) both on a fresh install from Ubuntu 20.04.2 iso and Dell 20.04.1 iso but the problem still here.
And, today I tried to modify the GRUB but nothing change, always this slow ethernet...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[hendrik (hendrikp)]

yes indeed. On the certification page they talk about "I219-V". I didn't even notice that. That's a good clue.

hmm you describe exactly the same problem as I have. I am almost 100% sure this fix should help you too. Did you reboot your system with the new kernel setting? Did you check in /boot/grub/grub.cfg if the new setting pcie_aspm=off really is there after the reboot?

[Yann C (yannc)]

Yes, the pcie_aspm is "off" but nothing change, the ethernet download is very slow.

I don't know if it's the same for you but the problem is only with "download" speed, the "upload" speed is almost normal (a little less than normal).

I tried to connect a USB key as you mentioned in your first message but I have no change, maybe a slight increase in speed for a few seconds then it becomes slow again.

[hendrik (hendrikp)]

Really strange that you cannot make the workaround work. Also in my case no issues when uploading. Only downloading was a problem. It sometimes went back to 100kb/sec...
You are describing the issue exactly the same as mine.

[hendrik (hendrikp)]

You are rebooting your system right? To make the kernel settings active?

[Andrey Sokolov (keremet)]

I have the same problem on the same laptop (dell latitude 5520). To solve the problem change MTU: ifconfig eth0 mtu 9000

[Japapoa (japapoa)]

sudo ethtool -C xxxx rx-usecs 6000

(replace xxxx with the name of your network card)

[hendrik (hendrikp)]

Might work. But I'm not sure of this setting can be done while doing a PXE boot.

[Yuan-Chen Cheng (ycheng-twn)]

I find some reliable ways to work around and AceLan push it further:

Ref: https://bugzilla.kernel.org/show_bug.cgi?id=213651

[Sebastian Luna-Valero (slunav)]

Thanks All for your great help. I am also affected by this issue with my Dell 5420 equipped with a I219-LM card (just arrived home yesterday).

I came across the AskUbuntu thread from Dell Community forums:
https://www.dell.com/community/Latitude/Dell-latitude-5420-5520-intel-ethernet-on-ubuntu-20-04/m-p/7889980

After the "sudo ethtool -C xxxx rx-usecs 6000" command I keep seeing rx_errors and rx_crc_errors with:

sudo ethtool -S <device> | grep -i err

Do you know why?

Many thanks,
Sebastian

[Yuan-Chen Cheng (ycheng-twn)]

@Sebastian, did you check #14 and apply the workaround?

[Sebastian Luna-Valero (slunav)]

Thanks @Yuan-Chen.

Yes, I did try but it didn't work for me.

[Yuan-Chen Cheng (ycheng-twn)]

Hi Sebastian

I test "sudo ethtool -C <device> rx-usecs 6000", it does work to a certain degree, but it will decrease the maximum LAN speed per test using iperf3. Not sure if this is the work around you are using for now?

For your test in #17, is this the command you try:

echo on | sudo tee /sys/bus/pci/devices/0000\:00\:16.0/power/control

If yes, and it does not work around this issue, can you run beload command

sudo lspci -nnvv > pci-nnvv

and upload pci-nnvv?

[Sebastian Luna-Valero (slunav)]

Hi Yuan-Chen

I tried:

"echo on | sudo tee /sys/bus/pci/devices/0000\:00\:16.0/power/control"

I also tried unloading mei* kernel modules.

However it didn't solve the isssue.

Here is the requested info:

# lspci -vvnns 00:1f.6
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (13) I219-LM [8086:15fb] (rev 20)
 Subsystem: Dell Ethernet Connection (13) I219-LM [1028:0a20]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0
 Interrupt: pin A routed to IRQ 147
 Region 0: Memory at a2300000 (32-bit, non-prefetchable) [size=128K]
 Capabilities: [c8] Power Management version 3
  Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
  Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
 Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
  Address: 00000000fee004f8 Data: 0000
 Kernel driver in use: e1000e
 Kernel modules: e1000e

I also compiled the latest version of e1000e in https://sourceforge.net/projects/e1000/

root@latitude-5420:~# modinfo e1000e | grep ver
filename: /lib/modules/5.10.0-1038-oem/updates/drivers/net/ethernet/intel/e1000e/e1000e.ko
version: 3.8.7-NAPI
description: Intel(R) PRO/1000 Network Driver
srcversion: 035BD57B8D93A45D1668FCC
vermagic: 5.10.0-1038-oem SMP mod_unload modversions

But it didn't help.

I am currently using the wireless adapter until a find a solution. Sadly, the only solution provided by technical support was to re-install Ubuntu, which I will do when I find free time.

Many thanks,
Sebastian

[Yuan-Chen Cheng (ycheng-twn)]

Hi Sebastian,

Sorry to hear it's not working on your side. That means the current direction currently been working on does not really solve the issue happens in your combination.

Here is what you can try:

1. upload sosreport log. like run `sudo sosreport`, and upload the output file in /tmp.
2. Try to find ways to work around the issue.
- does the set rx-usecs 6000 work around it?
- Let me assume you are using OEM image. With it, tlp and certain config files to make power consumption better are installed. Per my previous test, the other way to workaround is

$ sudo ethtool --set-eee _enp0XXX_ eee off
$ sudo tlp ac

[Yuan-Chen Cheng (ycheng-twn)]

One more thing, per my test, on LAN, you can kind of reproduce this issue by

On iperf server:

1. set packet delay to 5ms

   "tc qdisc add dev eno1 root netem delay 5ms" or
   "tc qdisc change dev eno1 root netem delay 5ms"

2. run iperf3 server by command "iperf3 -s"

On Client (say the machine have this issue) to simulate download:

   "iperf3 -c SERVER_IP -R"

[AceLan Kao (acelankao)]

Here is the test kernel for the e1000e network speed issue, it would be good if anyone of you could give it a try. Thanks.
http://people.canonical.com/~acelan/bugs/lp1933807

[hendrik (hendrikp)]

Hi,

I just tried and it looks like network is performing as expected.
Also my fix mentiod earlier to set 'pcie_aspm=off' does not seem to work anymore on kernel '5.11.0-34-generic'

So please release this as soon as possible.

Thanks

[Shengyao Xue (xueshengyao)]

another bug which share the same root cause with this one:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1927925

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1930754

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[AceLan Kao (acelankao)]

Patches have been submitted
https://patchwork.ozlabs.org/project/intel-wired-lan/list/?series=263466

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-oem-5.10/5.10.0-1049.51 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Yuan-Chen Cheng (ycheng-twn)]

5.10.0-1049.51 does test passed from my side.

[Sebastian Luna-Valero (slunav)]

5.10.0-1049.51 works for me too, many thanks indeed.

[hendrik (hendrikp)]

With cubic I have generated a new ISO with kernel 5.10.0-1049.51 installed.

1) PXE boot is working as fast as it should be.
2) Once O.S. installed download speeds are working as expected again.

So I can confirm this patch fixes the issue

When will this patch be included in the 5.11 generic kernel?

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-oem-5.10 - 5.10.0-1049.51

---------------
linux-oem-5.10 (5.10.0-1049.51) focal; urgency=medium

  * focal/linux-oem-5.10: 5.10.0-1049.50 -proposed tracker (LP: #1944209)

  * e1000e extremly slow (LP: #1930754)
    - SAUCE: e1000e: Separate TGP board type from SPT
    - SAUCE: e1000e: Fixing packet loss issues on new platforms

  * CVE-2021-41073
    - io_uring: ensure symmetry in handling iter types in loop_rw_iter()

 -- Chia-Lin Kao (AceLan) <email address hidden> Mon, 27 Sep 2021 18:33:36 +0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-oem-5.13 - 5.13.0-1017.21

---------------
linux-oem-5.13 (5.13.0-1017.21) focal; urgency=medium

  * focal/linux-oem-5.13: 5.13.0-1017.21 -proposed tracker (LP: #1946722)

  * Intel AX201 8086:7af0 subsys 8086:4070 hardware reset periodically: FW error
    in SYNC CMD UNKNOWN (LP: #1941665)
    - iwlwifi: mvm: support BIOS enable/disable for 11ax in Russia
    - iwlwifi: mvm: Read acpi dsm to get unii4 enable/disable bitmap.

 -- Timo Aaltonen <email address hidden> Tue, 12 Oct 2021 11:54:40 +0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-oem-5.14 - 5.14.0-1005.5

---------------
linux-oem-5.14 (5.14.0-1005.5) focal; urgency=medium

  * focal/linux-oem-5.14: 5.14.0-1005.5 -proposed tracker (LP: #1944905)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2021.09.27)

  * Update ADL-P dmc firmware (LP: #1945438)
    - drm/i915: Free all DMC payloads
    - drm/i915/dmc: Update to DMC v2.12

  * Enable runtime power management for yellow carp XHCI (LP: #1944382)
    - SAUCE: usb: xhci: Enable runtime-pm by default on AMD Yellow Carp platform

  * e1000e extremly slow (LP: #1930754)
    - SAUCE: e1000e: Separate TGP board type from SPT
    - SAUCE: e1000e: Fixing packet loss issues on new platforms

  * Fix missing HDMI audio on Intel RKL (LP: #1945556)
    - drm/i915/audio: Use BIOS provided value for RKL HDA link

  * Focal update: v5.14.9 upstream stable release (LP: #1945656)
    - mm, hwpoison: add is_free_buddy_page() in HWPoisonHandlable()
    - ocfs2: drop acl cache for directories too
    - mm/debug: sync up MR_CONTIG_RANGE and MR_LONGTERM_PIN
    - mm: fix uninitialized use in overcommit_policy_handler
    - usb: gadget: r8a66597: fix a loop in set_feature()
    - usb: gadget: u_audio: EP-OUT bInterval in fback frequency
    - usb: dwc2: gadget: Fix ISOC flow for BDMA and Slave
    - usb: dwc2: gadget: Fix ISOC transfer complete handling for DDMA
    - usb: musb: tusb6010: uninitialized data in tusb_fifo_write_unaligned()
    - cifs: Not to defer close on file when lock is set
    - cifs: Fix soft lockup during fsstress
    - cifs: fix incorrect check for null pointer in header_assemble
    - xen/x86: fix PV trap handling on secondary processors
    - usb-storage: Add quirk for ScanLogic SL11R-IDE older than 2.6c
    - USB: serial: cp210x: add ID for GW Instek GDM-834x Digital Multimeter
    - USB: cdc-acm: fix minor-number release
    - Revert "USB: bcma: Add a check for devm_gpiod_get"
    - binder: make sure fd closes complete
    - binder: fix freeze race
    - staging: greybus: uart: fix tty use after free
    - usb: isp1760: do not sleep in field register poll
    - Re-enable UAS for LaCie Rugged USB3-FW with fk quirk
    - usb: dwc3: core: balance phy init and exit
    - usb: cdns3: fix race condition before setting doorbell
    - usb: core: hcd: Add support for deferring roothub registration
    - USB: serial: mos7840: remove duplicated 0xac24 device ID
    - USB: serial: option: add Telit LN920 compositions
    - USB: serial: option: remove duplicate USB device ID
    - USB: serial: option: add device id for Foxconn T99W265
    - misc: bcm-vk: fix tty registration race
    - misc: genwqe: Fixes DMA mask setting
    - mcb: fix error handling in mcb_alloc_bus()
    - KVM: rseq: Update rseq when processing NOTIFY_RESUME on xfer to KVM guest
    - erofs: fix up erofs_lookup tracepoint
    - nexthop: Fix division by zero while replacing a resilient group
    - btrfs: prevent __btrfs_dump_space_info() to underflow its free space
    - xhci: Set HCD flag to defer primary roothub registration
    - serial: 8250: 8250_omap: Fix RX_LVL register offset
    - serial: mvebu-uart: fix dr...

[Merlin Hartley (mh332-maths)]

Brilliant! This seems to have solved the issues on our new Dell OptiPlex 7090s too .

Apologies if this is obvious ... but does this fix end up being back-ported into HWE (I seem to recall reading that it may do that after 3 weeks or something?)

Thanks!

[Roland Sommer (rsommer)]

As the fixes now have landed in the mainline linux kernel, that should just be a matter of time.

[Brian Murray (brian-murray)]

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.

[raghavendra hegde (raghu.rhegde)]

I have same dell optiplex 7090 desktop with ubuntu 20.04.03 kernel version 5.13 . Im getting network packet drop issues. Did anyone faced this problem and found any solution. Please let me know.

[Yuan-Chen Cheng (ycheng-twn)]

could you try linux-oem?

ex: $ sudo apt-get install linux-oem-20.04

[Timo Aaltonen (tjaalton)]

should be fixed by 639e298f432fb05 upstream, which is in jammy and up at least

[raghavendra hegde (raghu.rhegde)]

The issue is fixed after contacting ubuntu support team

systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

systemctl status sleep.target suspend.target hibernate.target hybrid-sleep.target

enter the below value under /etc/default/grub

#GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash acpi=off apm=off"

sudo update-grub & reboot the system

[Mike Osborne (osbormd)]

What's interesting is on a Dell Optiplex 3080 with a Realtek 8169 adapter I have similar symptoms (extremely slow downloads in the KB/s, and frequent packet drops seen in captures).

I'm seeing this also on Dell Precision 3650's with Intel I219-LM (rev 11) adapters, which lines up with some of the earlier posts in this thread.

The workaround above to disable auto power control resolves these symptoms on BOTH the Realtek 8169 and the Intel I219-LM adapters.

Oddly enough, a Dell Precision 3630 with a slightly different variant of the Intel I219-LM (rev10) does not have the same issues.

Not sure if any of this helps, but I was surprised to see it's not just Intel cards with these linux power control speed and packet loss symptoms...

Mike