e1000e extremly slow

Bug #1930754 reported by hendrik
72
This bug affects 13 people
Affects Status Importance Assigned to Milestone
HWE Next
New
Undecided
Unassigned
OEM Priority Project
Confirmed
Critical
Unassigned
linux (Ubuntu)
In Progress
Undecided
AceLan Kao
Focal
Invalid
Undecided
Unassigned
Hirsute
Won't Fix
Undecided
AceLan Kao
Impish
In Progress
Undecided
AceLan Kao
linux-oem-5.10 (Ubuntu)
Invalid
Critical
Unassigned
Focal
Fix Released
Undecided
AceLan Kao
Hirsute
Invalid
Undecided
Unassigned
Impish
Invalid
Critical
Unassigned
linux-oem-5.13 (Ubuntu)
Invalid
Undecided
Unassigned
Focal
Fix Released
Undecided
AceLan Kao
Hirsute
Invalid
Undecided
Unassigned
Impish
Invalid
Undecided
Unassigned
linux-oem-5.14 (Ubuntu)
Invalid
Undecided
Unassigned
Focal
Fix Released
Undecided
AceLan Kao
Hirsute
Invalid
Undecided
Unassigned
Impish
Invalid
Undecided
Unassigned

Bug Description

[Impact]
e1000e on TGP PCH encounters a network speed issue that rx speed is below 1MB/s

[Fix]
Intel provides 2 patches to fix this
https://patchwork.ozlabs.org/project/intel-wired-lan/list/?series=263466

[Test]
Verified by our Dell and Lenovo platforms and the bug reporter.

[Where problems could occur]
The fix only applies to e1000e with TGP PCH, and add a flag "E1000_FFLT_DBG_DONT_GATE_WAKE_DMA_CLK". The impact should be minimized and should not lead to other regressions.

=============================

We have dell latitude 5420 & 5520 laptops with onboard intel ethernet controller.
Problem is that the ethernet port is extremely slow. about 100kbyte/s download throughput while on a gigabit connection. Upload seems to work just fine.

Also when I pxe boot the system it's also painfully slow.

But here comes the funny part. When I attach a usb memory stick the throughput of the networkcontroller is as expected.

I've used latest ubuntu 20.04.2 kernel 5.8.0-55
I've used latest ubuntu 20.04.2 kernel 5.10.0-1029-oem

Turns out this problem is not showing up when I manually install the 5.8.18 kernel.

Also when I create my own iso with the help of Cubic and install the 5.8.18 kernel in it the pxe boot is working as expected with high throughput

system: Dell latitude 5420 + 5520
nic: Ethernet Connection (13) I219-LM
ubuntu 20.04.2
module: e1000e

Please assist. Thank you

CVE References

hendrik (hendrikp)
description: updated
description: updated
Revision history for this message
hendrik (hendrikp) wrote :

Issue also exists on 5.6.0-1032-oem. Which is certified.

https://certification.ubuntu.com/hardware/202010-28335

Revision history for this message
hendrik (hendrikp) wrote :
Revision history for this message
hendrik (hendrikp) wrote :

To whom is may concern. I found the workaround for this.

The trick is to set the boot kernel parameter "pcie_aspm=off" in '/etc/default/grub'

Like this:

GRUB_CMDLINE_LINUX_DEFAULT="splash pcie_aspm=off"

After that run;

update-grub

pcie_aspm is some sort of power management thingie which probably puts my networkcontroller to sleep or something.
And putting a usb memory stick in the laptop some how wakes it up.

Revision history for this message
hendrik (hendrikp) wrote :
Revision history for this message
Yann C (yannc) wrote :

Hi, I'm the one who created the post on AskUbuntu.

In the certification, it appears that the Latitude 5420 is equipped with Intel I219-V ethernet cards, but this is not the case on ours, which are equipped with Intel I219-LM cards, maybe it's a clue?

Here is the kernels I've tested :
- 5.8.0-43-generic (fresh install from Ubuntu 20.04.2 iso)
- 5.8.0-55-generic (upgrade from Ubuntu 20.04.2 iso)
- 5.6.0-1032-oem (fresh install from Dell 20.04.1 iso)
- 5.10.0-1029-oem (upgrade from Dell 20.04.1 iso)
- 5.11.0-16-generic (fresh install from Ubuntu 21.04 iso)
- 5.11.0-18-generic (upgrade from Ubuntu 21.04 iso)

Also, I try to force the installation of the last e1000e drivers (v3.8.4) both on a fresh install from Ubuntu 20.04.2 iso and Dell 20.04.1 iso but the problem still here.
And, today I tried to modify the GRUB but nothing change, always this slow ethernet...

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-oem-5.10 (Ubuntu):
status: New → Confirmed
Revision history for this message
hendrik (hendrikp) wrote :

yes indeed. On the certification page they talk about "I219-V". I didn't even notice that. That's a good clue.

hmm you describe exactly the same problem as I have. I am almost 100% sure this fix should help you too. Did you reboot your system with the new kernel setting? Did you check in /boot/grub/grub.cfg if the new setting pcie_aspm=off really is there after the reboot?

Revision history for this message
Yann C (yannc) wrote :

Yes, the pcie_aspm is "off" but nothing change, the ethernet download is very slow.

I don't know if it's the same for you but the problem is only with "download" speed, the "upload" speed is almost normal (a little less than normal).

I tried to connect a USB key as you mentioned in your first message but I have no change, maybe a slight increase in speed for a few seconds then it becomes slow again.

Revision history for this message
hendrik (hendrikp) wrote :

Really strange that you cannot make the workaround work. Also in my case no issues when uploading. Only downloading was a problem. It sometimes went back to 100kb/sec...
You are describing the issue exactly the same as mine.

Revision history for this message
hendrik (hendrikp) wrote :

You are rebooting your system right? To make the kernel settings active?

Revision history for this message
Andrey Sokolov (keremet) wrote :

I have the same problem on the same laptop (dell latitude 5520). To solve the problem change MTU: ifconfig eth0 mtu 9000

Revision history for this message
Japapoa (japapoa) wrote :

sudo ethtool -C xxxx rx-usecs 6000

(replace xxxx with the name of your network card)

Revision history for this message
hendrik (hendrikp) wrote :

Might work. But I'm not sure of this setting can be done while doing a PXE boot.

tags: added: oem-priority
Changed in oem-priority:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote :

I find some reliable ways to work around and AceLan push it further:

Ref: https://bugzilla.kernel.org/show_bug.cgi?id=213651

tags: added: originate-from-1933807 somerville
tags: added: originate-from-1933189
tags: added: originate-from-1934947
Changed in oem-priority:
importance: High → Critical
Revision history for this message
Sebastian Luna-Valero (slunav) wrote :

Thanks All for your great help. I am also affected by this issue with my Dell 5420 equipped with a I219-LM card (just arrived home yesterday).

I came across the AskUbuntu thread from Dell Community forums:
https://www.dell.com/community/Latitude/Dell-latitude-5420-5520-intel-ethernet-on-ubuntu-20-04/m-p/7889980

After the "sudo ethtool -C xxxx rx-usecs 6000" command I keep seeing rx_errors and rx_crc_errors with:

sudo ethtool -S <device> | grep -i err

Do you know why?

Many thanks,
Sebastian

Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote :

@Sebastian, did you check #14 and apply the workaround?

Revision history for this message
Sebastian Luna-Valero (slunav) wrote :

Thanks @Yuan-Chen.

Yes, I did try but it didn't work for me.

Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote (last edit ):

Hi Sebastian

I test "sudo ethtool -C <device> rx-usecs 6000", it does work to a certain degree, but it will decrease the maximum LAN speed per test using iperf3. Not sure if this is the work around you are using for now?

For your test in #17, is this the command you try:

echo on | sudo tee /sys/bus/pci/devices/0000\:00\:16.0/power/control

If yes, and it does not work around this issue, can you run beload command

sudo lspci -nnvv > pci-nnvv

and upload pci-nnvv?

Revision history for this message
Sebastian Luna-Valero (slunav) wrote :

Hi Yuan-Chen

I tried:

"echo on | sudo tee /sys/bus/pci/devices/0000\:00\:16.0/power/control"

I also tried unloading mei* kernel modules.

However it didn't solve the isssue.

Here is the requested info:

# lspci -vvnns 00:1f.6
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (13) I219-LM [8086:15fb] (rev 20)
 Subsystem: Dell Ethernet Connection (13) I219-LM [1028:0a20]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0
 Interrupt: pin A routed to IRQ 147
 Region 0: Memory at a2300000 (32-bit, non-prefetchable) [size=128K]
 Capabilities: [c8] Power Management version 3
  Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
  Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
 Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
  Address: 00000000fee004f8 Data: 0000
 Kernel driver in use: e1000e
 Kernel modules: e1000e

I also compiled the latest version of e1000e in https://sourceforge.net/projects/e1000/

root@latitude-5420:~# modinfo e1000e | grep ver
filename: /lib/modules/5.10.0-1038-oem/updates/drivers/net/ethernet/intel/e1000e/e1000e.ko
version: 3.8.7-NAPI
description: Intel(R) PRO/1000 Network Driver
srcversion: 035BD57B8D93A45D1668FCC
vermagic: 5.10.0-1038-oem SMP mod_unload modversions

But it didn't help.

I am currently using the wireless adapter until a find a solution. Sadly, the only solution provided by technical support was to re-install Ubuntu, which I will do when I find free time.

Many thanks,
Sebastian

Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote :

Hi Sebastian,

Sorry to hear it's not working on your side. That means the current direction currently been working on does not really solve the issue happens in your combination.

Here is what you can try:

1. upload sosreport log. like run `sudo sosreport`, and upload the output file in /tmp.
2. Try to find ways to work around the issue.
- does the set rx-usecs 6000 work around it?
- Let me assume you are using OEM image. With it, tlp and certain config files to make power consumption better are installed. Per my previous test, the other way to workaround is

$ sudo ethtool --set-eee _enp0XXX_ eee off
$ sudo tlp ac

Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote :

One more thing, per my test, on LAN, you can kind of reproduce this issue by

On iperf server:

1. set packet delay to 5ms

   "tc qdisc add dev eno1 root netem delay 5ms" or
   "tc qdisc change dev eno1 root netem delay 5ms"

2. run iperf3 server by command "iperf3 -s"

On Client (say the machine have this issue) to simulate download:

   "iperf3 -c SERVER_IP -R"

Yao Wei (medicalwei)
tags: added: fossa-edge-staging
Yao Wei (medicalwei)
tags: added: originate-from-1942834
tags: added: intel-csme
Revision history for this message
AceLan Kao (acelankao) wrote :

Here is the test kernel for the e1000e network speed issue, it would be good if anyone of you could give it a try. Thanks.
http://people.canonical.com/~acelan/bugs/lp1933807

Revision history for this message
hendrik (hendrikp) wrote :

Hi,

I just tried and it looks like network is performing as expected.
Also my fix mentiod earlier to set 'pcie_aspm=off' does not seem to work anymore on kernel '5.11.0-34-generic'

So please release this as soon as possible.

Thanks

Revision history for this message
Shengyao Xue (xueshengyao) wrote :

another bug which share the same root cause with this one:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1927925

Changed in linux-oem-5.10 (Ubuntu):
importance: Undecided → Critical
assignee: nobody → AceLan Kao (acelankao)
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1930754

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
AceLan Kao (acelankao) wrote :
Changed in linux (Ubuntu Focal):
assignee: nobody → AceLan Kao (acelankao)
status: New → In Progress
Changed in linux (Ubuntu Hirsute):
assignee: nobody → AceLan Kao (acelankao)
status: New → In Progress
Changed in linux (Ubuntu Impish):
assignee: nobody → AceLan Kao (acelankao)
status: Incomplete → In Progress
Changed in linux-oem-5.10 (Ubuntu Hirsute):
status: New → Invalid
Changed in linux-oem-5.10 (Ubuntu Impish):
assignee: AceLan Kao (acelankao) → nobody
status: Confirmed → Invalid
Changed in linux-oem-5.14 (Ubuntu Hirsute):
status: New → Invalid
Changed in linux-oem-5.14 (Ubuntu Impish):
status: New → Invalid
Changed in linux-oem-5.10 (Ubuntu Focal):
assignee: nobody → AceLan Kao (acelankao)
status: New → In Progress
Changed in linux-oem-5.14 (Ubuntu Focal):
assignee: nobody → AceLan Kao (acelankao)
status: New → In Progress
AceLan Kao (acelankao)
description: updated
AceLan Kao (acelankao)
description: updated
AceLan Kao (acelankao)
Changed in linux-oem-5.10 (Ubuntu Focal):
status: In Progress → Fix Committed
AceLan Kao (acelankao)
Changed in linux-oem-5.13 (Ubuntu Focal):
assignee: nobody → AceLan Kao (acelankao)
status: New → In Progress
Changed in linux-oem-5.13 (Ubuntu Hirsute):
status: New → Invalid
Changed in linux-oem-5.13 (Ubuntu Impish):
status: New → Invalid
AceLan Kao (acelankao)
Changed in linux (Ubuntu Focal):
status: In Progress → Invalid
assignee: AceLan Kao (acelankao) → nobody
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-oem-5.10/5.10.0-1049.51 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
AceLan Kao (acelankao)
tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote :

5.10.0-1049.51 does test passed from my side.

Revision history for this message
Sebastian Luna-Valero (slunav) wrote :

5.10.0-1049.51 works for me too, many thanks indeed.

Revision history for this message
hendrik (hendrikp) wrote :

With cubic I have generated a new ISO with kernel 5.10.0-1049.51 installed.

1) PXE boot is working as fast as it should be.
2) Once O.S. installed download speeds are working as expected again.

So I can confirm this patch fixes the issue

When will this patch be included in the 5.11 generic kernel?

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux-oem-5.10 - 5.10.0-1049.51

---------------
linux-oem-5.10 (5.10.0-1049.51) focal; urgency=medium

  * focal/linux-oem-5.10: 5.10.0-1049.50 -proposed tracker (LP: #1944209)

  * e1000e extremly slow (LP: #1930754)
    - SAUCE: e1000e: Separate TGP board type from SPT
    - SAUCE: e1000e: Fixing packet loss issues on new platforms

  * CVE-2021-41073
    - io_uring: ensure symmetry in handling iter types in loop_rw_iter()

 -- Chia-Lin Kao (AceLan) <email address hidden> Mon, 27 Sep 2021 18:33:36 +0800

Changed in linux-oem-5.10 (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux-oem-5.13 - 5.13.0-1017.21

---------------
linux-oem-5.13 (5.13.0-1017.21) focal; urgency=medium

  * focal/linux-oem-5.13: 5.13.0-1017.21 -proposed tracker (LP: #1946722)

  * Intel AX201 8086:7af0 subsys 8086:4070 hardware reset periodically: FW error
    in SYNC CMD UNKNOWN (LP: #1941665)
    - iwlwifi: mvm: support BIOS enable/disable for 11ax in Russia
    - iwlwifi: mvm: Read acpi dsm to get unii4 enable/disable bitmap.

 -- Timo Aaltonen <email address hidden> Tue, 12 Oct 2021 11:54:40 +0300

Changed in linux-oem-5.13 (Ubuntu Focal):
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (52.7 KiB)

This bug was fixed in the package linux-oem-5.14 - 5.14.0-1005.5

---------------
linux-oem-5.14 (5.14.0-1005.5) focal; urgency=medium

  * focal/linux-oem-5.14: 5.14.0-1005.5 -proposed tracker (LP: #1944905)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2021.09.27)

  * Update ADL-P dmc firmware (LP: #1945438)
    - drm/i915: Free all DMC payloads
    - drm/i915/dmc: Update to DMC v2.12

  * Enable runtime power management for yellow carp XHCI (LP: #1944382)
    - SAUCE: usb: xhci: Enable runtime-pm by default on AMD Yellow Carp platform

  * e1000e extremly slow (LP: #1930754)
    - SAUCE: e1000e: Separate TGP board type from SPT
    - SAUCE: e1000e: Fixing packet loss issues on new platforms

  * Fix missing HDMI audio on Intel RKL (LP: #1945556)
    - drm/i915/audio: Use BIOS provided value for RKL HDA link

  * Focal update: v5.14.9 upstream stable release (LP: #1945656)
    - mm, hwpoison: add is_free_buddy_page() in HWPoisonHandlable()
    - ocfs2: drop acl cache for directories too
    - mm/debug: sync up MR_CONTIG_RANGE and MR_LONGTERM_PIN
    - mm: fix uninitialized use in overcommit_policy_handler
    - usb: gadget: r8a66597: fix a loop in set_feature()
    - usb: gadget: u_audio: EP-OUT bInterval in fback frequency
    - usb: dwc2: gadget: Fix ISOC flow for BDMA and Slave
    - usb: dwc2: gadget: Fix ISOC transfer complete handling for DDMA
    - usb: musb: tusb6010: uninitialized data in tusb_fifo_write_unaligned()
    - cifs: Not to defer close on file when lock is set
    - cifs: Fix soft lockup during fsstress
    - cifs: fix incorrect check for null pointer in header_assemble
    - xen/x86: fix PV trap handling on secondary processors
    - usb-storage: Add quirk for ScanLogic SL11R-IDE older than 2.6c
    - USB: serial: cp210x: add ID for GW Instek GDM-834x Digital Multimeter
    - USB: cdc-acm: fix minor-number release
    - Revert "USB: bcma: Add a check for devm_gpiod_get"
    - binder: make sure fd closes complete
    - binder: fix freeze race
    - staging: greybus: uart: fix tty use after free
    - usb: isp1760: do not sleep in field register poll
    - Re-enable UAS for LaCie Rugged USB3-FW with fk quirk
    - usb: dwc3: core: balance phy init and exit
    - usb: cdns3: fix race condition before setting doorbell
    - usb: core: hcd: Add support for deferring roothub registration
    - USB: serial: mos7840: remove duplicated 0xac24 device ID
    - USB: serial: option: add Telit LN920 compositions
    - USB: serial: option: remove duplicate USB device ID
    - USB: serial: option: add device id for Foxconn T99W265
    - misc: bcm-vk: fix tty registration race
    - misc: genwqe: Fixes DMA mask setting
    - mcb: fix error handling in mcb_alloc_bus()
    - KVM: rseq: Update rseq when processing NOTIFY_RESUME on xfer to KVM guest
    - erofs: fix up erofs_lookup tracepoint
    - nexthop: Fix division by zero while replacing a resilient group
    - btrfs: prevent __btrfs_dump_space_info() to underflow its free space
    - xhci: Set HCD flag to defer primary roothub registration
    - serial: 8250: 8250_omap: Fix RX_LVL register offset
    - serial: mvebu-uart: fix dr...

Changed in linux-oem-5.14 (Ubuntu Focal):
status: In Progress → Fix Released
Revision history for this message
Merlin Hartley (mh332-maths) wrote :

Brilliant! This seems to have solved the issues on our new Dell OptiPlex 7090s too .

Apologies if this is obvious ... but does this fix end up being back-ported into HWE (I seem to recall reading that it may do that after 3 weeks or something?)

Thanks!

Revision history for this message
Roland Sommer (rsommer) wrote :

As the fixes now have landed in the mainline linux kernel, that should just be a matter of time.

Revision history for this message
Brian Murray (brian-murray) wrote :

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.

Changed in linux (Ubuntu Hirsute):
status: In Progress → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.