Ethernet Controller I225-V (rev 03) fails after some time with: igc 0000:09:00.0 eno1: PCIe link lost, device now detached

[Thomas Bechtold]

I have a "ASUSTeK COMPUTER INC. ROG STRIX X670E-E GAMING WIFI" motherboard and the builtin ethernet card does fail from time to time (randomly - usually some hours).

Steps to reproduce:

- activate ethernet
- wait hour(s)
- then ethernet will no longer work

The kernel logs do have then:

Nov 12 08:09:13 schwarzwurzel kernel: igc 0000:09:00.0 eno1: PCIe link lost, device now detached
Nov 12 08:09:13 schwarzwurzel kernel: ------------[ cut here ]------------
Nov 12 08:09:13 schwarzwurzel kernel: igc: Failed to read reg 0xc030!
Nov 12 08:09:13 schwarzwurzel kernel: WARNING: CPU: 13 PID: 176836 at drivers/net/ethernet/intel/igc/igc_main.c:6583 igc_rd32+0xa4/0xc0 [igc]
Nov 12 08:09:13 schwarzwurzel kernel: Modules linked in: tls nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype veth nft_masq xt_mark ccm zfs(PO) spl(O) ip6table_nat xt_comment vhost_vsock vmw_vsock_virtio_transport_common vhost vhost_iotlb vsock xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_t
cpudp nft_compat rfcomm nft_chain_nat nf_nat nf_conntrack snd_seq_dummy nf_defrag_ipv6 snd_hrtimer nf_defrag_ipv4 nf_tables nfnetlink cmac algif_hash overlay algif_skcipher af_alg bnep binfmt_misc nls_iso8859_1 snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_usb_audio snd_hda_codec snd_usbmid
i_lib snd_hda_core snd_ump snd_hwdep intel_rapl_msr snd_pcm intel_rapl_common snd_seq_midi btusb uvcvideo snd_seq_midi_event btrtl videobuf2_vmalloc snd_rawmidi iwlmvm edac_mce_amd btbcm uvc mfd_aaeon eeepc_wmi snd_seq kvm_amd btintel videobuf2_memops mac80211 iptable_filter asus_wmi snd_seq_device videobuf2_v4l2 btm
tk kvm libarc4 ip6table_filter ledtrig_audio snd_timer videode
v bluetooth irqbypass ip6_tables iwlwifi
Nov 12 08:09:13 schwarzwurzel kernel: sparse_keymap snd videobuf2_common ecdh_generic br_netfilter rapl platform_profile wmi_bmof ccp mc soundcore cfg80211 i2c_piix4 k10temp ecc input_leds gpio_amdpt bridge mac_hid stp llc arp_tables msr parport_pc ppdev lp parport efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic dm_crypt raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid amdgpu amdxcp iommu_v2 drm_buddy gpu_sched i2c_algo_bit crct10dif_pclmul drm_suballoc_helper crc32_pclmul drm_ttm_helper polyval_clmulni ttm polyval_generic drm_display_helper ghash_clmulni_intel cec aesni_intel rc_core nvme drm_kms_helper crypto_simd ahci xhci_pci nvme_core video igc cryptd drm libahci xhci_pci_renesas nvme_common wmi
Nov 12 08:09:13 schwarzwurzel kernel: CPU: 13 PID: 176836 Comm: kworker/13:0 Tainted: P W O 6.5.0-10-generic #10-Ubuntu
Nov 12 08:09:13 schwarzwurzel kernel: Hardware name: ASUS System Product Name/ROG STRIX X670E-E GAMING WIFI, BIOS 1602 08/15/2023
Nov 12 08:09:13 schwarzwurzel kernel: Workqueue: events igc_watchdog_task [igc]
Nov 12 08:09:13 schwarzwurzel kernel: RIP: 0010:igc_rd32+0xa4/0xc0 [igc]
Nov 12 08:09:13 schwarzwurzel kernel: Code: c7 c6 e0 a6 72 c0 e8 0b e9 32 f1 48 8b bb 28 ff ff ff e8 8f ae e1 f0 84 c0 74 b4 44 89 e6 48 c7 c7 08 a7 72 c0 e8 5c 12 5f f0 <0f> 0b eb a1 b8 ff ff ff ff 31 d2 31 f6 31 ff e9 38 63 65 f1 0f 1f
Nov 12 08:09:13 schwarzwurzel kernel: RSP: 0018:ffffb769009bfd98 EFLAGS: 00010246
Nov 12 08:09:13 schwarzwurzel kernel: RAX: 0000000000000000 RBX: ffff96c5ceda6cb8 RCX: 0000000000000000
Nov 12 08:09:13 schwarzwurzel kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Nov 12 08:09:13 schwarzwurzel kernel: RBP: ffffb769009bfdb0 R08: 0000000000000000 R09: 0000000000000000
Nov 12 08:09:13 schwarzwurzel kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000c030
Nov 12 08:09:13 schwarzwurzel kernel: R13: ffff96c5ceda6000 R14: 0000000000000000 R15: ffff96c5ced9ad40
Nov 12 08:09:13 schwarzwurzel kernel: FS: 0000000000000000(0000) GS:ffff96d4d8740000(0000) knlGS:0000000000000000
Nov 12 08:09:13 schwarzwurzel kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 12 08:09:13 schwarzwurzel kernel: CR2: 00001c0103e1500c CR3: 000000041863a000 CR4: 0000000000750ee0
Nov 12 08:09:13 schwarzwurzel kernel: PKRU: 55555554
Nov 12 08:09:13 schwarzwurzel kernel: Call Trace:
Nov 12 08:09:13 schwarzwurzel kernel: <TASK>
Nov 12 08:09:13 schwarzwurzel kernel: ? show_regs+0x6d/0x80
Nov 12 08:09:13 schwarzwurzel kernel: ? __warn+0x89/0x160
Nov 12 08:09:13 schwarzwurzel kernel: ? igc_rd32+0xa4/0xc0 [igc]
Nov 12 08:09:13 schwarzwurzel kernel: ? report_bug+0x17e/0x1b0
Nov 12 08:09:13 schwarzwurzel kernel: ? handle_bug+0x51/0xa0
Nov 12 08:09:13 schwarzwurzel kernel: ? exc_invalid_op+0x18/0x80
Nov 12 08:09:13 schwarzwurzel kernel: ? asm_exc_invalid_op+0x1b/0x20
Nov 12 08:09:13 schwarzwurzel kernel: ? igc_rd32+0xa4/0xc0 [igc]
Nov 12 08:09:13 schwarzwurzel kernel: ? igc_rd32+0xa4/0xc0 [igc]
Nov 12 08:09:13 schwarzwurzel kernel: igc_update_stats+0xab/0x770 [igc]
Nov 12 08:09:13 schwarzwurzel kernel: igc_watchdog_task+0xa1/0x540 [igc]
Nov 12 08:09:13 schwarzwurzel kernel: ? __schedule+0x2d4/0x770
Nov 12 08:09:13 schwarzwurzel kernel: process_one_work+0x220/0x440
Nov 12 08:09:13 schwarzwurzel kernel: worker_thread+0x4d/0x3f0
Nov 12 08:09:13 schwarzwurzel kernel: ? srso_alias_return_thunk+0x5/0x7f
Nov 12 08:09:13 schwarzwurzel kernel: ? _raw_spin_lock_irqsave+0xe/0x20
Nov 12 08:09:13 schwarzwurzel kernel: ? __pfx_worker_thread+0x10/0x10
Nov 12 08:09:13 schwarzwurzel kernel: kthread+0xef/0x120
Nov 12 08:09:13 schwarzwurzel kernel: ? __pfx_kthread+0x10/0x10
Nov 12 08:09:13 schwarzwurzel kernel: ret_from_fork+0x44/0x70
Nov 12 08:09:13 schwarzwurzel kernel: ? __pfx_kthread+0x10/0x10
Nov 12 08:09:13 schwarzwurzel kernel: ret_from_fork_asm+0x1b/0x30
Nov 12 08:09:13 schwarzwurzel kernel: </TASK>
Nov 12 08:09:13 schwarzwurzel kernel: ---[ end trace 0000000000000000 ]---

I already tried a couple of options suggested in some forums (eg. adding "pcie_port_pm=off pcie_aspm=off" to the kernel command line) but that doesn't help.

ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: linux-image-6.5.0-10-generic 6.5.0-10.10
ProcVersionSignature: Ubuntu 6.5.0-10.10-generic 6.5.3
Uname: Linux 6.5.0-10-generic x86_64
NonfreeKernelModules: zfs
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
CRDA: N/A
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Sun Nov 12 10:25:44 2023
InstallationDate: Installed on 2023-10-08 (34 days ago)
InstallationMedia: Ubuntu 23.10 "Mantic Minotaur" - Beta amd64 (20230919.1)
MachineType: {report['dmi.sys.vendor']} {report['dmi.product.name']}
ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/usr/bin/zsh
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
ProcFB: 0 amdgpudrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.5.0-10-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro quiet splash pcie_port_pm=off pcie_aspm=off
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-6.5.0-10-generic N/A
 linux-backports-modules-6.5.0-10-generic N/A
 linux-firmware 20230919.git3672ccab-0ubuntu2.1
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 08/15/2023
dmi.bios.release: 16.2
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1602
dmi.board.asset.tag: Default string
dmi.board.name: ROG STRIX X670E-E GAMING WIFI
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1602:bd08/15/2023:br16.2:svnASUS:pnSystemProductName:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnROGSTRIXX670E-EGAMINGWIFI:rvrRev1.xx:cvnDefaultstring:ct3:cvrDefaultstring:skuSKU:
dmi.product.family: To be filled by O.E.M.
dmi.product.name: System Product Name
dmi.product.sku: SKU
dmi.product.version: System Version
dmi.sys.vendor: ASUS

[Juerg Haefliger (juergh)]

Is this a regression, i.e., has it worked before? Are you running the latest BIOS? Seems to be a general problem with ASUS MBs. You could try to add the following to the kernel commandline:

pcie_aspm.policy=performance

or

pcie_aspm=off

[Thomas Bechtold (toabctl)]

> Is this a regression, i.e., has it worked before?

This is a new desktop PC so idk if this is a regression. But from what I read on the internet, it seems to be broken forever

> Are you running the latest BIOS?

Yes. This is with "ROG STRIX X670E-F GAMING WIFI BIOS 1709" (1709 is the version here)

> Seems to be a general problem with ASUS MBs. You could try to add the following to the kernel commandline:
> pcie_aspm=off

I use that already. doesn't help (see my initial comment which includes /proc/cmdline )

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]