NETDEV WATCHDOG: enp5s0 (r8169): transmit queue 0 timed out

Bug #1874464 reported by Dan Watkins on 2020-04-23

This bug report will be marked for expiration in 23 days if no further activity occurs. (find out why)

12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Status tracked in Groovy
Focal
Undecided
Unassigned
Groovy
Undecided
Unassigned

Bug Description

Running focal on a desktop, I accidentally clicked "Enable networking" in nm-applet, disabling my networking. When I clicked it again to reenable it, my networking did not return. After unsuccessfully poking at it for a while, I rebooted and saw the below (and still no networking). `rmmod r8169; modprobe r8169` had no (apparent) effect, nor did further reboots. I rebooted onto two different kernels, both exhibited the same behaviour: 5.4.0-21-generic, 5.4.0-26-generic.

I was finally only able to restore networking by _rebooting into Windows_ and then rebooting back into Ubuntu.

(My supposition is that NetworkManager/the kernel set *waves hands* something on the network card that persists across boots when it was disabled, and that wasn't correctly unset when I reenabled networking (or on following boots), but Windows _does_ correctly handle that case on boot, and reset it to a working state.)

Apr 23 10:07:43 surprise kernel: ------------[ cut here ]------------
Apr 23 10:07:43 surprise kernel: NETDEV WATCHDOG: enp5s0 (r8169): transmit queue 0 timed out
Apr 23 10:07:43 surprise kernel: WARNING: CPU: 9 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x258/0x260
Apr 23 10:07:43 surprise kernel: Modules linked in: zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlua(PO) xt_comment dummy xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_>
Apr 23 10:07:43 surprise kernel: autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_crypt hid_microsoft ff_memless hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) crct10dif_pclmul crc32_pclmul ghash_clmulni_intel drm_km>
Apr 23 10:07:43 surprise kernel: CPU: 9 PID: 0 Comm: swapper/9 Tainted: P OE 5.4.0-26-generic #30-Ubuntu
Apr 23 10:07:43 surprise kernel: Hardware name: Gigabyte Technology Co., Ltd. B450M DS3H/B450M DS3H-CF, BIOS F4 01/25/2019
Apr 23 10:07:43 surprise kernel: RIP: 0010:dev_watchdog+0x258/0x260
Apr 23 10:07:43 surprise kernel: Code: 85 c0 75 e5 eb 9f 4c 89 ff c6 05 bf 06 e8 00 01 e8 6d bb fa ff 44 89 e9 4c 89 fe 48 c7 c7 50 6d a3 b4 48 89 c2 e8 83 3f 71 ff <0f> 0b eb 80 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 41 57 49 89 d7
Apr 23 10:07:43 surprise kernel: RSP: 0018:ffffa90d40378e30 EFLAGS: 00010286
Apr 23 10:07:43 surprise kernel: RAX: 0000000000000000 RBX: ffff8a7578b00400 RCX: 0000000000000000
Apr 23 10:07:43 surprise kernel: RDX: ffff8a758ee67740 RSI: ffff8a758ee578c8 RDI: 0000000000000300
Apr 23 10:07:43 surprise kernel: RBP: ffffa90d40378e60 R08: ffff8a758ee578c8 R09: 0000000000000004
Apr 23 10:07:43 surprise kernel: R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000001
Apr 23 10:07:43 surprise kernel: R13: 0000000000000000 R14: ffff8a758cadc480 R15: ffff8a758cadc000
Apr 23 10:07:43 surprise kernel: FS: 0000000000000000(0000) GS:ffff8a758ee40000(0000) knlGS:0000000000000000
Apr 23 10:07:43 surprise kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 23 10:07:43 surprise kernel: CR2: 00007f4d2000d5eb CR3: 00000003fcfe2000 CR4: 00000000003406e0
Apr 23 10:07:43 surprise kernel: Call Trace:
Apr 23 10:07:43 surprise kernel: <IRQ>
Apr 23 10:07:43 surprise kernel: ? pfifo_fast_enqueue+0x150/0x150
Apr 23 10:07:43 surprise kernel: call_timer_fn+0x32/0x130
Apr 23 10:07:43 surprise kernel: __run_timers.part.0+0x180/0x280
Apr 23 10:07:43 surprise kernel: ? tick_sched_handle+0x33/0x60
Apr 23 10:07:43 surprise kernel: ? tick_sched_timer+0x3d/0x80
Apr 23 10:07:43 surprise kernel: ? ktime_get+0x3e/0xa0
Apr 23 10:07:43 surprise kernel: run_timer_softirq+0x2a/0x50
Apr 23 10:07:43 surprise kernel: __do_softirq+0xe1/0x2d6
Apr 23 10:07:43 surprise kernel: ? hrtimer_interrupt+0x13b/0x220
Apr 23 10:07:43 surprise kernel: irq_exit+0xae/0xb0
Apr 23 10:07:43 surprise kernel: smp_apic_timer_interrupt+0x7b/0x140
Apr 23 10:07:43 surprise kernel: apic_timer_interrupt+0xf/0x20
Apr 23 10:07:43 surprise kernel: </IRQ>
Apr 23 10:07:43 surprise kernel: RIP: 0010:cpuidle_enter_state+0xc5/0x450
Apr 23 10:07:43 surprise kernel: Code: ff e8 df 0d 81 ff 80 7d c7 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 65 03 00 00 31 ff e8 32 7a 87 ff fb 66 0f 1f 44 00 00 <45> 85 ed 0f 88 8f 02 00 00 49 63 cd 4c 8b 7d d0 4c 2b 7d c8 48 8d
Apr 23 10:07:43 surprise kernel: RSP: 0018:ffffa90d4016fe38 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Apr 23 10:07:43 surprise kernel: RAX: ffff8a758ee6ad00 RBX: ffffffffb4d69160 RCX: 000000000000001f
Apr 23 10:07:43 surprise kernel: RDX: 0000000000000000 RSI: 00000000239f52d0 RDI: 0000000000000000
Apr 23 10:07:43 surprise kernel: RBP: ffffa90d4016fe78 R08: 0000011276ac078f R09: 00000000000148ba
Apr 23 10:07:43 surprise kernel: R10: ffff8a758ee69a00 R11: ffff8a758ee699e0 R12: ffff8a757a318000
Apr 23 10:07:43 surprise kernel: R13: 0000000000000002 R14: 0000000000000002 R15: ffff8a757a318000
Apr 23 10:07:43 surprise kernel: ? cpuidle_enter_state+0xa1/0x450
Apr 23 10:07:43 surprise kernel: cpuidle_enter+0x2e/0x40
Apr 23 10:07:43 surprise kernel: call_cpuidle+0x23/0x40
Apr 23 10:07:43 surprise kernel: do_idle+0x1dd/0x270
Apr 23 10:07:43 surprise kernel: cpu_startup_entry+0x20/0x30
Apr 23 10:07:43 surprise kernel: start_secondary+0x167/0x1c0
Apr 23 10:07:43 surprise kernel: secondary_startup_64+0xa4/0xb0
Apr 23 10:07:43 surprise kernel: ---[ end trace cf93a9794ecfd126 ]---

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: linux-generic 5.4.0.26.32
ProcVersionSignature: Ubuntu 5.4.0-26.30-generic 5.4.30
Uname: Linux 5.4.0-26-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair nvidia_modeset nvidia
ApportVersion: 2.20.11-0ubuntu27
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: i3
Date: Thu Apr 23 10:38:05 2020
InstallationDate: Installed on 2019-05-07 (351 days ago)
InstallationMedia: Ubuntu 18.04.2 LTS "Bionic Beaver" - Release amd64 (20190210)
MachineType: Gigabyte Technology Co., Ltd. B450M DS3H
ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.4.0-26-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash resume=UUID=73909634-a75d-42c9-8f66-a69138690756 vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-5.4.0-26-generic N/A
 linux-backports-modules-5.4.0-26-generic N/A
 linux-firmware 1.187
RfKill:

SourcePackage: linux
UpgradeStatus: Upgraded to focal on 2019-11-15 (159 days ago)
dmi.bios.date: 01/25/2019
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: F4
dmi.board.asset.tag: Default string
dmi.board.name: B450M DS3H-CF
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrF4:bd01/25/2019:svnGigabyteTechnologyCo.,Ltd.:pnB450MDS3H:pvrDefaultstring:rvnGigabyteTechnologyCo.,Ltd.:rnB450MDS3H-CF:rvrx.x:cvnDefaultstring:ct3:cvrDefaultstring:
dmi.product.family: Default string
dmi.product.name: B450M DS3H
dmi.product.sku: Default string
dmi.product.version: Default string
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

Dan Watkins (oddbloke) wrote :

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Dan Watkins (oddbloke) wrote :
Download full text (10.8 KiB)

Here are the NetworkManager logs when I first disabled my networking (note that ensp5s0 does get its link up reported correctly, but no traffic seems to go over it):

Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1626] manager: disable requested (sleeping: no enabled: yes)
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1627] manager: NetworkManager state is now ASLEEP
Apr 23 09:42:54 surprise dbus-daemon[1280]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.9' (uid=0 pid=1281 comm="/usr/sbin/NetworkManager --no-daemon " label="unconfined")
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1654] device (enp5s0): state change: activated -> deactivating (reason 'sleeping', sys-iface-state: 'managed')
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1695] device (virbr0): state change: activated -> deactivating (reason 'sleeping', sys-iface-state: 'external')
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1703] device (lxdbr0): state change: activated -> deactivating (reason 'sleeping', sys-iface-state: 'external')
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1711] device (docker0): state change: activated -> deactivating (reason 'sleeping', sys-iface-state: 'external')
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1718] device (tun0): state change: activated -> deactivating (reason 'sleeping', sys-iface-state: 'external')
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1744] device (mpqemubr0): state change: activated -> deactivating (reason 'sleeping', sys-iface-state: 'external')
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1754] device (tap-2eb5e2d1a05): state change: activated -> deactivating (reason 'sleeping', sys-iface-state: 'external')
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1756] device (mpqemubr0): bridge port tap-2eb5e2d1a05 was detached
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1756] device (tap-2eb5e2d1a05): released from master device mpqemubr0
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1764] device (tap-8b9ffd8bf3f): state change: activated -> deactivating (reason 'sleeping', sys-iface-state: 'external')
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1765] device (mpqemubr0): bridge port tap-8b9ffd8bf3f was detached
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.1765] device (tap-8b9ffd8bf3f): released from master device mpqemubr0
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.2065] device (enp5s0): state change: deactivating -> disconnected (reason 'sleeping', sys-iface-state: 'managed')
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.2124] dhcp4 (enp5s0): canceled DHCP transaction
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.2124] dhcp4 (enp5s0): state changed extended -> done
Apr 23 09:42:54 surprise NetworkManager[1281]: <info> [1587649374.2132] dhcp6 (enp5s0): canceled DHCP tr...

Dan Watkins (oddbloke) wrote :

I'd be happy to assist in debugging this, is there anything I can do to help track it down?

You-Sheng Yang (vicamo) on 2020-04-27
tags: added: hwe-networking-ethernet
Nisalon Caje (nisalon-caje) wrote :
Download full text (7.0 KiB)

I have the exact same bug on focal.

But it happens randomly to me, while my server is under load.
2 of the servers I migrated to focal have the same issue (not in the same DC), so it excludes a hardware issue from ths particular machine.

Here is my syslog
May 26 10:04:15 service01K kernel: [161735.901135] ------------[ cut here ]------------
May 26 10:04:15 service01K kernel: [161735.901136] NETDEV WATCHDOG: eth1 (ixgbe): transmit queue 2 timed out
May 26 10:04:15 service01K kernel: [161735.901145] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x258/0x260
May 26 10:04:15 service01K kernel: [161735.901146] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_multiport isofs ip6table_filter ip6_tables xt_tcpudp xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bpfilter nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ipmi_ssif kvm_intel kvm joydev input_leds ipmi_si ipmi_devintf ipmi_msghandler video acpi_pad acpi_tad sch_fq_codel ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear hid_generic uas usbhid hid usb_storage raid1 crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i2c_algo_bit drm_vram_helper ttm drm_kms_helper syscopyarea aesni_intel sysfillrect sysimgblt crypto_simd fb_sys_fops ixgbe cryptd glue_helper nvme drm xfrm_algo ahci dca mdio libahci nvme_core
May 26 10:04:15 service01K kernel: [161735.901163] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.4.0-31-generic #35-Ubuntu
May 26 10:04:15 service01K kernel: [161735.901163] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./E3C246D4U2-2T, BIOS L2.02K 12/18/2019
May 26 10:04:15 service01K kernel: [161735.901164] RIP: 0010:dev_watchdog+0x258/0x260
May 26 10:04:15 service01K kernel: [161735.901165] Code: 85 c0 75 e5 eb 9f 4c 89 ff c6 05 ef f6 e7 00 01 e8 6d bb fa ff 44 89 e9 4c 89 fe 48 c7 c7 40 73 43 ba 48 89 c2 e8 03 30 71 ff <0f> 0b eb 80 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 41 57 49 89 d7
May 26 10:04:15 service01K kernel: [161735.901165] RSP: 0018:ffffb8774003ce30 EFLAGS: 00010286
May 26 10:04:15 service01K kernel: [161735.901166] RAX: 0000000000000000 RBX: ffff891bdf924ec0 RCX: 0000000000000006
May 26 10:04:15 service01K kernel: [161735.901166] RDX: 0000000000000007 RSI: 0000000000000096 RDI: ffff891bee8578c0
May 26 10:04:15 service01K kernel: [161735.901167] RBP: ffffb8774003ce60 R08: 000000000000046b R09: 0000000000000004
May 26 10:04:15 service01K kernel: [161735.901167] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000040
May 26 10:04:15 service01K kernel: [161735.901167] R13: 0000000000000002 R14: ffff891bdf980480 R15: ffff891bdf980000
May 26 10:04:15 service01K kernel: [161735.901168] FS: 0000000000000000(0000) GS:ffff891bee840000(0000) knlGS:0000000000000000
May 26 10:04:15 service01K kernel: [161735.901168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 10:04:15 service01K kernel: [161735.901169] CR2: 00007fb0077d6148 CR3: 0000000894f1c006 CR4: 00000000003606e0
May 26 10:04:15 service01K kernel: [161735.901169] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 2...

Read more...

Kai-Heng Feng (kaihengfeng) wrote :

Can you please test kernel parameter "pcie_aspm=off"?

Changed in linux (Ubuntu):
status: Confirmed → Incomplete
AceLan Kao (acelankao) wrote :

Dan,

Please try Kai-Heng's suggestion adding "pcie_aspm=off" to see if it helps
If it does, please also attach the following logs
   1. sudo lspci -xxxs 00:01.3
   2. sudo lspci -xxxs 05:00.0

Thanks.

Kai-Heng Feng (kaihengfeng) wrote :

Jeremy, the link you provided is the same as this bug.

Kai-Heng Feng (kaihengfeng) wrote :

Ok, it's LP: #1874464.

Does this issue also happen on System76 platforms?
In addition to L1.1, does it help if L1.2 gets disabled?

AceLan Kao (acelankao) on 2020-06-04
Changed in linux (Ubuntu Focal):
status: New → Incomplete
Rene Meier (meier.rene) wrote :

I can confirm this error on my system after upgrade from bionic to focal. System works "normaly" but error appears in the logs.

Adding "pcie_aspm=off" does not help in my case.

Rene Meier (meier.rene) wrote :
Download full text (4.4 KiB)

here comes the error:

[ 60.259403] ------------[ cut here ]------------
[ 60.259405] NETDEV WATCHDOG: enp5s0 (r8169): transmit queue 0 timed out
[ 60.259416] WARNING: CPU: 6 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x258/0x260
[ 60.259417] Modules linked in: vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter bpfilter aufs overlay nls_iso8859_1 snd_hda_codec_hdmi intel_rapl_msr mei_hdcp snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi intel_rapl_common bridge x86_pkg_temp_thermal intel_powerclamp stp snd_seq llc coretemp kvm_intel snd_seq_device kvm input_leds snd_timer intel_cstate intel_rapl_perf eeepc_wmi asus_wmi sparse_keymap wmi_bmof snd soundcore mei_me mei mac_hid acpi_pad sch_fq_codel parport_pc ppdev lp parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid uas usb_storage crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
[ 60.259438] crypto_simd i915 mxm_wmi i2c_algo_bit cryptd glue_helper drm_kms_helper nvme syscopyarea sysfillrect sysimgblt fb_sys_fops r8169 i2c_i801 drm nvme_core ahci realtek libahci wmi video
[ 60.259444] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G OE 5.4.0-33-generic #37-Ubuntu
[ 60.259445] Hardware name: bluechip Computer AG BUSINESSline/Z170-K, BIOS 3805 05/16/2018
[ 60.259446] RIP: 0010:dev_watchdog+0x258/0x260
[ 60.259447] Code: 85 c0 75 e5 eb 9f 4c 89 ff c6 05 ef f6 e7 00 01 e8 6d bb fa ff 44 89 e9 4c 89 fe 48 c7 c7 40 73 e3 8f 48 89 c2 e8 03 30 71 ff <0f> 0b eb 80 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 41 57 49 89 d7
[ 60.259448] RSP: 0018:ffffaf7200234e30 EFLAGS: 00010286
[ 60.259448] RAX: 0000000000000000 RBX: ffff9c6ded99fa00 RCX: 0000000000000006
[ 60.259449] RDX: 0000000000000007 RSI: 0000000000000096 RDI: ffff9c6df6b978c0
[ 60.259449] RBP: ffffaf7200234e60 R08: 00000000000004a0 R09: 0000000000000004
[ 60.259450] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000001
[ 60.259450] R13: 0000000000000000 R14: ffff9c6defed4480 R15: ffff9c6defed4000
[ 60.259451] FS: 0000000000000000(0000) GS:ffff9c6df6b80000(0000) knlGS:0000000000000000
[ 60.259451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 60.259452] CR2: 00001a62928112d0 CR3: 00000009e1e0a002 CR4: 00000000003606e0
[ 60.259452] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 60.259453] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 60.259453] Call Trace:
[ 60.259454] <IRQ>
[ 60.259456] ? pfifo_fast_enqueue+0x150/0x150
[ 60.259458] call_timer_fn+0x32/0x130
[ 60.259459] __run_timers.part.0+0x180/0x280
[ 60.259461] ? timerqueue_add+0x68/0xb0
[ 60.259462] ? enqueue_hrtimer+0x3d/0x90
[ 60.259464] ? recalibrate_cpu_khz+0x10/0x10
[ 60.259465] ? ktime_get+0x3e/0xa0
[ 60.259466] run_timer_softirq+0x2a/0x50
[ 60.259467] __do_soft...

Read more...

Dan Watkins (oddbloke) wrote :

I'll try the command line change after my morning meetings; here's the requested debug output:

$ sudo lspci -xxxs 00:01.3
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
00: 22 10 53 14 07 04 10 00 00 00 04 06 10 00 81 00
10: 00 00 00 00 00 00 00 00 00 02 06 00 f1 f1 00 20
20: 50 f7 60 f7 21 f2 21 f2 00 00 00 00 00 00 00 00
30: 00 00 00 00 50 00 00 00 00 00 00 00 ff 00 12 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 01 58 03 c8 00 00 00 00 10 a0 42 01 22 80 00 00
60: 1f 29 00 00 43 f8 72 02 40 00 43 70 00 00 04 00
70: 00 00 40 01 18 00 01 00 00 00 00 00 bf 01 70 00
80: 06 00 00 00 0e 00 00 00 03 00 1f 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 05 c0 81 00 00 00 e0 fe 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 0d c8 00 00 22 10 53 14 08 00 03 a8 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

$ sudo lspci -xxxs 05:00.0
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
00: ec 10 68 81 07 04 10 00 0c 00 00 02 10 00 00 00
10: 01 f0 00 00 00 00 00 00 04 00 50 f7 00 00 00 00
20: 0c 00 20 f2 00 00 00 00 00 00 00 00 58 14 00 e0
30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 00 00
40: 01 50 c3 ff 08 00 00 00 00 00 00 00 00 00 00 00
50: 05 70 80 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 b0 02 02 c0 8c 68 00 10 50 19 00 11 7c 47 00
80: 40 00 11 10 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 1f 08 0c 00 00 00 00 00 02 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 11 d0 03 80 04 00 00 00 04 08 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Dan Watkins (oddbloke) wrote :

OK, I just did the following (all on 5.4.0-33-generic):

* modified /etc/default/grub to include " pcie_aspm=off" in the kernel command line
* `update-grub`
* `reboot`
* double-checked that "pcie_aspm=off" was in the kernel command line, then booted
* once booted and logged in, I disabled networking

Unfortunately:

* re-enabling networking did not restore my network connection
* rebooting did not restore my network connection
* only rebooting into Windows fixed my network connection
* I see the same trace on the boots with broken networking as in the original report

AFAICT, the kernel command line option had no effect on the issue at hand.

Anything else I can try?

Kai-Heng Feng (kaihengfeng) wrote :
Kai-Heng Feng (kaihengfeng) wrote :

If latest kernel doesn't work then we need to bisect between Bionic kernel and Focal kernel.

Rene Meier (meier.rene) wrote :

I tried the mainline kernel and the error is gone. What next?

Kai-Heng Feng (kaihengfeng) wrote :

Would it be possible for you to do a reverse kernel bisection?

First, find the first -rc kernel works and the last -rc kernel doesn’t work from http://kernel.ubuntu.com/~kernel-ppa/mainline/

Then,
$ sudo apt build-dep linux
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
$ cd linux
$ git bisect start
$ git bisect new $(the working version you found)
$ git bisect old $(the non-working version found)
$ make localmodconfig
$ make -j`nproc` deb-pkg
Install the newly built kernel, then reboot with it.
If it doesn’t work,
$ git bisect old
Otherwise,
$ git bisect new
Repeat to "make -j`nproc` deb-pkg" until you find the commit fixes the issue.

Dan Watkins (oddbloke) wrote :
Download full text (5.4 KiB)

I have tested both rc3 and rc4; neither addresses the issue for me. I am seeing a slightly different call trace now though (this is from an rc3 boot):

Jul 06 08:44:48 surprise kernel: ------------[ cut here ]------------
Jul 06 08:44:48 surprise kernel: NETDEV WATCHDOG: enp5s0 (r8169): transmit queue 0 timed out
Jul 06 08:44:48 surprise kernel: WARNING: CPU: 15 PID: 0 at net/sched/sch_generic.c:442 dev_watchdog+0x25e/0x270
Jul 06 08:44:48 surprise kernel: Modules linked in: xt_comment dummy xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink bpfilter rdma_ucm ib_uverbs rdma_cm iw_cm ib_cm ib_core overlay snd_hda_codec_realtek snd_hda_codec_generic nls_iso8859_1 ledtrig_audio snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device joydev snd_timer edac_mce_amd ucsi_ccg mousedev input_leds amd_energy typec_ucsi rapl wmi_bmof efi_pstore snd typec k10temp soundcore mac_hid sch_fq_codel kvm_amd ccp kvm iptable_filter ip6table_filter ip6_tables br_netfilter bridge stp llc arp_tables parport_pc ppdev lp parport drm backlight ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c dm_crypt hid_microsoft ff_memless hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid
Jul 06 08:44:48 surprise kernel: crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper r8169 i2c_piix4 nvme ahci i2c_nvidia_gpu realtek xhci_pci nvme_core libahci xhci_pci_renesas wmi gpio_amdpt gpio_generic
Jul 06 08:44:48 surprise kernel: CPU: 15 PID: 0 Comm: swapper/15 Not tainted 5.8.0-050800-generic #202006282330
Jul 06 08:44:48 surprise kernel: Hardware name: Gigabyte Technology Co., Ltd. B450M DS3H/B450M DS3H-CF, BIOS F4 01/25/2019
Jul 06 08:44:48 surprise kernel: RIP: 0010:dev_watchdog+0x25e/0x270
Jul 06 08:44:48 surprise kernel: Code: 85 c0 75 e5 eb 9c 4c 89 ff c6 05 eb ef 22 01 01 e8 d7 d1 fa ff 44 89 e9 4c 89 fe 48 c7 c7 90 28 e8 b7 48 89 c2 e8 17 1a 6a ff <0f> 0b e9 7a ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00
Jul 06 08:44:48 surprise kernel: RSP: 0018:ffffa29e404b4e70 EFLAGS: 00010282
Jul 06 08:44:48 surprise kernel: RAX: 0000000000000000 RBX: ffff943bca232600 RCX: 0000000000000000
Jul 06 08:44:48 surprise kernel: RDX: ffff943bccfe9020 RSI: ffff943bccfd8cd0 RDI: 0000000000000300
Jul 06 08:44:48 surprise kernel: RBP: ffffa29e404b4ea0 R08: 0000000000000545 R09: 0000000000000004
Jul 06 08:44:48 surprise kernel: R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000001
Jul 06 08:44:48 surprise kernel: R13: 0000000000000000 R14: ffff943bca938480 R15: ffff943bca938000
Jul 06 08:44:48 surprise kernel: FS: 0000000000000000(0000) GS:ffff943bccfc0000(0000) knlGS:0000000000000000
Jul 06 08:44:48 surprise kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 06 08:44:48 surprise kernel: CR2: 00007f507eb35160 CR3: 00000003d81f4000 CR4: 00000000003406e0
Jul 06 08:44:48 surprise kernel: Call Trace:
Jul 06 08:44:48 surprise kernel: <IRQ>
Jul 06 08:44...

Read more...

Dan Watkins (oddbloke) wrote :

(I'm now running groovy on this system, in case that changes anything.)

Kai-Heng Feng (kaihengfeng) wrote :

Dan,

Possible to try older kernel releases? Like 4.15?

Dan Watkins (oddbloke) wrote :

> Possible to try older kernel releases? Like 4.15?

I'm happy to try; how would I go about installing them on groovy?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers