Hate to say it but it happened again. Only once which is a lot better in terms of frequency but still happening, here are details:
[23144.764734] hrtimer: interrupt took 46767 ns [41628.563552] e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang: TDH <db> TDT <65> next_to_use <65> next_to_clean <da> buffer_info[next_to_clean]: time_stamp <1027691ea> next_to_watch <db> jiffies <102769c40> next_to_watch.status <0> MAC Status <80083> PHY Status <796d> PHY 1000BASE-T Status <7c00> PHY Extended Status <3000> PCI Status <10> [41630.611608] e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang: TDH <db> TDT <65> next_to_use <65> next_to_clean <da> buffer_info[next_to_clean]: time_stamp <1027691ea> next_to_watch <db> jiffies <10276a440> next_to_watch.status <0> MAC Status <80083> PHY Status <796d> PHY 1000BASE-T Status <7c00> PHY Extended Status <3000> PCI Status <10> [41632.595800] e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang: TDH <db> TDT <65> next_to_use <65> next_to_clean <da> buffer_info[next_to_clean]: time_stamp <1027691ea> next_to_watch <db> jiffies <10276ac00> next_to_watch.status <0> MAC Status <80083> PHY Status <796d> PHY 1000BASE-T Status <7c00> PHY Extended Status <3000> PCI Status <10> [41634.579772] e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang: TDH <db> TDT <65> next_to_use <65> next_to_clean <da> buffer_info[next_to_clean]: time_stamp <1027691ea> next_to_watch <db> jiffies <10276b3c0> next_to_watch.status <0> MAC Status <80083> PHY Status <796d> PHY 1000BASE-T Status <7c00> PHY Extended Status <3000> PCI Status <10> [41635.667409] ------------[ cut here ]------------ [41635.667411] NETDEV WATCHDOG: eno1 (e1000e): transmit queue 0 timed out [41635.667424] WARNING: CPU: 9 PID: 65 at /build/linux-5s7Xkn/linux-4.15.0/net/sched/sch_generic.c:323 dev_watchdog+0x21d/0x230 [41635.667424] Modules linked in: tcp_diag inet_diag vhost_net vhost tap xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 nfsv4 nfs fscache msr bridge stp llc binfmt_misc quota_v2 quota_tree nls_iso8859_1 intel_rapl x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc snd_hda_codec_hdmi aesni_intel aes_x86_64 crypto_simd snd_hda_codec_realtek snd_hda_codec_generic glue_helper snd_hda_intel cryptd input_leds snd_hda_codec intel_cstate intel_rapl_perf snd_hda_core snd_hwdep snd_seq_midi snd_seq_midi_event snd_rawmidi [41635.667448] snd_seq snd_seq_device snd_pcm eeepc_wmi asus_wmi snd_timer sparse_keymap wmi_bmof intel_wmi_thunderbolt mei_me snd soundcore lpc_ich shpchp mei mac_hid sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nct6775 hwmon_vid nfsd coretemp auth_rpcgss nfs_acl lockd grace sunrpc parport_pc ppdev lp parport ip_tables x_tables autofs4 btrfs zstd_compress raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror dm_region_hash dm_log hid_generic usbhid hid raid10 nouveau video i2c_algo_bit ttm drm_kms_helper mxm_wmi syscopyarea sysfillrect sysimgblt e1000e fb_sys_fops ahci drm libahci ptp pps_core wmi [41635.667479] CPU: 9 PID: 65 Comm: ksoftirqd/9 Not tainted 4.15.0-20-lowlatency #21-Ubuntu [41635.667480] Hardware name: ASUS All Series/X99-E, BIOS 1801 08/11/2017 [41635.667481] RIP: 0010:dev_watchdog+0x21d/0x230 [41635.667482] RSP: 0018:ffffa1a9064fbd60 EFLAGS: 00010282 [41635.667483] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006 [41635.667483] RDX: 0000000000000007 RSI: 0000000000000096 RDI: ffff93b8bf456490 [41635.667484] RBP: ffffa1a9064fbd90 R08: 0000000000000527 R09: 0000000000000004 [41635.667484] R10: ffffa1a9064fbde8 R11: 0000000000000001 R12: ffff93b8ab370c80 [41635.667485] R13: ffff93b8aa51c000 R14: ffff93b8aa51c478 R15: 0000000000000001 [41635.667486] FS: 0000000000000000(0000) GS:ffff93b8bf440000(0000) knlGS:0000000000000000 [41635.667486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41635.667487] CR2: 0000000002b16980 CR3: 0000001ba4a0a004 CR4: 00000000003626e0 [41635.667488] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41635.667488] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41635.667488] Call Trace: [41635.667492] ? qdisc_reset+0x70/0x70 [41635.667496] call_timer_fn+0x30/0x160 [41635.667497] ? qdisc_reset+0x70/0x70 [41635.667498] run_timer_softirq+0x422/0x470 [41635.667502] ? __switch_to+0x4c6/0x530 [41635.667503] ? __switch_to+0x4c6/0x530 [41635.667506] __do_softirq+0xdf/0x2e4 [41635.667509] run_ksoftirqd+0x20/0x60 [41635.667510] smpboot_thread_fn+0x131/0x1f0 [41635.667512] kthread+0x121/0x140 [41635.667513] ? sort_range+0x30/0x30 [41635.667514] ? kthread_create_worker_on_cpu+0x70/0x70 [41635.667515] ret_from_fork+0x35/0x40 [41635.667516] Code: 37 00 49 63 4e e8 eb 92 4c 89 ef c6 05 3b 1c dc 00 01 e8 67 34 fd ff 89 d9 48 89 c2 4c 89 ee 48 c7 c7 a0 74 d9 b2 e8 83 30 80 ff <0f> 0b eb c0 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f [41635.667533] ---[ end trace 52f8fb6536c9e574 ]--- [41635.667544] e1000e 0000:00:19.0 eno1: Reset adapter unexpectedly [41635.667674] bridge0: port 1(eno1) entered disabled state [41635.667704] bridge0: topology change detected, propagating [41639.839470] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [41639.839503] bridge0: port 1(eno1) entered blocking state [41639.839507] bridge0: port 1(eno1) entered listening state [41655.123763] bridge0: port 1(eno1) entered learning state [41670.485859] bridge0: port 1(eno1) entered forwarding state [41670.485860] bridge0: topology change detected, sending tcn bpdu
Hate to say it but it happened again. Only once which is a lot better in terms of frequency but still happening, here are details:
[23144.764734] hrtimer: interrupt took 46767 ns
TDH <db>
TDT <65>
next_ to_use <65>
next_ to_clean <da>
buffer_ info[next_ to_clean] :
time_ stamp <1027691ea>
next_ to_watch <db>
jiffies <102769c40>
next_ to_watch. status <0>
TDH <db>
TDT <65>
next_ to_use <65>
next_ to_clean <da>
buffer_ info[next_ to_clean] :
time_ stamp <1027691ea>
next_ to_watch <db>
jiffies <10276a440>
next_ to_watch. status <0>
TDH <db>
TDT <65>
next_ to_use <65>
next_ to_clean <da>
buffer_ info[next_ to_clean] :
time_ stamp <1027691ea>
next_ to_watch <db>
jiffies <10276ac00>
next_ to_watch. status <0>
TDH <db>
TDT <65>
next_ to_use <65>
next_ to_clean <da>
buffer_ info[next_ to_clean] :
time_ stamp <1027691ea>
next_ to_watch <db>
jiffies <10276b3c0>
next_ to_watch. status <0> linux-5s7Xkn/ linux-4. 15.0/net/ sched/sch_ generic. c:323 dev_watchdog+ 0x21d/0x230 masquerade_ ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 nfsv4 nfs fscache msr bridge stp llc binfmt_misc quota_v2 quota_tree nls_iso8859_1 intel_rapl x86_pkg_ temp_thermal intel_powerclamp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc snd_hda_codec_hdmi aesni_intel aes_x86_64 crypto_simd snd_hda_ codec_realtek snd_hda_ codec_generic glue_helper snd_hda_intel cryptd input_leds snd_hda_codec intel_cstate intel_rapl_perf snd_hda_core snd_hwdep snd_seq_midi snd_seq_midi_event snd_rawmidi thunderbolt mei_me snd soundcore lpc_ich shpchp mei mac_hid sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_ iscsi nct6775 hwmon_vid nfsd coretemp auth_rpcgss nfs_acl lockd grace sunrpc parport_pc ppdev lp parport ip_tables x_tables autofs4 btrfs zstd_compress raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror dm_region_hash dm_log hid_generic usbhid hid raid10 nouveau video i2c_algo_bit ttm drm_kms_helper mxm_wmi syscopyarea sysfillrect sysimgblt e1000e fb_sys_fops ahci drm libahci ptp pps_core wmi 20-lowlatency #21-Ubuntu watchdog+ 0x21d/0x230 4fbd60 EFLAGS: 00010282 0(0000) GS:ffff93b8bf44 0000(0000) knlGS:000000000 0000000 0x70/0x70 fn+0x30/ 0x160 0x70/0x70 softirq+ 0x422/0x470 to+0x4c6/ 0x530 to+0x4c6/ 0x530 0xdf/0x2e4 0x20/0x60 thread_ fn+0x131/ 0x1f0 0x30/0x30 create_ worker_ on_cpu+ 0x70/0x70 fork+0x35/ 0x40
[41628.563552] e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
MAC Status <80083>
PHY Status <796d>
PHY 1000BASE-T Status <7c00>
PHY Extended Status <3000>
PCI Status <10>
[41630.611608] e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
MAC Status <80083>
PHY Status <796d>
PHY 1000BASE-T Status <7c00>
PHY Extended Status <3000>
PCI Status <10>
[41632.595800] e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
MAC Status <80083>
PHY Status <796d>
PHY 1000BASE-T Status <7c00>
PHY Extended Status <3000>
PCI Status <10>
[41634.579772] e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
MAC Status <80083>
PHY Status <796d>
PHY 1000BASE-T Status <7c00>
PHY Extended Status <3000>
PCI Status <10>
[41635.667409] ------------[ cut here ]------------
[41635.667411] NETDEV WATCHDOG: eno1 (e1000e): transmit queue 0 timed out
[41635.667424] WARNING: CPU: 9 PID: 65 at /build/
[41635.667424] Modules linked in: tcp_diag inet_diag vhost_net vhost tap xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_
[41635.667448] snd_seq snd_seq_device snd_pcm eeepc_wmi asus_wmi snd_timer sparse_keymap wmi_bmof intel_wmi_
[41635.667479] CPU: 9 PID: 65 Comm: ksoftirqd/9 Not tainted 4.15.0-
[41635.667480] Hardware name: ASUS All Series/X99-E, BIOS 1801 08/11/2017
[41635.667481] RIP: 0010:dev_
[41635.667482] RSP: 0018:ffffa1a906
[41635.667483] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006
[41635.667483] RDX: 0000000000000007 RSI: 0000000000000096 RDI: ffff93b8bf456490
[41635.667484] RBP: ffffa1a9064fbd90 R08: 0000000000000527 R09: 0000000000000004
[41635.667484] R10: ffffa1a9064fbde8 R11: 0000000000000001 R12: ffff93b8ab370c80
[41635.667485] R13: ffff93b8aa51c000 R14: ffff93b8aa51c478 R15: 0000000000000001
[41635.667486] FS: 000000000000000
[41635.667486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[41635.667487] CR2: 0000000002b16980 CR3: 0000001ba4a0a004 CR4: 00000000003626e0
[41635.667488] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[41635.667488] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[41635.667488] Call Trace:
[41635.667492] ? qdisc_reset+
[41635.667496] call_timer_
[41635.667497] ? qdisc_reset+
[41635.667498] run_timer_
[41635.667502] ? __switch_
[41635.667503] ? __switch_
[41635.667506] __do_softirq+
[41635.667509] run_ksoftirqd+
[41635.667510] smpboot_
[41635.667512] kthread+0x121/0x140
[41635.667513] ? sort_range+
[41635.667514] ? kthread_
[41635.667515] ret_from_
[41635.667516] Code: 37 00 49 63 4e e8 eb 92 4c 89 ef c6 05 3b 1c dc 00 01 e8 67 34 fd ff 89 d9 48 89 c2 4c 89 ee 48 c7 c7 a0 74 d9 b2 e8 83 30 80 ff <0f> 0b eb c0 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
[41635.667533] ---[ end trace 52f8fb6536c9e574 ]---
[41635.667544] e1000e 0000:00:19.0 eno1: Reset adapter unexpectedly
[41635.667674] bridge0: port 1(eno1) entered disabled state
[41635.667704] bridge0: topology change detected, propagating
[41639.839470] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[41639.839503] bridge0: port 1(eno1) entered blocking state
[41639.839507] bridge0: port 1(eno1) entered listening state
[41655.123763] bridge0: port 1(eno1) entered learning state
[41670.485859] bridge0: port 1(eno1) entered forwarding state
[41670.485860] bridge0: topology change detected, sending tcn bpdu