igb module crashing server

[Andrew Lowther]

Description: Ubuntu 20.04.6 LTS
Release: 20.04

linux-image-generic:
  Installed: 5.4.0.169.167
  Candidate: 5.4.0.169.167

A server has started to crash with logs that suggest the igb module is the reason. Investigating the logs of the server showed the issue started with kernel
linux-image-5.4.0-166-generic

Because of the timing and the log messages this patch is suspected as the cause
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal/commit/?h=Ubuntu-5.4.0-166.183&id=64285e660b7ab4614b44b19d062944be1c1971de

The crashes are inconsistent. They appear to be triggered by a hardware error which is unexpected. The logs show the hardware error has been happening for a while but only results in a crash with newer kernels.

The hardware is a quad-port NIC
Ethernet controller: Intel Corporation I350 Gigabit Network Connection

########
Up through kernel
5.4.0-165-generic
the kernel logs include the following error and does *not* result in a crash

Nov 01 07:13:38 ksq-a-mon-p13 kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrected (Non-Fatal) error received: 0000:04:00.3
Nov 01 07:13:38 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
Nov 01 07:13:38 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: device [8086:1521] error status/mask=00004000/00000000
Nov 01 07:13:38 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: [14] CmpltTO
Nov 01 07:13:38 ksq-a-mon-p13 kernel: pcieport 0000:00:03.0: AER: Device recovery successful
Nov 01 07:13:41 ksq-a-mon-p13 kernel: igb 0000:04:00.2 enp4s0f2: igb: enp4s0f2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Nov 01 07:13:41 ksq-a-mon-p13 kernel: audit: type=1400 audit(1698837221.513:31): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/sys/devices/pci0000:00/0000:00:03.0/0000:04:00.2/net/enp4s0f2/type" pid=514 comm="sssd" requested_mask="r" denied_mask="r"
 fsuid=0 ouid=0
Nov 01 07:13:42 ksq-a-mon-p13 kernel: igb 0000:04:00.3 enp4s0f3: igb: enp4s0f3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Nov 01 07:13:42 ksq-a-mon-p13 kernel: audit: type=1400 audit(1698837222.301:32): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/sys/devices/pci0000:00/0000:00:03.0/0000:04:00.3/net/enp4s0f3/type" pid=514 comm="sssd" requested_mask="r" denied_mask="r"
 fsuid=0 ouid=0

########
Starting with kernel
5.4.0-166-generic
the kernel logs include the following and the server crashes

Dec 27 09:26:55 ksq-a-mon-p13 kernel: pcieport 0000:00:03.0: AER: Uncorrected (Non-Fatal) error received: 0000:04:00.3
Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: device [8086:1521] error status/mask=00004000/00000000
Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: [14] CmpltTO
Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.0: Non-correctable non-fatal error reported.
Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.1: Non-correctable non-fatal error reported.
Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.2: Non-correctable non-fatal error reported.
Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.3: Non-correctable non-fatal error reported.
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ------------[ cut here ]------------
Dec 27 09:26:55 ksq-a-mon-p13 kernel: kernel BUG at include/linux/netdevice.h:525!
Dec 27 09:26:55 ksq-a-mon-p13 kernel: invalid opcode: 0000 [#1] SMP PTI
Dec 27 09:26:55 ksq-a-mon-p13 kernel: CPU: 4 PID: 131 Comm: irq/26-aerdrv Not tainted 5.4.0-169-generic #187-Ubuntu
Dec 27 09:26:55 ksq-a-mon-p13 kernel: Hardware name: Supermicro SYS-5018A-FTN4/A1SRi, BIOS 2.1 01/18/2018
Dec 27 09:26:55 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb]
Dec 27 09:26:55 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00
Dec 27 09:26:55 ksq-a-mon-p13 kernel: RSP: 0018:ffffab48c02d7ce8 EFLAGS: 00010246
Dec 27 09:26:55 ksq-a-mon-p13 kernel: RAX: ffff9489f40be000 RBX: ffff9489e9d7c8c0 RCX: 0000000000000000
Dec 27 09:26:55 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff9489e94b0b00
Dec 27 09:26:55 ksq-a-mon-p13 kernel: RBP: ffffab48c02d7cf8 R08: 00000000000003d1 R09: ffffab48c08f5000
Dec 27 09:26:55 ksq-a-mon-p13 kernel: R10: 0000000000000168 R11: ffff9489f7b2eba0 R12: ffff9489e9d7c000
Dec 27 09:26:55 ksq-a-mon-p13 kernel: R13: ffff9489e9d7c8c0 R14: ffff9489f5e1d000 R15: ffff9489f5e81000
Dec 27 09:26:55 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff9489f7b00000(0000) knlGS:0000000000000000
Dec 27 09:26:55 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 27 09:26:55 ksq-a-mon-p13 kernel: CR2: 0000561bc783e140 CR3: 00000001fb00a000 CR4: 00000000001006e0
Dec 27 09:26:55 ksq-a-mon-p13 kernel: Call Trace:
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? show_regs.cold+0x1a/0x1f
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? __die+0x90/0xd9
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? die+0x30/0x50
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? do_trap+0x85/0xf0
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? do_error_trap+0x7c/0xc0
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb]
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? do_invalid_op+0x3c/0x50
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb]
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? invalid_op+0x1e/0x30
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb]
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? igb_up+0x14/0x160 [igb]
Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb_io_resume+0x31/0x50 [igb]
Dec 27 09:26:55 ksq-a-mon-p13 kernel: report_resume+0x5c/0x80
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? report_frozen_detected+0x20/0x20
Dec 27 09:26:55 ksq-a-mon-p13 kernel: pci_walk_bus+0x75/0x90
Dec 27 09:26:55 ksq-a-mon-p13 kernel: pcie_do_recovery+0x163/0x280
Dec 27 09:26:55 ksq-a-mon-p13 kernel: aer_process_err_devices+0xa2/0xd1
Dec 27 09:26:55 ksq-a-mon-p13 kernel: aer_isr.cold+0x52/0xa1
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? irq_finalize_oneshot.part.0+0xf0/0xf0
Dec 27 09:26:55 ksq-a-mon-p13 kernel: irq_thread_fn+0x28/0x60
Dec 27 09:26:55 ksq-a-mon-p13 kernel: irq_thread+0xf8/0x190
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? irq_forced_thread_fn+0x90/0x90
Dec 27 09:26:55 ksq-a-mon-p13 kernel: kthread+0x104/0x140
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? irq_thread_check_affinity+0xf0/0xf0
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? kthread_park+0x90/0x90
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ret_from_fork+0x35/0x40
Dec 27 09:26:55 ksq-a-mon-p13 kernel: Modules linked in: nfnetlink bluetooth ecdh_generic ecc ipmi_ssif intel_powerclamp coretemp kvm_intel kvm ast crct10dif_pclmul ghash_clmulni_intel aesni_intel drm_vram_helper ttm crypto_simd cryptd glue_helper intel_cstate joydev input_leds drm_kms_helper fb_sys_fops syscopyarea ipmi_si sysfillrect sysimgblt ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel ramoops reed_solomon drm efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid gpio_ich crc32_pclmul igb ahci lpc_ich i2c_i801 libahci i2c_ismt i2c_algo_bit dca
Dec 27 09:26:55 ksq-a-mon-p13 kernel: ---[ end trace 26c0760dce96415f ]---
Dec 27 09:26:55 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb]
Dec 27 09:26:55 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00
Dec 27 09:26:55 ksq-a-mon-p13 kernel: RSP: 0018:ffffab48c02d7ce8 EFLAGS: 00010246
Dec 27 09:26:55 ksq-a-mon-p13 kernel: RAX: ffff9489f40be000 RBX: ffff9489e9d7c8c0 RCX: 0000000000000000
Dec 27 09:26:55 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff9489e94b0b00
Dec 27 09:26:55 ksq-a-mon-p13 kernel: RBP: ffffab48c02d7cf8 R08: 00000000000003d1 R09: ffffab48c08f5000
Dec 27 09:26:55 ksq-a-mon-p13 kernel: R10: 0000000000000168 R11: ffff9489f7b2eba0 R12: ffff9489e9d7c000
Dec 27 09:26:55 ksq-a-mon-p13 kernel: R13: ffff9489e9d7c8c0 R14: ffff9489f5e1d000 R15: ffff9489f5e81000
Dec 27 09:26:55 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff9489f7b00000(0000) knlGS:0000000000000000
Dec 27 09:26:55 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 27 09:26:55 ksq-a-mon-p13 kernel: CR2: 0000561bc783e140 CR3: 00000001fb00a000 CR4: 00000000001006e0
Dec 27 09:26:55 ksq-a-mon-p13 kernel: BUG: stack guard page was hit at 0000000042cc7117 (stack is 0000000046d88908..0000000062c2f40a)

########
The behavior is not always consistent. For example, running kernel 5.4.0-166-generic the following was logged and the server did *not* crash

Nov 13 18:30:22 ksq-a-mon-p13 kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrected (Non-Fatal) error received: 0000:04:00.3
Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: device [8086:1521] error status/mask=00004000/00000000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: [14] CmpltTO
Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.0: Non-correctable non-fatal error reported.
Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.1: Non-correctable non-fatal error reported.
Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.2: Non-correctable non-fatal error reported.
Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.3: Non-correctable non-fatal error reported.
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ------------[ cut here ]------------
Nov 13 18:30:22 ksq-a-mon-p13 kernel: kernel BUG at include/linux/netdevice.h:525!
Nov 13 18:30:22 ksq-a-mon-p13 kernel: invalid opcode: 0000 [#1] SMP PTI
Nov 13 18:30:22 ksq-a-mon-p13 kernel: CPU: 4 PID: 131 Comm: irq/26-aerdrv Not tainted 5.4.0-166-generic #183-Ubuntu
Nov 13 18:30:22 ksq-a-mon-p13 kernel: Hardware name: Supermicro SYS-5018A-FTN4/A1SRi, BIOS 2.1 01/18/2018
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb]
Nov 13 18:30:22 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RSP: 0018:ffff9ad4402d7ce8 EFLAGS: 00010246
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RAX: ffff89d9aa57b800 RBX: ffff89d9a9f9c8c0 RCX: 0000000000000000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff89d9aa57cf00
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RBP: ffff9ad4402d7cf8 R08: 0000000000000001 R09: ffff9ad44088d000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: R10: ffff89d9b7b2ebc0 R11: ffff89d9b7b2eba0 R12: ffff89d9a9f9c000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: R13: ffff89d9a9f9c8c0 R14: ffff89d9b5e19800 R15: ffff89d9b5e80000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff89d9b7b00000(0000) knlGS:0000000000000000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 13 18:30:22 ksq-a-mon-p13 kernel: CR2: 00007fc0857570a0 CR3: 00000001e3c0a000 CR4: 00000000001006e0
Nov 13 18:30:22 ksq-a-mon-p13 kernel: Call Trace:
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? show_regs.cold+0x1a/0x1f
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? __die+0x90/0xd9
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? die+0x30/0x50
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? do_trap+0x85/0xf0
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? do_error_trap+0x7c/0xc0
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb]
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? do_invalid_op+0x3c/0x50
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb]
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? invalid_op+0x1e/0x30
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb]
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? igb_up+0x14/0x160 [igb]
Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb_io_resume+0x31/0x50 [igb]
Nov 13 18:30:22 ksq-a-mon-p13 kernel: report_resume+0x5c/0x80
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? report_frozen_detected+0x20/0x20
Nov 13 18:30:22 ksq-a-mon-p13 kernel: pci_walk_bus+0x75/0x90
Nov 13 18:30:22 ksq-a-mon-p13 kernel: pcie_do_recovery+0x163/0x280
Nov 13 18:30:22 ksq-a-mon-p13 kernel: aer_process_err_devices+0xa2/0xd1
Nov 13 18:30:22 ksq-a-mon-p13 kernel: aer_isr.cold+0x52/0xa1
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? irq_finalize_oneshot.part.0+0xf0/0xf0
Nov 13 18:30:22 ksq-a-mon-p13 kernel: irq_thread_fn+0x28/0x60
Nov 13 18:30:22 ksq-a-mon-p13 kernel: irq_thread+0xf8/0x190
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? irq_forced_thread_fn+0x90/0x90
Nov 13 18:30:22 ksq-a-mon-p13 kernel: kthread+0x104/0x140
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? irq_thread_check_affinity+0xf0/0xf0
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? kthread_park+0x90/0x90
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ret_from_fork+0x35/0x40
Nov 13 18:30:22 ksq-a-mon-p13 kernel: Modules linked in: nfnetlink bluetooth ecdh_generic ecc ipmi_ssif intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul ghash_clmulni_intel ast drm_vram_helper ttm input_leds joydev aesni_intel drm_kms_helper crypto_simd fb_sys_fo
ps cryptd glue_helper syscopyarea sysfillrect intel_cstate sysimgblt ipmi_si ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel ramoops reed_solomon drm efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid gpio_ich ahci crc32_pclmul i2c_i801 lpc_ich libahci igb i2
c_algo_bit i2c_ismt dca
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ---[ end trace 9ec172007066a0c4 ]---
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb]
Nov 13 18:30:22 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RSP: 0018:ffff9ad4402d7ce8 EFLAGS: 00010246
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RAX: ffff89d9aa57b800 RBX: ffff89d9a9f9c8c0 RCX: 0000000000000000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff89d9aa57cf00
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RBP: ffff9ad4402d7cf8 R08: 0000000000000001 R09: ffff9ad44088d000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: R10: ffff89d9b7b2ebc0 R11: ffff89d9b7b2eba0 R12: ffff89d9a9f9c000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: R13: ffff89d9a9f9c8c0 R14: ffff89d9b5e19800 R15: ffff89d9b5e80000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff89d9b7b00000(0000) knlGS:0000000000000000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 13 18:30:22 ksq-a-mon-p13 kernel: CR2: 00007fc0857570a0 CR3: 00000001e3c0a000 CR4: 00000000001006e0
Nov 13 18:30:22 ksq-a-mon-p13 kernel: BUG: stack guard page was hit at 00000000844f184b (stack is 00000000f8764a81..00000000b1472243)
Nov 13 18:30:22 ksq-a-mon-p13 kernel: kernel stack overflow (page fault): 0000 [#2] SMP PTI
Nov 13 18:30:22 ksq-a-mon-p13 kernel: CPU: 4 PID: 131 Comm: irq/26-aerdrv Tainted: G D 5.4.0-166-generic #183-Ubuntu
Nov 13 18:30:22 ksq-a-mon-p13 kernel: Hardware name: Supermicro SYS-5018A-FTN4/A1SRi, BIOS 2.1 01/18/2018
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RIP: 0010:mutex_lock+0x1e/0x40
Nov 13 18:30:22 ksq-a-mon-p13 kernel: Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc e8 0d e8 ff ff 31 c0 65 48 8b 14 25 c0 bb 01 00 <f0> 49 0f b1 14 24 74 08 4c 89 e7 e8 b2 ff ff ff 41 5c 5d c3 66 66
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RSP: 0018:ffff9ad4402d7df8 EFLAGS: 00010246
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RAX: 0000000000000000 RBX: ffff9ad4402d7f38 RCX: 0000000000000000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RDX: ffff89d9b5d02c80 RSI: 0000000000000000 RDI: ffff9ad4402d8c80
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RBP: ffff9ad4402d7e00 R08: ffffffffbc05f080 R09: 0000000000000000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: R10: ffff89d9b625f380 R11: 0000000000000001 R12: ffff9ad4402d8c80
Nov 13 18:30:22 ksq-a-mon-p13 kernel: R13: ffff89d9b5d037ec R14: ffff89d9b5d037b0 R15: ffff9ad4402d7e80
Nov 13 18:30:22 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff89d9b7b00000(0000) knlGS:0000000000000000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 13 18:30:22 ksq-a-mon-p13 kernel: CR2: ffff9ad4402d8c80 CR3: 00000001e3c0a000 CR4: 00000000001006e0
Nov 13 18:30:22 ksq-a-mon-p13 kernel: Call Trace:
Nov 13 18:30:22 ksq-a-mon-p13 kernel: <#DF>
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? show_regs.cold+0x1a/0x1f
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? __die+0x90/0xd9
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? die+0x30/0x50
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? handle_stack_overflow+0x43/0x52
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? no_context+0x23b/0x380
Nov 13 18:30:22 ksq-a-mon-p13 kernel: </#DF>
Nov 13 18:30:22 ksq-a-mon-p13 kernel: Modules linked in: nfnetlink bluetooth ecdh_generic ecc ipmi_ssif intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul ghash_clmulni_intel ast drm_vram_helper ttm input_leds joydev aesni_intel drm_kms_helper crypto_simd fb_sys_fops cryptd glue_helper syscopyarea sysfillrect intel_cstate sysimgblt ipmi_si ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel ramoops reed_solomon drm efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid gpio_ich ahci crc32_pclmul i2c_i801 lpc_ich libahci igb i2c_algo_bit i2c_ismt dca
Nov 13 18:30:22 ksq-a-mon-p13 kernel: ---[ end trace 9ec172007066a0c5 ]---
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb]
Nov 13 18:30:22 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RSP: 0018:ffff9ad4402d7ce8 EFLAGS: 00010246
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RAX: ffff89d9aa57b800 RBX: ffff89d9a9f9c8c0 RCX: 0000000000000000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff89d9aa57cf00
Nov 13 18:30:22 ksq-a-mon-p13 kernel: RBP: ffff9ad4402d7cf8 R08: 0000000000000001 R09: ffff9ad44088d000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: R10: ffff89d9b7b2ebc0 R11: ffff89d9b7b2eba0 R12: ffff89d9a9f9c000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: R13: ffff89d9a9f9c8c0 R14: ffff89d9b5e19800 R15: ffff89d9b5e80000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff89d9b7b00000(0000) knlGS:0000000000000000
Nov 13 18:30:22 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 13 18:30:22 ksq-a-mon-p13 kernel: CR2: ffff9ad4402d8c80 CR3: 00000001e3c0a000 CR4: 00000000001006e0
Nov 13 18:30:22 ksq-a-mon-p13 kernel: Fixing recursive fault but reboot is needed!

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: linux-image-5.4.0-169-generic 5.4.0-169.187
ProcVersionSignature: Ubuntu 5.4.0-169.187-generic 5.4.257
Uname: Linux 5.4.0-169-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jan 13 09:56 seq
 crw-rw---- 1 root audio 116, 33 Jan 13 09:56 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu27.27
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckResult: skip
Date: Thu Jan 18 16:51:14 2024
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: Supermicro SYS-5018A-FTN4
PciMultimedia:

ProcEnviron:
 LANGUAGE=en_US:
 TERM=screen-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.4.0-169-generic root=/dev/mapper/systemvg-rootlv ro crashkernel=512M-:192M
RelatedPackageVersions:
 linux-restricted-modules-5.4.0-169-generic N/A
 linux-backports-modules-5.4.0-169-generic N/A
 linux-firmware 1.187.39
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/18/2018
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 2.1
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: A1SRi
dmi.board.vendor: Supermicro
dmi.board.version: 123456789
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 1
dmi.chassis.vendor: Supermicro
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2.1:bd01/18/2018:svnSupermicro:pnSYS-5018A-FTN4:pvr123456789:rvnSupermicro:rnA1SRi:rvr123456789:cvnSupermicro:ct1:cvrToBeFilledByO.E.M.:
dmi.product.family: SMC X10
dmi.product.name: SYS-5018A-FTN4
dmi.product.sku: 081315D9
dmi.product.version: 123456789
dmi.sys.vendor: Supermicro