Kernel Panic on shutdown / reboot on NUC8I3BEH

[b]

Kernel 5.4.0-146 panics on power-down (e.g. shutdown -P now)

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: linux-image-5.4.0-146-generic 5.4.0-146.163
ProcVersionSignature: Ubuntu 5.4.0-144.161-generic 5.4.229
Uname: Linux 5.4.0-144-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.26
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: bbogart 1051 F.... pulseaudio
CasperMD5CheckResult: skip
CurrentDesktop: XFCE
Date: Sat Apr 15 09:07:33 2023
InstallationDate: Installed on 2019-02-09 (1526 days ago)
InstallationMedia: Xubuntu 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180725)
MachineType: Intel(R) Client Systems NUC8i3BEH
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-144-generic root=UUID=9c982b36-8142-4719-810a-e06f81cab223 ro quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-5.4.0-144-generic N/A
 linux-backports-modules-5.4.0-144-generic N/A
 linux-firmware 1.187.38
SourcePackage: linux
UpgradeStatus: Upgraded to focal on 2022-06-30 (289 days ago)
dmi.bios.date: 02/14/2023
dmi.bios.vendor: Intel Corp.
dmi.bios.version: BECFL357.86A.0092.2023.0214.1114
dmi.board.name: NUC8BEB
dmi.board.vendor: Intel Corporation
dmi.board.version: J72693-304
dmi.chassis.type: 3
dmi.chassis.vendor: Intel Corporation
dmi.chassis.version: 2.0
dmi.modalias: dmi:bvnIntelCorp.:bvrBECFL357.86A.0092.2023.0214.1114:bd02/14/2023:svnIntel(R)ClientSystems:pnNUC8i3BEH:pvrJ72753-303:rvnIntelCorporation:rnNUC8BEB:rvrJ72693-304:cvnIntelCorporation:ct3:cvr2.0:
dmi.product.family: Intel NUC
dmi.product.name: NUC8i3BEH
dmi.product.sku: BOXNUC8i3BEH
dmi.product.version: J72753-303
dmi.sys.vendor: Intel(R) Client Systems

[b (ben-ekran)]

This issue seems to persist falling back to kernel 5.4.0-144

[b (ben-ekran)]

Photo of console showing panic trace using 5.4.0-144

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[b (ben-ekran)]

hmm. The issue persists going back to 5.4.0-135 also...

Trying 5.4.0-131 next, but if that it's still happening the issue may be else ware?

I don't reboot or shutdown this machine very often, the last time I can confirm I shut down without a panic was around March 14th.

[b (ben-ekran)]

Same thing with 131. So I have no idea what's going on. I upgraded the RAM in this machine around March 14th, which is why I know that was the last time I rebooted and I believe I would have noticed a panic.

I also confirm it seems all reboots result in the same panic (e.g. "sudo reboot" and also using the XFCE gui to shutdown or reboot).

Looking at my dpkg output, it looks like I've installed a lot of kernels and presumably they all would have required a reboot, I only noticed this kernel panic on shutdown yesterday.

rc linux-image-5.4.0-121-generic 5.4.0-121.137 amd64 Signed ke>
rc linux-image-5.4.0-122-generic 5.4.0-122.138 amd64 Signed ke>
rc linux-image-5.4.0-124-generic 5.4.0-124.140 amd64 Signed ke>
rc linux-image-5.4.0-126-generic 5.4.0-126.142 amd64 Signed ke>
ii linux-image-5.4.0-131-generic 5.4.0-131.147 amd64 Signed ke>
rc linux-image-5.4.0-135-generic 5.4.0-135.152 amd64 Signed ke>
ii linux-image-5.4.0-144-generic 5.4.0-144.161 amd64 Signed ke>
ii linux-image-5.4.0-146-generic 5.4.0-146.163 amd64 Signed ke>

I'll try 121 as a final attempt.

[b (ben-ekran)]

Confirmed on 121.

Another thing I realized is that I also updated the BIOS when I upgraded the RAM. Updated to BE0092. I have no idea what the previous BIOS version I was using (whatever shipped with the machine initially), but this one sill seems the most up to date version.

I'll try downgrading BIOS to BE0090.

[b (ben-ekran)]

Same thing with BIOS 0090 and kernel 5.4.0-135

[b (ben-ekran)]

I was unable to go back to an older version of the BIOS; both BE0088 and BE0089 resulted in the following error:

"Incompatible BIOS version, Update Aborted!"

This happened both using the built in UEFI BIOS update option and running IFLASH2 in FreeDOS booting in legacy mode.

[b (ben-ekran)]

Confirmed on 5.4.0-147

[b (ben-ekran)]

Problem persists using BIOS 0089, trying 0088 next...

[b (ben-ekran)]

Issue persists using BIOS 0088 and 0089.

[b (ben-ekran)]

I was advised to use a different method of flashing the bios ("BIOS recovery" by holding down the power on button with USB including BIOS inserted) on the Intel forum (https://community.intel.com/t5/Intel-NUCs/Linux-kernel-panics-on-reboot-and-shutdown-on-NUC8I3BEH-using/m-p/1477101/emcs_t/S2h8ZW1haWx8dG9waWNfc3Vic2NyaXB0aW9ufExHSTlaMk5RS0FYSlZRfDE0NzcxMDF8U1VCU0NSSVBUSU9OU3xoSw#M99529)

The issue persists with BIOS 0092 and 5.4.0-147.

[b (ben-ekran)]

The issue persists with 5.4.0-148

[b (ben-ekran)]

The issue persists with 5.4.0-152

[Kai-Heng Feng (kaihengfeng)]

Please install later kernel by issuing `sudo apt install linux-generic-hwe-20.04` and reboot.

[b (ben-ekran)]

The issue persists using kernel 5.15.0-76, unfortunately, the console
remains blank, no panic message is printed but the machine stays on
(power light stays on) and is unresponsive. Holding down the power
button is the only action possible.

Could not find anything in the log files, looks like the machine totally
locks up now rather than having a panic.

On 2023-07-13 17:49, Kai-Heng Feng wrote:
> Please install later kernel by issuing `sudo apt install linux-generic-
> hwe-20.04` and reboot.
>

--
B. Bogart, PhD
they/them
www.ekran.org

[b (ben-ekran)]

Using 5.15.0-76-generic, this is all the ends up in kern.log after shutdown -P now (note the lack of panic) the system still locks up:

Jul 30 10:54:43 domi kernel: [92497.892330] e1000e 0000:00:1f.6 eno1: NIC Link is Down
Jul 30 10:54:45 domi kernel: [92499.676912] e1000e 0000:00:1f.6 eno1: NIC Link is Up 100 Mbps Full Duplex, Flow Control: Rx/Tx
Jul 30 10:54:45 domi kernel: [92499.676925] e1000e 0000:00:1f.6 eno1: 10/100 speed: disabling TSO
Jul 30 10:54:45 domi kernel: [92499.677063] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready
Jul 30 10:54:47 domi kernel: [92501.265080] wlp0s20f3: authenticate with 9c:1e:95:fa:fc:76
Jul 30 10:54:47 domi kernel: [92501.267044] wlp0s20f3: send auth to 9c:1e:95:fa:fc:76 (try 1/3)
Jul 30 10:54:47 domi kernel: [92501.306897] wlp0s20f3: authenticated
Jul 30 10:54:47 domi kernel: [92501.308672] wlp0s20f3: associate with 9c:1e:95:fa:fc:76 (try 1/3)
Jul 30 10:54:47 domi kernel: [92501.311769] wlp0s20f3: RX AssocResp from 9c:1e:95:fa:fc:76 (capab=0x1511 status=0 aid=99)
Jul 30 10:54:47 domi kernel: [92501.314875] wlp0s20f3: associated
Jul 30 10:54:47 domi kernel: [92501.338878] wlp0s20f3: Limiting TX power to 30 (30 - 0) dBm as advertised by 9c:1e:95:fa:fc:76
Jul 30 10:54:47 domi kernel: [92501.477160] IPv6: ADDRCONF(NETDEV_CHANGE): wlp0s20f3: link becomes ready
Jul 30 10:57:18 domi kernel: [92652.328803] audit: type=1400 audit(1690739838.317:91): apparmor="ALLOWED" operation="open" profile="libreoffice-soffice" name="/home/bbogart/.thunderbird/profiles.ini" pid=100668 comm="soffice.bin" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
Jul 30 10:57:18 domi kernel: [92652.339332] audit: type=1400 audit(1690739838.329:92): apparmor="ALLOWED" operation="open" profile="libreoffice-soffice" name="/home/bbogart/.thunderbird/fp6psh9b.default/cert9.db" pid=100668 comm="soffice.bin" requested_mask="wrc" denied_mask="wrc" fsuid=1000 ouid=1000
Jul 30 10:57:18 domi kernel: [92652.339559] audit: type=1400 audit(1690739838.329:93): apparmor="ALLOWED" operation="file_lock" profile="libreoffice-soffice" name="/home/bbogart/.thunderbird/fp6psh9b.default/cert9.db" pid=100668 comm="soffice.bin" requested_mask="k" denied_mask="k" fsuid=1000 ouid=1000
Jul 30 10:57:18 domi kernel: [92652.340726] audit: type=1400 audit(1690739838.329:94): apparmor="ALLOWED" operation="open" profile="libreoffice-soffice" name="/home/bbogart/.thunderbird/fp6psh9b.default/key4.db" pid=100668 comm="soffice.bin" requested_mask="wrc" denied_mask="wrc" fsuid=1000 ouid=1000
Jul 30 10:57:18 domi kernel: [92652.340826] audit: type=1400 audit(1690739838.329:95): apparmor="ALLOWED" operation="file_lock" profile="libreoffice-soffice" name="/home/bbogart/.thunderbird/fp6psh9b.default/key4.db" pid=100668 comm="soffice.bin" requested_mask="k" denied_mask="k" fsuid=1000 ouid=1000
Jul 30 10:57:19 domi kernel: [92653.009914] audit: type=1400 audit(1690739838.997:96): apparmor="ALLOWED" operation="open" profile="libreoffice-soffice" name="/usr/share/zoneinfo-icu/44/le/zoneinfo64.res" pid=100668 comm="soffice.bin" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Jul 30 10:57:19 domi kernel: [92653.010691] audit: type=1400 audit(1690739839.001:97): appar...

[b (ben-ekran)]

Confirmed on 5.15.0-78

[b (ben-ekran)]

The issue persists on 5.4.0-163 and 5.15.0-84

[b (ben-ekran)]

I'm not sure how, but I managed to get to a console and see the panic trace on 5.15.0-84. (see attachment below) It does not match the same panic that I've been previously getting with other kernels.

I also confirm the issue persists on 5.15.0-86 and 5.4.0-164

[b (ben-ekran)]

This issue persists on 5.15.0-91

[b (ben-ekran)]

Issue confirmed on 5.4.0-170.

[b (ben-ekran)]

confirmed on 5.15.0-92

[b (ben-ekran)]

Confirmed on 5.15.0-100 and 5.15.0-101

[b (ben-ekran)]

Confirmed on 5.15.0-102

[b (ben-ekran)]

Confirmed on 5.15.0-105

[b (ben-ekran)]

confirmed on 5.15.0-107