kernel 5.8 general protection fault at boot

[Ben Poliakoff]

Ubuntu kernel version 5.8.0-28 (as well as the previous 5.8.0-26) crashes at boot with many general protection faults, on my Intel NUC 10. Ubuntu kernel version 5.4.0-52 did not exhibit this problem.

ProblemType: Bug
DistroRelease: Ubuntu 20.10
Package: linux-image-5.8.0-28-generic 5.8.0-28.30
ProcVersionSignature: Ubuntu 5.4.0-52.57-generic 5.4.65
Uname: Linux 5.4.0-52-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair wl
ApportVersion: 2.20.11-0ubuntu50.1
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: benp 2912 F.... pulseaudio
 /dev/snd/controlC1: benp 2912 F.... pulseaudio
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
Date: Thu Nov 12 11:51:07 2020
InstallationDate: Installed on 2020-10-08 (35 days ago)
InstallationMedia: Ubuntu 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731)
MachineType: Intel(R) Client Systems NUC10i7FNH
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.4.0-52-generic root=/dev/mapper/vgubuntu-root ro quiet splash crashkernel=512M-:192M
RelatedPackageVersions:
 linux-restricted-modules-5.4.0-52-generic N/A
 linux-backports-modules-5.4.0-52-generic N/A
 linux-firmware 1.190.1
SourcePackage: linux
UpgradeStatus: Upgraded to groovy on 2020-10-28 (15 days ago)
dmi.bios.date: 08/17/2020
dmi.bios.vendor: Intel Corp.
dmi.bios.version: FNCML357.0045.2020.0817.1709
dmi.board.asset.tag: Default string
dmi.board.name: NUC10i7FNB
dmi.board.vendor: Intel Corporation
dmi.board.version: K61360-302
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 35
dmi.chassis.vendor: Intel Corporation
dmi.chassis.version: 2.0
dmi.modalias: dmi:bvnIntelCorp.:bvrFNCML357.0045.2020.0817.1709:bd08/17/2020:svnIntel(R)ClientSystems:pnNUC10i7FNH:pvrK61081-302:rvnIntelCorporation:rnNUC10i7FNB:rvrK61360-302:cvnIntelCorporation:ct35:cvr2.0:
dmi.product.family: FN
dmi.product.name: NUC10i7FNH
dmi.product.sku: BXNUC10i7FNH
dmi.product.version: K61081-302
dmi.sys.vendor: Intel(R) Client Systems

[Ben Poliakoff (bpoliakoff)]

I was able to successfully boot with upstream kernel version: 5.10.0-051000rc3-generic

This kernel package notably does not include the ZFS kernel modules (and those kernel modules did not build when I installed zfs-dkms). So it does seem possible that the bug I'm encountering relates to ubuntu kernel 5.8 and the various ZFS related kernel modules.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Ben Poliakoff (bpoliakoff)]

Ah, according to the openzfs release notes, openzfs version 0.8.4 (the current version in the ubuntu repositories) isn't compatible with kernel versions later than 5.6. Presumably this is the issue. I guess I'll stick with kernel 5.4.x until update ZFS packages are released.

[Colin Ian King (colin-king)]

I may be useful to add "debug ignore_loglevel" to the kernel boot command line to get more debugging information during boot to see which module is tripping this issue.

edit the /etc/default/grub (as the root user) and add the debug and ignore_loglevel keywords to the GRUB_CMDLINE_LINUX_DEFAULT

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash debug ignore_loglevel"

then run:

sudo update-grub

and reboot. Can you then attach the full kernel log.

Thanks.

[Ben Poliakoff (bpoliakoff)]

I updated GRUB_CMDLINE_LINUX_DEFAULT. I was able to boot to multiuser mode once (kernel boot log for that is attached, titled "kern.log.debug-successful"). The system locked up shortly after I stashed a copy of that log.

[Ben Poliakoff (bpoliakoff)]

Subsequent attempts to boot the 5.8.0-28-generic kernel exhibited GPFs and at least one full kernel panic, screenshot of that is attached. Ubuntu kernel version 5.4.0-52-generic remains the only stable option at the moment.

[Colin Ian King (colin-king)]

It maybe worth checking with one or more of the mainline kernels to see if these behave differently from the Ubuntu kernel just to see if they boot fine or if they suffer from the same issue.

The mainline kernels can be found in https://kernel.ubuntu.com/~kernel-ppa/mainline

For example, it may be worth installing the initial 5.8 kernel release https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.8/ - use the following:

https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.8/amd64/linux-headers-5.8.0-050800-generic_5.8.0-050800.202008022230_amd64.deb
https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.8/amd64/linux-headers-5.8.0-050800_5.8.0-050800.202008022230_all.deb
https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.8/amd64/linux-image-unsigned-5.8.0-050800-generic_5.8.0-050800.202008022230_amd64.deb
https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.8/amd64/linux-modules-5.8.0-050800-generic_5.8.0-050800.202008022230_amd64.deb

If this boots OK, then maybe then try the latest mainline 5.8 kernel in https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.8.18/

Failing that, it may be worth trying the mainline kernels from 5.5..to 5.7 too to see when the issue starts to occur.

[Ben Poliakoff (bpoliakoff)]

Mainline kernel version 5.8.0-050800-generic boots up without any issues.

Mainline kernel version *5.8.18-050818-generic* doesn't (GPF and kernel panics).

I'll work backwards from 5.8.18-050818-generic to see when the GPFs and kernel panics go away.

[Colin Ian King (colin-king)]

Thanks Ben, that's much appreciated.

[Ben Poliakoff (bpoliakoff)]

OK, after a lot of kernel package installs and rebooting...

Mainline kernel version 5.8.5-050805 boots and runs successfully.

But the *next* version (5.8.6-050806) generally fails to boot, exhibiting the pattern of GPFs and occasional kernel panics. Once it booted up all the way, but the system hung shortly after booting up.

Ubuntu kernel version 5.4.0.52-generic continues to be very reliable on this system.

[Ben Poliakoff (bpoliakoff)]

Tried the recently released ubuntu linux kernel 5.8.29 just now, still getting the GPFs. Attaching the output of "journalctl -xb" from that failed boot (I was dropped into "emergency mode").

[Ben Poliakoff (bpoliakoff)]

Anything else I can try to further debug this issue?

[Ben Poliakoff (bpoliakoff)]

Interestingly 5.9.10-050910-generic (from the mainline kernel ppa that was mentioned earlier) is booting and running stably.

[Ben Poliakoff (bpoliakoff)]

Just tried the newly released 5.8.0-31-generic, but I'm still seeing the GPFs (see attached screenshot).

I'm continuing to run with the mainline 5.9.10 kernel that I mentioned previously, not ideal since I had to switch to the upstream openzfs-2.0 release which supports the 5.9 kernel.

[Ben Poliakoff (bpoliakoff)]

As of 5.8.0-40-generic (and possibly the previous version as well) this issue seems to be 100% resolved for me.