X1 carbon gen 6 cannot boot 5.3.0-12

[Joshua Powers]

My X1 carbon gen6 will not boot the 5.3.0-12 (or 5.3.0-10) kernel. It gets as far as saying:

EFI stub: Secure boot enabled

Nothing else is printed.
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu7
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: powersj 1385 F.... pulseaudio
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 19.10
HibernationDevice: RESUME=UUID=1eb43198-8ef5-4035-8b81-74c3e2936ad7
InstallationDate: Installed on 2018-12-06 (288 days ago)
InstallationMedia: Ubuntu 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180725)
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 002: ID 5986:2115 Acer, Inc Integrated Camera
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: LENOVO 20KHCTO1WW
Package: linux (not installed)
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.2.0-15-generic root=/dev/mapper/ubuntu--vg-root ro
ProcVersionSignature: Ubuntu 5.2.0-15.16-generic 5.2.9
RelatedPackageVersions:
 linux-restricted-modules-5.2.0-15-generic N/A
 linux-backports-modules-5.2.0-15-generic N/A
 linux-firmware 1.182
RfKill:
 0: phy0: Wireless LAN
  Soft blocked: no
  Hard blocked: no
Tags: eoan
Uname: Linux 5.2.0-15-generic x86_64
UpgradeStatus: Upgraded to eoan on 2019-08-27 (24 days ago)
UserGroups: adm cdrom dip docker kvm libvirt lpadmin lxd plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 07/02/2019
dmi.bios.vendor: LENOVO
dmi.bios.version: N23ET65W (1.40 )
dmi.board.asset.tag: Not Available
dmi.board.name: 20KHCTO1WW
dmi.board.vendor: LENOVO
dmi.board.version: SDK0J40709 WIN
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: None
dmi.modalias: dmi:bvnLENOVO:bvrN23ET65W(1.40):bd07/02/2019:svnLENOVO:pn20KHCTO1WW:pvrThinkPadX1Carbon6th:rvnLENOVO:rn20KHCTO1WW:rvrSDK0J40709WIN:cvnLENOVO:ct10:cvrNone:
dmi.product.family: ThinkPad X1 Carbon 6th
dmi.product.name: 20KHCTO1WW
dmi.product.sku: LENOVO_MT_20KH_BU_Think_FM_ThinkPad X1 Carbon 6th
dmi.product.version: ThinkPad X1 Carbon 6th
dmi.sys.vendor: LENOVO

[Joshua Powers (powersj)]

apport information

[Joshua Powers (powersj)]

apport information

[Joshua Powers (powersj)]

apport information

[Joshua Powers (powersj)]

apport information

[Joshua Powers (powersj)]

apport information

[Joshua Powers (powersj)]

apport information

[Joshua Powers (powersj)]

apport information

[Joshua Powers (powersj)]

apport information

[Joshua Powers (powersj)]

apport information

[Joshua Powers (powersj)]

apport information

[Joshua Powers (powersj)]

apport information

[Joshua Powers (powersj)]

apport information

[Joshua Powers (powersj)]

apport information

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Seth Forshee (sforshee)]

Can you try booting with "earlycon=efifb" and see if you get any more output?

[Joshua Powers (powersj)]

This resulted in what looked like a kernel trace or crash output, but the characters were garbage and unreadable.

[Seth Forshee (sforshee)]

Please try the test kernel below. This will hopefully store the kernel panic information into persistent storage, so after rebooting to this kernel and waiting 30 seconds or so, reboot back to a working kernel and look in /sys/fs/pstore for a file with the name dmesg-ramoops-N, where N is a number. If you have one, please provide the contents of this file.

https://people.canonical.com/~sforshee/lp1844784/5.3.0-12.13+lp1844784v201909230933/

[Seth Forshee (sforshee)]

I forgot to mention a couple of things.

First, this is not a signed kernel, so to boot it you'll either have to disable secure boot or install a suitable MOK with shim and sign it with that key. If you disable secure boot it's probably a good idea to confirm that the 5.3 kernel you already have installed still fails to boot with secure boot disabled.

Since you already have 5.3.0-12.13 installed (iirc) you'll probably need to provide force flags to dpkg to allow linux-unsigned to overwrite files installed by linux. I think --force-overwrite is probably enough.

[Joshua Powers (powersj)]

1. Fully updated system
2. Disabled secure boot
3. Tried to reproduce failure with 5.3.0-12, however I was able to boot successfully

[Seth Forshee (sforshee)]

So it seems like this may be related to having secure boot enabled. Please try the following:

1. Re-enable secure boot to see whether the problem returns, then disable again and see if you can boot normally. If the problem still looks to be related to secure boot ...
2. With secure boot disabled, add "lockdown" to the kernel command line, and see if the problem returns.

[Joshua Powers (powersj)]

1. Re-enabled secure boot
2. Failed to boot 5.3.0-12
3. Disabled secure boot
4. Successfully booted 5.3.0-12
5. Added 'lockdown' to /etc/default/grub, ran 'sudo update-grub', rebooted
6. Successfully booted 5.3.0-12 with secure boot disabled
7. Failed to boot 5.3.0-12 with secure boot enabled

[Seth Forshee (sforshee)]

Thanks! This definitely seems to be related to secure boot, but not lockdown (which gets enabled automatically when booted under secure boot). I'll try following that lead and see what I can find.

The other thing which might still help is testing the kernel from comment #17 with secure boot enabled, to see if we can get the oops message captured in persistent storage. This means generating your own MOK with the correct extendedKeyUsage, enrolling it with shim, and signing the kernel with that key. If you're up for it, instructions are here:

https://ubuntu.com/blog/how-to-sign-things-for-secure-boot

Just be sure to read the section under "Enrolling the key" before actually generating your keys, which notes that you need to remove an OID from extendedKeyUsage if you want to use the key for signing kernels.

[Joshua Powers (powersj)]

1. Re-enabled secure boot
2. Failed to boot 5.3.0-12
3. Disabled TPM
4. Successfully booted 5.3.0-12 with secure boot

[Joshua Powers (powersj)]

FWIW here is the firmware version:

DMI: LENOVO 20KHCTO1WW/20KHCTO1WW, BIOS N23ET65W (1.40 ) 07/02/2019

I am running with an encrypted disk with UEFI secure boot enabled. While I was not doing anything with the TPM it was enabled by default I believe.