Make linux-kvm bootable in LXD VMs

Bug #1873809 reported by Stéphane Graber on 2020-04-20
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-images
Undecided
Unassigned
linux-kvm (Ubuntu)
High
Unassigned
Focal
High
Stefan Bader

Bug Description

The `disk-kvm.img` images which are to be preferred when run under virtualization, currently completely fail to boot under UEFI.

A workaround was put in place such that LXD instead will pull generic-based images until this is resolved, this however does come with a much longer boot time (as the kernel panics, reboots and then boots) and also reduced functionality from cloud-init, so we'd still like this fixed in the near future.

To get things behaving, it looks like we need the following config options to be enable in linux-kvm:

 - CONFIG_EFI_STUB
 - CONFIG_VSOCKETS
 - CONFIG_VIRTIO_VSOCKETS
 - CONFIG_VIRTIO_VSOCKETS_COMMON

== Rationale ==
We'd like to be able to use the linux-kvm based images for LXD, those will directly boot without needing the panic+reboot behavior of generic images and will be much lighter in general.

We also need the LXD agent to work, which requires functional virtio vsock.

== Test case ==
 - wget http://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-lxd.tar.xz
 - wget http://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-disk-kvm.img
 - lxc image import focal-server-cloudimg-amd64-lxd.tar.xz focal-server-cloudimg-amd64-disk-kvm.img --alias bug1873809
 - lxc launch bug1873809 v1
 - lxc console v1
 - <check that it boots to login prompt>
 - <disconnect with ctrl+a-q>
 - lxc exec v1 bash

To validate a new kernel, you'll need to manually repack the .img file and install the new kernel in there.

== Regression potential ==
I don't know who else is using those kvm images right now, but those changes will cause a change to the kernel binary such that it contains the EFI stub bits + a signature. This could cause some (horribly broken) systems to no longer be able to boot that kernel. Though considering that such a setup is common to our other kernels, this seems unlikely.

Also, this will be introducing virtio vsock support which again, could maybe confused some horribly broken systems?

In either case, the kernel conveniently is the only package which ships multiple versions concurently, so rebooting on the previous kernel is always an option, mitigating some of the risks.

-- Details from original report --
User report on the LXD side: https://github.com/lxc/lxd/issues/7224

I've reproduced this issue with:
 - wget http://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-disk-kvm.img
 - qemu-system-x86_64 -bios /usr/share/ovmf/OVMF.fd -hda focal-server-cloudimg-amd64-disk-kvm.img -m 1G

On the graphical console, you'll see EDK2 load (TianoCore) followed by basic boot messages and then a message from grub (error: can't find command `hwmatch`).
Those also appear on successful boots of other images so I don't think there's anything concerning that. However it'll hang indefinitely and eat up all your CPU.

Switching to the text console view (serial0), you'll see the same issue as that LXD report:

BdsDxe: failed to load Boot0001 "UEFI QEMU DVD-ROM QM00003 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Secondary,Master,0x0): Not Found
BdsDxe: loading Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)
BdsDxe: starting Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)
error: can't find command `hwmatch'.
e!!!! X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID - 00000000 !!!!
ExceptionData - 0000000000000000
RIP - 000000003FF2DA12, CS - 0000000000000038, RFLAGS - 0000000000200202
RAX - AFAFAFAFAFAFAFAF, RCX - 000000003E80F108, RDX - AFAFAFAFAFAFAFAF
RBX - 0000000000000398, RSP - 000000003FF1C638, RBP - 000000003FF34360
RSI - 000000003FF343B8, RDI - 0000000000001000
R8 - 000000003E80F108, R9 - 000000003E815B98, R10 - 0000000000000065
R11 - 0000000000002501, R12 - 0000000000000004, R13 - 000000003E80F100
R14 - 0000000000000000, R15 - 0000000000000000
DS - 0000000000000030, ES - 0000000000000030, FS - 0000000000000030
GS - 0000000000000030, SS - 0000000000000030
CR0 - 0000000080010033, CR2 - 0000000000000000, CR3 - 000000003FC01000
CR4 - 0000000000000668, CR8 - 0000000000000000
DR0 - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3 - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 000000003FBEEA98 0000000000000047, LDTR - 0000000000000000
IDTR - 000000003F2D8018 0000000000000FFF, TR - 0000000000000000
FXSAVE_STATE - 000000003FF1C290
!!!! Find image based on IP(0x3FF2DA12) /build/edk2-dQLD17/edk2-0~20191122.bd85bf54/Build/OvmfX64/RELEASE_GCC5/X64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll (ImageBase=000000003FF1E000, EntryPoint=000000003FF30781) !!!!

If booting in a SecureBoot enabled environment, you instead get a `Access Denied` at kernel loading time, indicating that the kernel binary isn't a normal signed kernel. That has the same result (boot hangs) but without the crash message.

Stéphane Graber (stgraber) wrote :

Ff we can't easily track down & fix the source of this issue, an alternative is to modify the streams to strip the kvm-img hash from the LXD metadata record, forcing all our users onto the non-kvm image. Do note that this comes with its own issues though.

Those non-kvm images attempt to boot without an initrd, which fails under LXD due to missing virtio drivers. The kernel then panics and reboots back into grub, this time booting with an initrd.

On top of the added boot time and CPU load this causes, it also means that the VMs never boot properly on first boot. This breaks LXD's cloud-init support, so those VMs will never get their cloud-init data.

Robert C Jennings (rcj) wrote :

I'm seeing this on boot...

BdsDxe: failed to load Boot0001 "UEFI QEMU DVD-ROM QM00003 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Secondary,Master,0x0): Not Found
BdsDxe: loading Boot0002 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x4,0x0)
BdsDxe: starting Boot0002 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x4,0x0)
error: kernel doesn't support EFI handover.
error: you need to load the kernel first.

Stéphane Graber (stgraber) wrote :

Yeah, I think that's when you're lucky and the firmware doesn't explode on you instead.

Louis Bouchard (louis) wrote :

FWIW, I can confirm that the issue is also present on QEMU on Bionic hosts.

This was working correctly with the following image version :

ubuntu@focal:/etc$ cat /etc/cloud/build.info
build_name: server
serial: 20200223

Colin Ian King (colin-king) wrote :

I could only boot when using a minimum of 2GB of memory, for some reason 1GB was not enough. Also, when using 2GB.

Secondly I get the error:

error: kernel doesn't support EFI handover
error: you need to load the kernel firs.

Press any key to continue...

and then one returns back to grub.

Colin Ian King (colin-king) wrote :

Can't boot the 20200414.1 image either.

Colin Ian King (colin-king) wrote :

I mounted the boot partition and looked at the kernel config and it is missing CONFIG_EFI_STUB which I believe is required to make the kernel bootable.

Robert C Jennings (rcj) wrote :

@colin-king, Can we get that config option added for this kernel?

Colin Ian King (colin-king) wrote :

Attached is the kvm 5.4 kernel config with the missing EFI stub config

Colin Ian King (colin-king) wrote :

So I just tested the same focal kernel with:

git diff
diff --git a/debian.kvm/config/config.common.ubuntu b/debian.kvm/config/config.common.ubuntu
index b69d54e43965..1d4ac88b9db2 100644
--- a/debian.kvm/config/config.common.ubuntu
+++ b/debian.kvm/config/config.common.ubuntu
@@ -695,7 +695,7 @@ CONFIG_EFI_PARTITION=y
 # CONFIG_EFI_RCI2_TABLE is not set
 # CONFIG_EFI_RUNTIME_MAP is not set
 CONFIG_EFI_RUNTIME_WRAPPERS=y
-# CONFIG_EFI_STUB is not set
+CONFIG_EFI_STUB=y
 # CONFIG_EFI_TEST is not set
 CONFIG_EFI_VARS=y
 CONFIG_EFS_FS=m

and co-erced it onto the same image and was able to boot it successfully using:

 qemu-system-x86_64 -bios /usr/share/ovmf/OVMF.fd -hda focal-server-cloudimg-amd64-disk-kvm.img -m 1G -nographic

Stéphane Graber (stgraber) wrote :

Ok, so the fact that we thought this worked is clearly the result from bad testing on our part, probably because of our simplestreams parsing code we fixed yesterday...

We obviously still need to move LXD onto this images as booting the non-kvm images takes twice as long as it should (due to them panic + reboot every time) AND also breaks cloud-init, at least in the way we'd like to use it.

Now realistically this can't be fixed in time for 20.04, so what we've done is submitted a change to simplestreams to force all LXD users onto the non-kvm image:
  https://code.launchpad.net/~stgraber/simplestreams/+git/simplestreams/+merge/382597

We'll also tell all users of `ubuntu:` and `ubuntu-daily:` that they need to do:
 - lxc config device add NAME config disk source=cloud-init:config

Which passes a stable config drive to cloud-init, avoiding the cloud-init issue they'd be getting out of the box.

Moving forward, we'd like the -kvm kernel to be both EFI enabled AND signed. This will then allow those images to work properly inside LXD, at which point we can undo the simplestreams change and have those images used once again by our users.

Stéphane Graber (stgraber) wrote :

I've tested a kernel with CONFIG_EFI_STUB added (thanks cking!).

This does boot with secureboot enabled, though the LXD agent fails to start due to lack of vsock.

So in addition to CONFIG_EFI_STUB, it looks like we also need:
 - CONFIG_VSOCKETS
 - CONFIG_VIRTIO_VSOCKETS
 - CONFIG_VIRTIO_VSOCKETS_COMMON

Which should give us the bits needed for virtio vsock.

The rest all looked good, so we should be fine with those tweaks and the kernel getting signed.

Stéphane Graber (stgraber) wrote :

Marking cloud-images side of this as Invalid since the images themselves are built correctly.
Re-packing with an updated kernel boots just fine, so we only need to track this against linux-kvm.

Changed in cloud-images:
status: New → Invalid
summary: - disk-kvm.img aren't UEFI bootable
+ Make linux-kvm bootable in LXD VMs
description: updated
Stéphane Graber (stgraber) wrote :

Just tested it now, confirmed that this still boots fine and that this time the LXD agent successfully starts too.

So this config seems suitable for us. That + enabling kernel signing will get us working images.

Thanks!

Changed in linux-kvm (Ubuntu):
assignee: nobody → Colin Ian King (colin-king)
importance: Undecided → High
Changed in cloud-images:
status: Invalid → Confirmed
assignee: nobody → Roufique Hossain (roufique)
Changed in linux-kvm (Ubuntu):
assignee: Colin Ian King (colin-king) → Roufique Hossain (roufique)
status: New → Incomplete
status: Incomplete → Confirmed
Changed in cloud-bl-tutorials:
status: New → Confirmed
Louis Bouchard (louis) wrote :

Hello,
thanks for the quick turnaround.

For the record, some EFI configs did exist in prior 5.3 images :

root@focal:/boot# cat /etc/cloud/build.info
build_name: server
serial: 20200223
root@focal:/boot# uname -r
5.3.0-1009-kvm
root@focal:/boot# grep -i ^CONFIG_EFI config-5.3.0-1009-kvm
CONFIG_EFI=y
CONFIG_EFI_VARS=y
CONFIG_EFI_ESRT=y
CONFIG_EFI_RUNTIME_WRAPPERS=y
CONFIG_EFI_EARLYCON=y
CONFIG_EFI_PARTITION=y
CONFIG_EFIVAR_FS=m

I'll be using the other image for testing in the meantime
...Louis

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux-kvm - 5.4.0-1009.9

---------------
linux-kvm (5.4.0-1009.9) focal; urgency=medium

  * focal/linux-kvm: 5.4.0-1009.9 -proposed tracker (LP: #1873934)

  * multipathd.service fails to start with linux-kvm kernel (LP: #1873912)
    - [Config] Enable dm-multipath modules

  * Make linux-kvm bootable in LXD VMs (LP: #1873809)
    - [Config] CONFIG_EFI_STUB=y
    - [Config] Enable vsocket config options

 -- Seth Forshee <email address hidden> Mon, 20 Apr 2020 14:30:41 -0500

Changed in linux-kvm (Ubuntu):
status: Confirmed → Fix Released
Stéphane Graber (stgraber) wrote :

Thanks Louis, so our testing may in fact have been accurate and things regressed afterwards :)

Stéphane Graber (stgraber) wrote :

Hmm, actually, CONFIG_EFI_STUB is the one we were missing and I'm not seeing that in your VM either, which makes me wonder how it was booted in the first place :)

Stéphane Graber (stgraber) wrote :

Ok, fixed the bug tasks and re-opened the bug as we still need this kernel to get signed.

Changed in linux-kvm (Ubuntu):
status: Fix Released → Triaged
Changed in cloud-images:
assignee: Roufique Hossain (roufique) → nobody
Changed in linux-kvm (Ubuntu):
assignee: Roufique Hossain (roufique) → nobody
Changed in cloud-bl-tutorials:
status: Confirmed → Invalid
status: Invalid → New
affects: cloud-bl-tutorials → linux (Ubuntu)
no longer affects: linux (Ubuntu)
Changed in cloud-images:
status: Confirmed → Invalid
Louis Bouchard (louis) wrote :

Hello,

Just tested the latest image in ./current and it boots fine :

root@focal-ok:~# uname -a
Linux focal-ok 5.4.0-1009-kvm #9-Ubuntu SMP Mon Apr 20 20:23:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@focal-ok:~# cat /etc/cloud/build.info
build_name: server
serial: 20200421
root@focal-ok:~# uname -r
5.4.0-1009-kvm
root@focal-ok:~# grep -i ^CONFIG_EFI /boot/config-5.4.0-1009-kvm
CONFIG_EFI=y
CONFIG_EFI_STUB=y
CONFIG_EFI_MIXED=y
CONFIG_EFI_VARS=y
CONFIG_EFI_ESRT=y
CONFIG_EFI_RUNTIME_WRAPPERS=y
CONFIG_EFI_EARLYCON=y
CONFIG_EFI_PARTITION=y
CONFIG_EFIVAR_FS=m

Thanks for the quick turnaround !
...Louis

Khaled El Mously (kmously) wrote :

@stgraber I'm getting mixed messages from the last few comments about whether CONFIG_EFI_STUB is needed or not - can you please clarify?

Stéphane Graber (stgraber) wrote :

@Khaled yes, it is and we have it now. What's still needed is for the kernel to be signed so it can be used under secureboot.

Khaled El Mously (kmously) wrote :

@stgraber sorry I should have clarified that I'm referring to Eoan specifically.
In Eoan, EFI_STUB is still not enabled. Is it needed there?

Launchpad Janitor (janitor) wrote :
Download full text (39.0 KiB)

This bug was fixed in the package linux-kvm - 5.3.0-1017.19

---------------
linux-kvm (5.3.0-1017.19) eoan; urgency=medium

  * eoan/linux-kvm: 5.3.0-1017.18 -proposed tracker (LP: #1877951)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync dkms-build and family
    - [Packaging] add libcap-dev dependency

  * Make linux-kvm bootable in LXD VMs (LP: #1873809)
    - [config] Enable CONFIG_EFI_STUB
    - [Config] Enable VSOCKETS in KVM

  [ Ubuntu: 5.3.0-53.47 ]

  * eoan/linux: 5.3.0-53.47 -proposed tracker (LP: #1877257)
  * Intermittent display blackouts on event (LP: #1875254)
    - drm/i915: Limit audio CDCLK>=2*BCLK constraint back to GLK only
  * Unable to handle kernel pointer dereference in virtual kernel address space
    on Eoan (LP: #1876645)
    - SAUCE: overlayfs: fix shitfs special-casing

  [ Ubuntu: 5.3.0-52.46 ]

  * eoan/linux: 5.3.0-52.46 -proposed tracker (LP: #1874752)
  * alsa: make the dmic detection align to the mainline kernel-5.6
    (LP: #1871284)
    - ALSA: hda: add Intel DSP configuration / probe code
    - ALSA: hda: fix intel DSP config
    - ALSA: hda: Allow non-Intel device probe gracefully
    - ALSA: hda: More constifications
    - ALSA: hda: Rename back to dmic_detect option
    - [Config] SND_INTEL_DSP_CONFIG=m
    - [packaging] Remove snd-intel-nhlt from modules
  * built-using constraints preventing uploads (LP: #1875601)
    - temporarily drop Built-Using data
  * ubuntu/focal64 fails to mount Vagrant shared folders (LP: #1873506)
    - [Packaging] Move virtualbox modules to linux-modules
    - [Packaging] Remove vbox and zfs modules from generic.inclusion-list
  * linux-image-5.0.0-35-generic breaks checkpointing of container
    (LP: #1857257)
    - SAUCE: overlayfs: use shiftfs hacks only with shiftfs as underlay
  * shiftfs: broken shiftfs nesting (LP: #1872094)
    - SAUCE: shiftfs: record correct creator credentials
  * Add debian/rules targets to compile/run kernel selftests (LP: #1874286)
    - [Packaging] add support to compile/run selftests
  * shiftfs: O_TMPFILE reports ESTALE (LP: #1872757)
    - SAUCE: shiftfs: fix dentry revalidation
  * getitimer returns it_value=0 erroneously (LP: #1349028)
    - [Config] CONTEXT_TRACKING_FORCE policy should be unset
  * 5.3.0-46-generic - i915 - frequent GPU hangs / resets rcs0 (LP: #1872001)
    - drm/i915/execlists: Preempt-to-busy
    - drm/i915/gt: Detect if we miss WaIdleLiteRestore
    - drm/i915/execlists: Always force a context reload when rewinding RING_TAIL
  * alsa/sof: external mic can't be deteced on Lenovo and HP laptops
    (LP: #1872569)
    - SAUCE: ASoC: intel/skl/hda - set autosuspend timeout for hda codecs
  * Eoan update: upstream stable patchset 2020-04-22 (LP: #1874325)
    - ARM: dts: sun8i-a83t-tbs-a711: HM5065 doesn't like such a high voltage
    - bus: sunxi-rsb: Return correct data when mixing 16-bit and 8-bit reads
    - net: vxge: fix wrong __VA_ARGS__ usage
    - hinic: fix a bug of waitting for IO stopped
    - hinic: fix wrong para of wait_for_completion_timeout
    - cxgb4/ptp: pass the sign of offset delta in FW CMD
    - qlcnic: Fix bad kzalloc null test
    - i2c: st: fix missing struct parameter descr...

Changed in linux-kvm (Ubuntu):
status: Triaged → Fix Released
Stéphane Graber (stgraber) wrote :

Re-opening as I'm not seeing any mention of this being signed now.

Changed in linux-kvm (Ubuntu):
status: Fix Released → Triaged
Stéphane Graber (stgraber) wrote :

Pinged in #ubuntu-kernel today for an update. It'd be good to have groovy signed soon so we can then roll this out to focal users.

Stefan Bader (smb) on 2020-06-15
Changed in linux-kvm (Ubuntu Focal):
assignee: nobody → Stefan Bader (smb)
importance: Undecided → High
status: New → In Progress
Stefan Bader (smb) on 2020-06-15
Changed in linux-kvm (Ubuntu Focal):
status: In Progress → Fix Committed
Stefan Bader (smb) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Stéphane Graber (stgraber) wrote :

Trying to boot the proposed kernel in LXD:

"""
BdsDxe: loading Boot0007 "ubuntu" from HD(1,GPT,25633192-5DBD-412A-8A50-E29B79F72A50,0x800,0x32000)/\EFI\ubuntu\shimx64.efi
BdsDxe: starting Boot0007 "ubuntu" from HD(1,GPT,25633192-5DBD-412A-8A50-E29B79F72A50,0x800,0x32000)/\EFI\ubuntu\shimx64.efi
RAMDISK: incomplete write (4194304 != 8388608)
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-1017-kvm #17-Ubuntu
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)/LXD, BIOS 0.0.0 02/06/2015
Call Trace:
 0xffffffff9392230d
 0xffffffff932a8a21
 0xffffffff9412e5c1
 0xffffffff9412e80f
 0xffffffff9412e976
 0xffffffff9412e274
 ? 0xffffffff93938a70
 0xffffffff93938a79
 0xffffffff93a00215
Kernel Offset: 0x12200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---
"""

This appears to be lz4 related. Changing the initramfs to gzip makes the VM boot just fine.
It's worth noting that when booting the generic kernel, we get the unpack error showed in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835660 but things still boot fine.

Marking verification as failed based on this. We need images to work properly with a standard Ubuntu config so need lz4 fixed.

tags: added: verification-failed-focal
removed: verification-needed-focal
Stéphane Graber (stgraber) wrote :

"""
Jun 18 13:56:15 f1 kernel: [ 0.383207] Trying to unpack rootfs image as initramfs...
Jun 18 13:56:15 f1 kernel: [ 0.463102] Initramfs unpacking failed: Decoding failed
"""

Is what we're getting on current generic kernel, though boot continues after that.
I don't know if when that happens we're actually skipping the initrd entirely and just get lucky that the generic kernel has everything we need builtin so it boots or if the error in that case is just wrong and the initrd is still properly unpacked and run.

Either way, this needs sorting, looking at the other bug report, there's been something wrong with our kernel and lz4 initrd for a long time and it's apparently biting us a lot more now.

Stefan Bader (smb) wrote :

@Stephane, if this bug was not phrased that generically I would feel inclined to say it verified ok if the kernel image can be booted in secureboot mode and has the requested config options set. And treat the initrd issue as a separate one on the path to completion.

For us, I would propose to count this feedback as success (and still release the kernel that way if there is no other problems).

Stéphane Graber (stgraber) wrote :

@Stefan, so actually this is an actual regression.

1015 will boot just fine in LXD with secureboot disabled.
1017 will not boot at all in LXD with or without secureboot disabled.

I don't know if it's switching to a signed kernel which causes the lz4 issue but the result is a clear regression so I would not consider this kernel suitable for release to anyone.

Stefan Bader (smb) wrote :

@Stéphane, the uncompress message appears to be something that may happen if blocks are not aligned but not harming anything. By now I have played around with both a self created secureboot VM on bionic and also (following your turorial) a LXD 4.0 vM. And for both I could not repeat what you saw. Both booted. There is a minor issue for people creating their own VMs and using a graphical interface (like virt-manager). The kvm kernel has no framebuffer devices enabled so one does not see anything if one does not enable a serial console.

For LXD, when I repeated the steps from https://discuss.linuxcontainers.org/t/running-virtual-machines-with-lxd-4-0/7519 I saw one crash not finding the root disk after the initial lxc start when I ran lxc console. But that was with the generic kernel and that seems to be handled by the panic handler. Attaching to he console a second time I saw cloud-init finishing and then I installed the proposed kvm kernel in that running vm and rebooted. And this gave me a working boot:

ubuntu@ubuntu:~$ uname -a
Linux ubuntu 5.4.0-1017-kvm #17-Ubuntu SMP Mon Jun 15 13:05:31 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ubuntu:~$ dmesg|grep secure
[ 0.000000] secureboot: Secure boot enabled
[ 0.260090] secureboot: Secure boot enabled

Stéphane Graber (stgraber) wrote :

Yeah, I think you're right, I also had the exact same panic happen now on 1015, so it's likely some grub weirdness rather than kernel regression.

It just so happened that in my last test I managed to get a working grub config after moving to 1015 and not with 1017. Looks like we'll need to poke at grub...

Stéphane Graber (stgraber) wrote :

Hmm, actually no luck at booting either 1015 or 1017 on security.secureboot=false here, poked at grub and it does load both kernel and initrd...

Stéphane Graber (stgraber) wrote :
Download full text (24.0 KiB)

"""
Loading Linux 5.4.0-1015-kvm ...
Loading initial ramdisk ...
Linux version 5.4.0-1015-kvm (buildd@lcy01-amd64-027) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #15-Ubuntu SMP Fri Jun 5 00:55:20 UTC 2020 (Ubuntu 5.4.0-1015.15-kvm 5.4.41)
Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-1015-kvm root=UUID=03167f19-fb7f-4ba9-b4da-5e4acc0d97e3 ro single nomodeset
x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
x86/fpu: xstate_offset[3]: 832, xstate_sizes[3]: 64
x86/fpu: xstate_offset[4]: 896, xstate_sizes[4]: 64
x86/fpu: Enabled xstate features 0x1f, context size is 960 bytes, using 'compacted' format.
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
BIOS-e820: [mem 0x0000000000100000-0x000000003ee6afff] usable
BIOS-e820: [mem 0x000000003ee6b000-0x000000003ef2bfff] reserved
BIOS-e820: [mem 0x000000003ef2c000-0x000000003f8eefff] usable
BIOS-e820: [mem 0x000000003f8ef000-0x000000003faeefff] reserved
BIOS-e820: [mem 0x000000003faef000-0x000000003fb75fff] usable
BIOS-e820: [mem 0x000000003fb76000-0x000000003fb7efff] ACPI data
BIOS-e820: [mem 0x000000003fb7f000-0x000000003fbfefff] ACPI NVS
BIOS-e820: [mem 0x000000003fbff000-0x000000003ffdffff] usable
BIOS-e820: [mem 0x000000003ffe0000-0x000000003fffffff] reserved
BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved
BIOS-e820: [mem 0x00000000ffe00000-0x00000000ffffffff] reserved
NX (Execute Disable) protection: active
extended physical RAM map:
reserve setup_data: [mem 0x0000000000000000-0x000000000009ffff] usable
reserve setup_data: [mem 0x0000000000100000-0x000000003df4b017] usable
reserve setup_data: [mem 0x000000003df4b018-0x000000003df86457] usable
reserve setup_data: [mem 0x000000003df86458-0x000000003df87017] usable
reserve setup_data: [mem 0x000000003df87018-0x000000003df90a57] usable
reserve setup_data: [mem 0x000000003df90a58-0x000000003ee6afff] usable
reserve setup_data: [mem 0x000000003ee6b000-0x000000003ef2bfff] reserved
reserve setup_data: [mem 0x000000003ef2c000-0x000000003f8eefff] usable
reserve setup_data: [mem 0x000000003f8ef000-0x000000003faeefff] reserved
reserve setup_data: [mem 0x000000003faef000-0x000000003fb75fff] usable
reserve setup_data: [mem 0x000000003fb76000-0x000000003fb7efff] ACPI data
reserve setup_data: [mem 0x000000003fb7f000-0x000000003fbfefff] ACPI NVS
reserve setup_data: [mem 0x000000003fbff000-0x000000003ffdffff] usable
reserve setup_data: [mem 0x000000003ffe0000-0x000000003fffffff] reserved
reserve setup_data: [mem 0x00000000b0000000-0x00000000bfffffff] reserved
reserve setup_data: [mem 0x00000000ffe00000-0x00000000ffffffff] reserved
efi: EFI v2.70 by EDK II
efi: SMBIOS=0x3f915000 ACPI=0x3fb7e000 ACPI 2.0=0x3fb7e014 MEMATTR=0x3e115118
secureboot: Secure boot disabled
SMBIOS 2.8 present.
DMI: QEMU Standard PC (Q35 + ICH9, 2009)/LXD, BIOS 0.0.0 02/06/2015
Hypervisor detected: KVM
kvm-clock: Using ...

Stéphane Graber (stgraber) wrote :

@smb Can you confirm that your system indeed goes through the initrd and isn't just silently falling back to directly mounting and booting /?

Booting with break=mount would likely be a valid way to test this (should drop you in a shell).

Stéphane Graber (stgraber) wrote :
Download full text (25.4 KiB)

"""
stgraber@castiana:~$ lxc launch images:ubuntu/focal f1 --vm
Creating f1
Starting f1
stgraber@castiana:~$ lxc exec f1 bash
root@f1:~# echo "deb http://archive.ubuntu.com/ubuntu focal-proposed main restricted universe multiverse" >> /etc/apt/sources.list
root@f1:~# apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [107 kB]
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [107 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-proposed InRelease [265 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [201 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [80.2 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [11.1 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [3036 B]
Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [114 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages [82.4 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal-proposed/main Translation-en [35.0 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal-proposed/restricted amd64 Packages [7132 B]
Get:13 http://archive.ubuntu.com/ubuntu focal-proposed/restricted Translation-en [2144 B]
Get:14 http://archive.ubuntu.com/ubuntu focal-proposed/universe amd64 Packages [35.8 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-proposed/universe Translation-en [24.5 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-proposed/multiverse Translation-en [3404 B]
Fetched 1079 kB in 1s (794 kB/s)
Reading package lists... Done
root@f1:~# apt-get install linux-kvm
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  linux-headers-5.4.0-1017-kvm linux-headers-kvm linux-image-5.4.0-1017-kvm linux-image-kvm linux-kvm-headers-5.4.0-1017 linux-modules-5.4.0-1017-kvm
Suggested packages:
  fdutils linux-kvm-doc-5.4.0 | linux-kvm-source-5.4.0 linux-kvm-tools
The following NEW packages will be installed:
  linux-headers-5.4.0-1017-kvm linux-headers-kvm linux-image-5.4.0-1017-kvm linux-image-kvm linux-kvm linux-kvm-headers-5.4.0-1017
  linux-modules-5.4.0-1017-kvm
0 upgraded, 7 newly installed, 0 to remove and 18 not upgraded.
Need to get 28.4 MB of archives.
After this operation, 126 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 linux-kvm-headers-5.4.0-1017 all 5.4.0-1017.17 [11.3 MB]
Get:2 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 linux-headers-5.4.0-1017-kvm amd64 5.4.0-1017.17 [1254 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 linux-headers-kvm amd64 5.4.0.1017.16 [4376 B]
Get:4 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 linux-modules-5.4.0-1017-kvm amd64 5.4.0-1017.17 [10.6 MB]
Get:5 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 linux-image-5.4.0-1017-kvm amd64 5.4.0-1017.17 [5158 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-proposed/main amd6...

Stéphane Graber (stgraber) wrote :

https://paste.ubuntu.com/p/7yHDCFt75m/ for additional proof that the initrd is never executed (break=top would immediately drop to a shell).

I'm not sure if it's related, but maybe worth the mention: could this be due to the initrd-less boot? I've noticed that in some VMs, it first fails to boot (it tries an initrd-less boot), reboots and then loads the initrd. This is an Ubuntu grub-thing, and you can prevent that by deleting a file *40*partuuid* form /etc/default/grub.d/ (need to recreate grub config file after that).

Cheers,

Guilherme

Stéphane Graber (stgraber) wrote :

It's not the log above clearly shows the kernel loading an initrd.

Stefan Bader (smb) wrote :

@Stéphane, I have now followed your steps from commant #38 (was using the official ubuntu image before with cloud-init and that does a initrd-less boot first which is successful for me) but this does not show me the error you see either. And a break=top does work (with some complaints about tty1 which is expected due to the missing efi-fb).
I double-checked the initramfs compression setting and that is lz4.

This is rather weird. I will try on a different hw but that really should not be relevant. For double checking: which LXD version are you using? I am on the focal/stable/4.0 stream:

lxd 4.0.1 15682 4.0/stable/… canonical✓ -

Stefan Bader (smb) wrote :

Hm, and maybe also relevant: what kind of pool is used? My setup was "lxd init --auto" which is a file based image backend.

Stefan Bader (smb) wrote :

Ok, so with the same software versions installed but on a different hw platform, I did at least once get this incomplete write error. The working box is an AMD/BIOS one and the non-working an Intel/EFI. Not sure why that has any impact.

Stefan Bader (smb) wrote :

Alright, at least partially understanding things now. So it looks like "Decoding failed" happens generally on some platforms (and I do not understand still why this is). The difference between the generic kernel and the kvm kernel is that the kvm kernel has a built-in BLK_DEV_RAM of size (4096) whereas the generic kernel uses a module for that.

It looks like only if BLK_DEV_RAM=y is set, the code which populates the initial rootfs does try to flush and refill the ramdisk device. Otherwise the partially expanded rootfs is retained. And I guess the ramdisk size is set to small so we get the incomplete write there.

I guess the solution (which does not solve the decode failure) is to change the kvm kernel config to

BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_SIZE=65536

Stefan Bader (smb) wrote :

Latest version: 5.4.0-1018-kvm

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
removed: verification-failed-focal
Stefan Bader (smb) wrote :

Note that for me the latest kernel works past the decoding failed stage. But it would be good to have a second verification is possible.

Stéphane Graber (stgraber) wrote :

Confirmed, 1018 boots fine here under Secure Boot, all good!

tags: added: verification-done-focal
removed: verification-needed-focal
Stéphane Graber (stgraber) wrote :

@smb what's the state of groovy, did you push the config update there too?

For the cloud images, we'll want to switch over to those using linux-kvm in groovy first, then focal, so just want to make sure we'll get a working kernel on there too!

Stefan Bader (smb) wrote :

@Stéphane, right now (and that might be for a while still) the linux-kvm kernel in Groovy is a copy-forward from Focal. So once this releases, you should be able to use it there as well. I started to have annotated config enforcement for the adjusted config which hopefully will survive when the specific kvm kernel for Groovy gets started.

Launchpad Janitor (janitor) wrote :
Download full text (30.5 KiB)

This bug was fixed in the package linux-kvm - 5.4.0-1018.18

---------------
linux-kvm (5.4.0-1018.18) focal; urgency=medium

  * focal/linux-kvm: 5.4.0-1018.18 -proposed tracker (LP: #1885099)

  * LXD 4.2 broken on linux-kvm due to missing VLAN filtering (LP: #1882955)
    - [Config] kvm: VLAN_8021Q=m && BRIDGE_VLAN_FILTERING=y

  * Make linux-kvm bootable in LXD VMs (LP: #1873809)
    - [Config] kvm: Match ramdisk config with master
    - [Config] kvm: Build-in EFI framebuffer

linux-kvm (5.4.0-1017.17) focal; urgency=medium

  * focal/linux-kvm: 5.4.0-1017.17 -proposed tracker (LP: #1883517)

  * Make linux-kvm bootable in LXD VMs (LP: #1873809)
    - [Packaging] Start to sign the KVM kernel

linux-kvm (5.4.0-1016.16) focal; urgency=medium

  * focal/linux-kvm: 5.4.0-1016.16 -proposed tracker (LP: #1882691)

  * Focal update: v5.4.42 upstream stable release (LP: #1879759)
    - [Config] kvm: Record CC_HAS_WARN_MAYBE_UNINITIALIZED drop

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts

  [ Ubuntu: 5.4.0-38.42 ]

  * CVE-2020-0543
    - UBUNTU/SAUCE: x86/speculation/srbds: do not try to turn mitigation off when
      not supported
  * Realtek 8723DE [10ec:d723] subsystem [10ec:d738] disconnects unsolicitedly
    when Bluetooth is paired: Reason: 23=IEEE8021X_FAILED (LP: #1878147)
    - SAUCE: Revert "UBUNTU: SAUCE: rtw88: Move driver IQK to set channel before
      association for 11N chip"
    - SAUCE: Revert "UBUNTU: SAUCE: rtw88: fix rate for a while after being
      connected"
    - SAUCE: Revert "UBUNTU: SAUCE: rtw88: No retry and report for auth and assoc"
    - SAUCE: Revert "UBUNTU: SAUCE: rtw88: 8723d: Add coex support"
    - rtw88: add a debugfs entry to dump coex's info
    - rtw88: add a debugfs entry to enable/disable coex mechanism
    - rtw88: 8723d: Add coex support
    - SAUCE: rtw88: coex: 8723d: set antanna control owner
    - SAUCE: rtw88: coex: 8723d: handle BT inquiry cases
    - SAUCE: rtw88: fix EAPOL 4-way failure by finish IQK earlier
  * CPU stress test fails with focal kernel (LP: #1867900)
    - [Config] Disable hisi_sec2 temporarily
  * Enforce all config annotations (LP: #1879327)
    - [Config]: do not enforce CONFIG_VERSION_SIGNATURE
    - [Config]: prepare to enforce all
    - [Config]: enforce all config options
  * Focal update: v5.4.44 upstream stable release (LP: #1881927)
    - ax25: fix setsockopt(SO_BINDTODEVICE)
    - dpaa_eth: fix usage as DSA master, try 3
    - net: don't return invalid table id error when we fall back to PF_UNSPEC
    - net: dsa: mt7530: fix roaming from DSA user ports
    - net: ethernet: ti: cpsw: fix ASSERT_RTNL() warning during suspend
    - __netif_receive_skb_core: pass skb by reference
    - net: inet_csk: Fix so_reuseport bind-address cache in tb->fast*
    - net: ipip: fix wrong address family in init error path
    - net/mlx5: Add command entry handling completion
    - net: mvpp2: fix RX hashing for non-10G ports
    - net: nlmsg_cancel() if put fails for nhmsg
    - net: qrtr: Fix passing invalid reference to qrtr_local_enqueue()
    - net: revert "net: get rid of an signed integer overflow in
      ip_idents_reserve()"
    - net sched: f...

Changed in linux-kvm (Ubuntu Focal):
status: Fix Committed → Fix Released
Michał Sawicz (saviq) wrote :

Hey all, will we get backports of this to previous releases?

Stéphane Graber (stgraber) wrote :

We weren't planning to as the previous releases (xenial and bionic) did not have "-kvm" image and their default image includes an initrd making them boot just fine under LXD.

So it's really just groovy+focal that we need before we can start using those images.
focal has been taken care of so we're just waiting for linux-kvm to hit the release pocket on groovy before we can switch over to those.

Launchpad Janitor (janitor) wrote :
Download full text (31.6 KiB)

This bug was fixed in the package linux-kvm - 5.4.0-1020.20

---------------
linux-kvm (5.4.0-1020.20) focal; urgency=medium

  * focal/linux-kvm: 5.4.0-1020.20 -proposed tracker (LP: #1887063)

  [ Ubuntu: 5.4.0-42.46 ]

  * focal/linux: 5.4.0-42.46 -proposed tracker (LP: #1887069)
  * linux 4.15.0-109-generic network DoS regression vs -108 (LP: #1886668)
    - SAUCE: Revert "netprio_cgroup: Fix unlimited memory leak of v2 cgroups"

linux-kvm (5.4.0-1019.19) focal; urgency=medium

  * focal/linux-kvm: 5.4.0-1019.19 -proposed tracker (LP: #1885848)

  [ Ubuntu: 5.4.0-41.45 ]

  * focal/linux: 5.4.0-41.45 -proposed tracker (LP: #1885855)
  * Packaging resync (LP: #1786013)
    - update dkms package versions
  * CVE-2019-19642
    - kernel/relay.c: handle alloc_percpu returning NULL in relay_open
  * CVE-2019-16089
    - SAUCE: nbd_genl_status: null check for nla_nest_start
  * CVE-2020-11935
    - aufs: do not call i_readcount_inc()
  * ip_defrag.sh in net from ubuntu_kernel_selftests failed with 5.0 / 5.3 / 5.4
    kernel (LP: #1826848)
    - selftests: net: ip_defrag: ignore EPERM
  * Update lockdown patches (LP: #1884159)
    - SAUCE: acpi: disallow loading configfs acpi tables when locked down
  * seccomp_bpf fails on powerpc (LP: #1885757)
    - SAUCE: selftests/seccomp: fix ptrace tests on powerpc
  * Introduce the new NVIDIA 418-server and 440-server series, and update the
    current NVIDIA drivers (LP: #1881137)
    - [packaging] add signed modules for the 418-server and the 440-server
      flavours

  [ Ubuntu: 5.4.0-40.44 ]

  * linux-oem-5.6-tools-common and -tools-host should be dropped (LP: #1881120)
    - [Packaging] Add Conflicts/Replaces to remove linux-oem-5.6-tools-common and
      -tools-host
  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts
  * Slow send speed with Intel I219-V on Ubuntu 18.04.1 (LP: #1802691)
    - e1000e: Disable TSO for buffer overrun workaround
  * CVE-2020-0543
    - UBUNTU/SAUCE: x86/speculation/srbds: do not try to turn mitigation off when
      not supported
  * Realtek 8723DE [10ec:d723] subsystem [10ec:d738] disconnects unsolicitedly
    when Bluetooth is paired: Reason: 23=IEEE8021X_FAILED (LP: #1878147)
    - SAUCE: Revert "UBUNTU: SAUCE: rtw88: Move driver IQK to set channel before
      association for 11N chip"
    - SAUCE: Revert "UBUNTU: SAUCE: rtw88: fix rate for a while after being
      connected"
    - SAUCE: Revert "UBUNTU: SAUCE: rtw88: No retry and report for auth and assoc"
    - SAUCE: Revert "UBUNTU: SAUCE: rtw88: 8723d: Add coex support"
    - rtw88: add a debugfs entry to dump coex's info
    - rtw88: add a debugfs entry to enable/disable coex mechanism
    - rtw88: 8723d: Add coex support
    - SAUCE: rtw88: coex: 8723d: set antanna control owner
    - SAUCE: rtw88: coex: 8723d: handle BT inquiry cases
    - SAUCE: rtw88: fix EAPOL 4-way failure by finish IQK earlier
  * CPU stress test fails with focal kernel (LP: #1867900)
    - [Config] Disable hisi_sec2 temporarily
  * Enforce all config annotations (LP: #1879327)
    - [Config]: do not enforce CONFIG_VERSION_SIGNATURE
    - [Config]: prepare to enforce all
    - [Config]: enforce all config opt...

Changed in linux-kvm (Ubuntu):
status: Triaged → Fix Released

Both the 18.04 & 16.04 Ubuntu Minimal cloud images use the kvm kernel by default, so we need this fix applied to those kernels as well. Are there plans for that so that we can use those Minimal images in LXD?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

  • auto-roufique Edit

Bug watches keep track of this bug in other bug trackers.