Ubuntu 20.04 Xen Dom0 Cannot Boot

[David Barton]

I have an Ubuntu 20.04 Server running Xen as a Dom0. It's worked fun, but I recently rebooted and the error below appeared

(XEN) ELF: not an ELF binary
   (XEN)
   (XEN) ****************************************
   (XEN) Panic on CPU 0:
   (XEN) Could not set up DOM0 guest OS
   (XEN) ****************************************
   (XEN)
   (XEN) Reboot in 5 seconds...

The LTS release is the same as before, no distribution upgrade. Just a new kernel was installed causing the failure. I tried the extract-vmlinux but I think I might be missing some decompression software as it did not work.

In the end I uninstalled the newer kernels, leaving my last good kernel. This forced Xen to boot into that (vmlinuz-5.11.0-43-generic) which works perfectly.

It's not great having to run an old version and being forced to run extract-vmlinux (If it worked, which it does not) each time I upgrade the kernel is pretty bad. Since it impacts the Dom0 it's pretty significant.

This bug is related to
https://answers.launchpad.net/ubuntu/+source/xen/+question/699338 and https://bugs.launchpad.net/ubuntu/+source/xen/+bug/1956166

[David Barton (dave-oneit)]

Even being able to select an uncompressed kernel image via apt would be a step up.

I found this issue a week back when trying to set up Ubuntu 20.04 as new DomUs. This was on an older 18.04 Xen host, so I was planning on upgrading and retesting. It seems like Ubuntu 20.04 would not longer run in an Ubuntu Dom0 of any flavour.

[Bryce Harrington (bryce)]

Hi David,

You're right, this appears to be well-known issue with xen, e.g. see also LP: #1854575; you already seem familiar with workarounds, and sounds like you've already resolved the problem for yourself. So rather than dupe this bug to one of those, this report can focus on the wishlist request for better packaging to handle kernel upgrades with xen.

With Ubuntu 20.04, xen has moved from Ubuntu main to universe for maintenance/support. Because of this, we should look more towards Debian for a solution to this. Can you please file a bug report about this issue with Debian? Let us know the URL of the bug report, and we'll link this one to that so we can follow discussions and evaluate backporting the fix if/when one becomes available.

Thanks ahead of time!

[David Barton (dave-oneit)]

Thanks, Bryce. I can report it on Debian, but not sure what version to report it against.

Given that 20.04 is an LTS, doesn't changing the compression of the kernel in a way that causes breakages a significant regression?

[Paride Legovini (paride)]

Hi David, ideally you should test the version that is in Debian at the moment before reporting a bug to Debian. As an interim data point: can you verify if the issue is still present in Ubuntu 22.04 (Jammy)? Thanks!

[David Barton (dave-oneit)]

@paride I finally got my hands on an Ubuntu 22.04 server. I am able to get an Ubuntu 20.04 VM working in an Ubuntu 22.04 DomU

An Ubuntu 22.04 VM does not appear to work, although I have libzstd1 and zstd installed.

xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD decompress support unavailable
: Internal error
xc: error: panic: xg_dom_bzimageloader.c:772: xc_dom_probe_bzimage_kernel unable to ZSTD decompress kernel: Invalid kernel
xc: error: panic: xg_dom_core.c:690: xc_dom_find_loader: no loader found: Invalid kernel

If Xen is no longer main for Ubuntu, does that I should be using direct Debian or potentially even switching across to kvm?