xen 4.11 panic at boot because missing support for XZ compressed kernel

[Sébastien Chaumat]

Upgrading to 2110 from 20.04 on a machine running the Xen hypervisor leads to a fail-to-boot into Xen, although it will boot into native Ubuntu.The error message is
   (XEN) ELF: not an ELF binary
   (XEN)
   (XEN) ****************************************
   (XEN) Panic on CPU 0:
   (XEN) Could not set up DOM0 guest OS
   (XEN) ****************************************
   (XEN)
   (XEN) Reboot in 5 seconds...

The problem is well-known: Xen does not understand all of the compression formats for vmlinuz, and cannot load newer kernels that are compressed with the newer formats. Redhat has a bug report (https://bugzilla.redhat.com/show_bug.cgi?id=1898662), ArchLinux has a fix (https://bbs.archlinux.org/viewtopic.php?id=118525). That solution is to ue Torvalds' `extract-vmlinux' script (https://github.com/torvalds/linux/blob/master/scripts/extract-vmlinux) to obtain an uncompressed kernel [a vmlinux-... file] and then correct the module2 line to use the uncompressed kernel in all the XEN-related /boot/grub/grub.cfg entries.

I put extract-vmlinux into /boot, and used it to extract vmlinux-5.13.0-19-generic from vmlinuz-5.13.0-19-generic -- beware, extract-vmlinux takes a filename, but extracts the kernel to stdout (!); so you need to type something like
        # cd /boot
        # ./extract-vmlinux ./vmlinuz-5.13.0-19.generic > ./vmlinux-5.13.0-19-generic
I did this for the 5.13.0-20 kernel too.

Changing the lines in /boot/grup/grub.cfg is simple, here's an example of what mine says now:
       multiboot2 /boot/xen-4.11-amd64.gz placeholder ucode=scan ${xen_rm_opts}
        echo 'Loading Linux 5.13.0-20-generic ...'
        #module2 /boot/vmlinuz-5.13.0-20-generic placeholder root=UUID=cc5800e2-d9d9-4138-b390-014f77aec5ea ro
        module2 /boot/vmlinux-5.13.0-20-generic placeholder root=UUID=cc5800e2-d9d9-4138-b390-014f77aec5ea ro
        echo 'Loading initial ramdisk ...'
        module2 --nounzip /boot/initrd.img-5.13.0-20-generic

The commented-out module2 line is what was present in the original generated grub.cfg; I duplicated the line, commented out the first instance, and changed the 'z' in vmlinuz to 'x' in the second instance.

This will need to be done manually anytime a new kernel is installed; at least until Xen is improved to understand the new kernel compression formats. Given the laissez-faire attitude espoused by Michael Young (a frequent contributor to Xen) in the Redhat report, "I was wonder how long it would be before someone else noticed that", I expect it may never be fixed. ArchLinux's solution (essentially what I described above) has been incorporated as a change into their automatic update system, so the uncompressed kernel is always available. Ubuntu could do the same thing.

The answer given below is irrelevent, appearing to be little more than a copy-paste of the top 2 hits from Google search with obvious keywords -- probably without even verifying that they affect the situation. Fortunately, a work-around is given above as a manual activity, and directs us to ArchLinux's automated fix. Note that the work-around can be effected once the system boots into Linux-without-Xen, which can be selected from the GRUB boot menu; so the system is repairable.

[Christopher Dutchyn (cdutchyn)]

Yup, that's my bug-report and manual fix on launchpad.net/ubuntu (https://answers.launchpad.net/ubuntu/+source/xen/+question/699338). Good on Mr Chaumat for duplicating it.

[Andrea C G Mennucci (am70xk)]

as a side note: use

# ./extract-vmlinux ./vmlinuz-5.13.0-19.generic > ./vmlinux-5.13.0-19-generic~~ && mv ./vmlinux-5.13.0-19-generic~~ ./vmlinux-5.13.0-19-generic

do not overwrite the input with the output, you may run into problems