Comment 14 for bug 1956166

Tests steps w/ a KVM guest from multipass with the libvirt driver
(for serial console) with original packages and patched packages
from ppa:mfo/lp1956166.

Host Setup:

$ sudo snap install multipass
$ sudo multipass set local.driver=libvirt
$ sudo snap connect multipass:libvirt

$ sudo apt install libvirt-daemon{,-system{,-systemd},-driver-qemu} qemu-system dnsmasq
$ sudo systemctl restart libvirtd.service
$ virsh net-start default # or reboot

Create a Focal VM for Xen, and log into it.

$ multipass launch -c 2 -d 8g -m 6g -n xen-focal focal
$ multipass shell xen-focal

Install Xen, configure serial console on grub/linux/xen:

$ sudo apt update
$ sudo apt install --yes xen-hypervisor-4.11-amd64

$ dpkg -s xen-hypervisor-4.11-amd64 | grep Version:
Version: 4.11.3+24-g14b62ab3e5-1ubuntu2

$ echo 'GRUB_CMDLINE_XEN_DEFAULT="com1=115200,8n1 console=com1 noreboot=true"' | sudo tee -a /etc/default/grub.d/xen.cfg

$ cat <<EOF | sudo tee /etc/default/grub.d/99-serial.cfg
GRUB_CMDLINE_LINUX_DEFAULT='console=ttyS0'
GRUB_TERMINAL=serial
GRUB_TIMEOUT=5
GRUB_TIMEOUT_STYLE=
GRUB_RECORDFAIL_TIMEOUT=
EOF

$ sudo update-grub

$ sudo passwd ubuntu # for serial console access
$ sudo reboot

Switch to serial console:

Check that Xen Dom0 works:

$ virsh console xen-focal

 |*Advanced options for Ubuntu GNU/Linux (with Xen hypervisor) |

 |*Xen hypervisor, version 4.11-amd64 |

 |*Ubuntu GNU/Linux, with Xen 4.11-amd64 and Linux 5.4.0-120-generic |

Loading Xen 4.11-amd64 ...
Loading Linux 5.4.0-120-generic ...
Loading initial ramdisk ...
(XEN) Xen version 4.11.4-pre (Ubuntu 4.11.3+24-g14b62ab3e5-1ubuntu2) ...
...
(XEN) Brought up 2 CPUs
(XEN) Dom0 has maximum 400 PIRQs
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x3a00000
...
[ 0.000000] Linux version 5.4.0-120-generic ...
...

$ sudo xl info | head -n4
host : xen-focal
release : 5.4.0-120-generic
version : #136-Ubuntu SMP Fri Jun 10 13:40:48 UTC 2022
machine : x86_64

Install HWE kernel:

$ sudo apt install --yes linux-image-generic-hwe-20.04-edge

$ ls -1 /boot/{vmlinuz,initrd.img}-5.15.0-*-generic
/boot/initrd.img-5.15.0-33-generic
/boot/vmlinuz-5.15.0-33-generic

Tests w/ the 5.15 kernel:
========================

DomU / HVM: (works)
---

cat <<EOF >xen-5.15-vm.hvm
name = "xen-5.15-vm.hvm"
type = "hvm"

kernel = "/boot/vmlinuz-5.15.0-33-generic"
ramdisk = "/boot/initrd.img-5.15.0-33-generic"
cmdline = "console=ttyS0 break=top"

vcpus = 1
memory = 1024
serial = [ "pty" ]
vnc = 0
EOF

$ sudo xl create -c xen-5.15-vm.hvm
Parsing config from xen-5.15-vm.hvm
<Wait until QEMU's CPU% goes down.>
...
[ 0.000000] Linux version 5.15.0-33-generic ...
...

(initramfs) uname -r
5.15.0-33-generic

(initramfs) dmesg | grep DMI:
[ 0.000000] DMI: Xen HVM domU, BIOS 4.11.4-pre 03/10/2020

Press ctrl+]

$ sudo xl destroy xen-5.15-vm.hvm

DomU / PV: (fails)
---

cat <<EOF >xen-5.15-vm.pv
name = "xen-5.15-vm.pv"
type = "pv"

kernel = "/boot/vmlinuz-5.15.0-33-generic"
ramdisk = "/boot/initrd.img-5.15.0-33-generic"
cmdline = "console=ttyS0 break=top"

vcpus = 1
memory = 1024
EOF

$ sudo xl create -c xen-5.15-vm.pv
Parsing config from xen-5.15-vm.pv
xc: error: panic: xc_dom_bzimageloader.c:774: xc_dom_probe_bzimage_kernel: unknown compression format: Invalid kernel
xc: error: panic: xc_dom_core.c:691: xc_dom_find_loader: no loader found: Invalid kernel
libxl: error: libxl_dom.c:722:libxl__build_dom: xc_dom_parse_image failed
libxl: error: libxl_create.c:1267:domcreate_rebuild_done: Domain 6:cannot (re-)build domain: -3
libxl: error: libxl_domain.c:1034:libxl__destroy_domid: Domain 6:Non-existant domain
libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain 6:Unable to destroy guest
libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain 6:Destruction of domain failed

Dom0 / Host: (fails)
---

$ sudo reboot
$ virsh console xen-focal

 |*Advanced options for Ubuntu GNU/Linux (with Xen hypervisor) |

 |*Xen hypervisor, version 4.11-amd64 |

 |*Ubuntu GNU/Linux, with Xen 4.11-amd64 and Linux 5.15.0-33-generic |

Loading Xen 4.11-amd64 ...
Loading Linux 5.15.0-33-generic ...
Loading initial ramdisk ...
(XEN) Xen version 4.11.4-pre (Ubuntu 4.11.3+24-g14b62ab3e5-1ubuntu2) ...
...
(XEN) Brought up 2 CPUs
(XEN) Dom0 has maximum 400 PIRQs
(XEN) ELF: not an ELF binary
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Could not set up DOM0 guest OS
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)

Patched packages:
---

$ sudo add-apt-repository --yes ppa:mfo/lp1956166

$ sudo apt install $(apt list --upgradable 2>/dev/null | cut -d/ -f1 | grep xen)

$ dpkg -s xen-hypervisor-4.11-amd64 | grep Version:
Version: 4.11.3+24-g14b62ab3e5-1ubuntu2.1

$ sudo reboot

Dom0 / Host: (now works)
---

 |*Advanced options for Ubuntu GNU/Linux (with Xen hypervisor) |

 |*Xen hypervisor, version 4.11-amd64 |

 |*Ubuntu GNU/Linux, with Xen 4.11-amd64 and Linux 5.15.0-33-generic |

Loading Xen 4.11-amd64 ...
Loading Linux 5.15.0-33-generic ...
Loading initial ramdisk ...
(XEN) Xen version 4.11.4-pre (Ubuntu 4.11.3+24-g14b62ab3e5-1ubuntu2.1) ...
...
(XEN) Brought up 2 CPUs
(XEN) Dom0 has maximum 400 PIRQs
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x4000000
...
[ 0.000000] Linux version 5.15.0-33-generic ...
...

$ sudo xl info | head -n4
host : xen-focal
release : 5.15.0-33-generic
version : #34~20.04.1-Ubuntu SMP Thu May 19 15:51:16 UTC 2022
machine : x86_64

DomU / PV (now works):
---

$ sudo xl create -c xen-5.15-vm.pv
Parsing config from xen-5.15-vm.pv
[ 0.000000] Linux version 5.15.0-33-generic ...
...
[ 0.000000] Hypervisor detected: Xen PV
...

DomU / HVM (still work):

$ sudo xl create -c xen-5.15-vm.hvm
Parsing config from xen-5.15-vm.hvm
[ 0.000000] Linux version 5.15.0-33-generic ...
...
(initramfs) uname -r
5.15.0-33-generic
...
(initramfs) dmesg | grep DMI:
[ 0.000000] DMI: Xen HVM domU, BIOS 4.11.4-pre 07/04/2022

---

Tests w/ the 5.4 kernel: (regression testing)
=======================

Dom0:
---

 | Ubuntu GNU/Linux, with Xen 4.11.4-pre and Linux 5.4.0-120-generic |

$ sudo xl info | head -n4
host : xen-focal
release : 5.4.0-120-generic
version : #136-Ubuntu SMP Fri Jun 10 13:40:48 UTC 2022
machine : x86_64

DomU/HVM: (still work)
---

$ cat <<EOF >xen-5.4-vm.hvm
name = "xen-5.4-vm.hvm"
type = "hvm"

kernel = "/boot/vmlinuz-5.4.0-120-generic"
ramdisk = "/boot/initrd.img-5.4.0-120-generic"
cmdline = "console=ttyS0 break=top"

vcpus = 1
memory = 1024
serial = [ "pty" ]
vnc = 0
EOF

$ sudo xl create -c xen-5.4-vm.hvm
Parsing config from xen-5.4-vm.hvm
[ 0.000000] Linux version 5.4.0-120-generic (buildd@lcy02-amd64-006) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #136-Ubuntu SMP Fri Jun 10 13:40:48 UTC 2022 (Ubuntu 5.4.0-120.136-generic 5.4.189)

(initramfs) uname -r
5.4.0-120-generic

(initramfs) dmesg | grep DMI:
[ 0.000000] DMI: Xen HVM domU, BIOS 4.11.4-pre 07/04/2022

Press ctrl+]

$ sudo xl destroy xen-5.4-vm.hvm

DomU/PV: (still work)
---

$ cat <<EOF >xen-5.4-vm.pv
name = "xen-5.4-vm.pv"
type = "pv"

kernel = "/boot/vmlinuz-5.4.0-120-generic"
ramdisk = "/boot/initrd.img-5.4.0-120-generic"
cmdline = "console=ttyS0 break=top"

vcpus = 1
memory = 1024
EOF

ubuntu@xen-focal:~$ sudo xl create -c xen-5.4-vm.pv
Parsing config from xen-5.4-vm.pv
[ 0.000000] Linux version 5.4.0-120-generic ...
...