Ubuntu 22.04 doesn't boot with xen

[Belgarath]

[Note]

 * This bug addresses Dom0 on Focal and later,
   plus DomU pv/pvh/pygrub on Focal _only_
   (unintended inconsistency).

 * For DomU pv/pvh/pygrub on Jammy and later,
   see bug 1998026.

[Impact]

 * Xen fails to boot as Dom0 and DomU PV (Dom0 HVM works)
   with the HWE kernel from Jammy (5.15) on Focal.

 * Xen 4.11 doesn't have support zstd kernel compression,
   which was added in Xen 4.15, but it backports nicely.
   (First 3 patches for Dom0, and 2 patches for DomU PV).

   ~/git/xen$ git log --oneline RELEASE-4.11.3..RELEASE-4.15.0
   ...
   8169f82049ef libxenguest: support zstd compressed kernels
   d8099d94dfaa libxenguest: add get_unaligned_le32()
   ...
   d6627cf1b63c x86/Dom0: support zstd compressed kernels
   35d2960ae65f lib: introduce xxhash
   7c9f81687ad6 introduce unaligned.h
   ...

[Test Plan]

 * On Focal, test Xen with the 5.15 HWE kernel from Jammy,
   on the 3 modes (Dom0, DomU PV, and DomU HVM).

 * Likewise, check for regressions with the 5.4 GA kernel.

 * Basically (full steps in comment #14), on a KVM guest
   with serial console enabled for grub/xen/linux kernel:

   $ sudo apt install --yes xen-hypervisor-4.11-amd64 linux-image-generic-hwe-20.04-edge

   Before:

   * Dom0:

   - Boot 'Ubuntu GNU/Linux, with Xen 4.11-amd64 and Linux 5.15.0-33-generic'
     ...
     (XEN) Brought up 2 CPUs
     (XEN) Dom0 has maximum 400 PIRQs
     (XEN) ELF: not an ELF binary
     ...
     (XEN) Could not set up DOM0 guest OS

   * DomU PV:

     $ sudo xl create -c xen-5.15-vm.pv
     Parsing config from xen-5.15-vm.pv
     xc: error: panic: xc_dom_bzimageloader.c:774: xc_dom_probe_bzimage_kernel: unknown compression format: Invalid kernel
     ...

   * DomU HVM: (works; it runs in QEMU)

     $ sudo xl create -c xen-5.15-vm.hvm
     Parsing config from xen-5.15-vm.hvm
     [ 0.000000] Linux version 5.15.0-33-generic ...
     ...
     [ 0.000000] DMI: Xen HVM domU, BIOS 4.11.4-pre 07/04/2022
     ...

   After: (ppa:mfo/lp1956166)

   * Dom0:

   - Boot 'Ubuntu GNU/Linux, with Xen 4.11-amd64 and Linux 5.15.0-33-generic'
     ...
     (XEN) Brought up 2 CPUs
     (XEN) Dom0 has maximum 400 PIRQs
     (XEN) Xen kernel: 64-bit, lsb, compat32
     (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x4000000
     ...
     [ 0.000000] Linux version 5.15.0-33-generic ...

   * DomU PV:

     $ sudo xl create -c xen-5.15-vm.pv
     Parsing config from xen-5.15-vm.pv
     [ 0.000000] Linux version 5.15.0-33-generic ...
     ...
     [ 0.000000] Hypervisor detected: Xen PV
     ...

   * DomU HVM: (still work)

     $ sudo xl create -c xen-5.15-vm.hvm
     Parsing config from xen-5.15-vm.hvm
     [ 0.000000] Linux version 5.15.0-33-generic ...
     ...
     [ 0.000000] DMI: Xen HVM domU, BIOS 4.11.4-pre 07/04/2022
     ...

[Where problems could occur]

 * Even though the patches _add a lot_ of code to support zstd
   (ie, read/decompress), the actual _changes_ to existing code
   are usually just calling its decompression functions after
   the other compression methods. (And some helper factor-out).

 * Regressions would likely manifest in the kernel decompression
   functions, i.e., _boot_, so should be quick/easy to find out
   (with older/non-zstd compressed kernel versions; eg, GA/5.4).

[Other Info]

 * Only Focal needs a SRU. The patches are applied in Xen 4.15,
   so Kinetic and Jammy (4.16) already have it, and Impish and
   Focal (4.11) don't; but Impish is almost EOL, so, only Focal.

 * The DomU PV mode hits kernel errors during early boot, and
   doesn't get to the initramfs. These issues seem related to
   the PV mode only (seen elsewhere, and regardless of kernel
   version, 5.4 or 5.15), but it looks like the DomU HVM mode
   is more used in practice?

   So, theoretically, Ubuntu support for DomU PV wouldn't be
   needed, but maybe let's just add it all at once, in case
   users are booting other kernels that are patched to work.

[Original Bug Description]

Ubuntu 22.04 doesn't boot with xen
kernels compressed using zstd are only supported in upstream xen 4.15 but xen-4.11 is installed in ubuntu 22.04 together with zstd compressed kernel 5.13.0-19

Either compression needs to be changed to one that is supported by xen or xen packages needs to be upgraded to 4.15 or higher.

[Belgarath (belgarath)]

This affect Dom0 boot and DomU

[Christian Ehrhardt  (paelzer)]

Thanks for the report Belgarath.

Do happen to already know the commit(s) that added this to 4.15 and have a link?

Assigning to smb for xen opinions ...

@smb - not for updating to 4.15 (violating the SRU policy) but maybe for identifying the fix and backporting it at least to Focal?

@smb - And OTOH maybe getting rid of the remaining delta to sync 4.14.3+32-g9de3671772-1 and hopefully 4.15 before the end of the cycle to at least fix jammy? Any chance to get qemu-system-x86-xen into Debian? Or at least have that to be the only trivially merged delta left?

[Mark Pryor (prymar56)]

As Belgarath says, it takes xen-4.15+

tlviewer@xen2204:~$ sudo xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 600 2 r----- 45.8
ub2004 1 804 2 -b---- 10.7
ub2204 2 360 2 -b---- 9.2
fc34min 3 604 2 -b---- 22.8
al8pv 4 854 2 -b---- 13.4
fc35min 5 784 2 -b---- 28.3

sudo systemd-analyze time
Startup finished in 8.070s (kernel) + 32.310s (userspace) = 40.380s
multi-user.target reached after 32.271s in userspace

tlviewer@xen2204:~$ sudo xl info
host : xen2204
release : 5.13.0-19-generic
version : #19-Ubuntu SMP Thu Oct 7 21:58:00 UTC 2021
machine : x86_64
nr_cpus : 2
max_cpu_id : 3
nr_nodes : 1
cores_per_socket : 2
threads_per_core : 1
cpu_mhz : 3062.672
hw_caps : bfebfbff:0008e3bd:20100800:00000001:00000000:00000000:00000000:00000000
virt_caps : pv hvm shadow
total_memory : 4094
free_memory : 36
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 15
xen_extra : .1
xen_version : 4.15.1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit2
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : a763f8f158e81158b-x86/spec-ctrl: Fix default calculation of opt
xen_commandline : dom0_mem=600M,max:648M guest_loglvl=all loglvl=all sched=credit2 console=vga vga=gfx-1280x1024x16 conring_size=48k xsave=0 xpti=false cpufreq=xen:performance
cc_compiler : gcc (Ubuntu 11.2.0-13ubuntu1) 11.2.0
cc_compile_by : pbuser
cc_compile_domain : prymar56.org
cc_compile_date : Sat Jan 8 13:42:29 PST 2022
build_id : 3a9d3ef047996d4f6be41d9dadda2363d6538db7
xend_config_format : 4

tlviewer@xen2204:~$ sudo dpkg -l | grep -P 'xen|qemuu'
ii libxen-4.15:amd64 4.15.1-1+ub22u04.2 amd64 Public libs for Xen
ii libxenstore3.0:amd64 4.15.1-1+ub22u04.2 amd64 Xenstore communications library for Xen
ii libxentoolcore1:amd64 4.15.1-1+ub22u04.2 amd64 helper for qemu & libxenstore
ii qemuu 6.1.0+1-1+ub22u04.2 amd64 qemu-system-i386 (6.1.0/xen-4.15) with 9pfs support
ii xen-hypervisor-4.15-amd64 4.15.1-1+ub22u04.2 amd64 Xen Hypervisor on AMD64
ii xen-system-amd64 4.15.1-1+ub22u04.2 amd64 Xen System on AMD64 (meta-package)
ii xen-utils-4.15 4.15.1-1+ub22u04.2 ...

[Christian Ehrhardt  (paelzer)]

FYI - I wanted to look if I could make at least xen 4.16 available on jammy as I really think a new LTS should have a newer xen, no matter how much it is in universe :-)

But unfortunately there are awkward issues blocking further progress at the moment, with platform dependent segfaults: https://bugs.launchpad.net/ubuntu/+source/xen/+bug/1958389

[Christian Ehrhardt  (paelzer)]

It would be great if anyone with xen experience and setup would be willing to test the PPA [1] with:
- xen 4.16
- qemu 6.2
- libvirt 8.0

I'd be especially interested if it works in regard to this bug here with the different kernel compression.

But you can test it in general and let me know, I can't promise to to fix everything - but at least better than the outdated 4.11 it should be.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4753

[Stefan Bader (smb)]

The Jammy 5.15 kernel comes up as dom0 with the xen-4.16 from the PPA. I can also interact with libvirt:

# uname -a
Linux argabuthon 5.15.0-18-generic #18-Ubuntu SMP Fri Jan 21 14:57:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

# virsh version
Compiled against library: libvirt 8.0.0
Using library: libvirt 8.0.0
Using API: Xen 8.0.0
Running hypervisor: Xen 4.16.0

Right now the creation of a HVM guest fails because a matching qemu is not found:

libvirtd[1183]: unsupported configuration: emulator '/usr/lib/xen/bin/qemu-system-i386' not found

# dpkg -L qemu-system-x86-xen
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/qemu-system-x86-xen
/usr/share/doc/qemu-system-x86-xen/copyright
/usr/share/doc/qemu-system-x86-xen/changelog.Debian.gz

[Christian Ehrhardt  (paelzer)]

Thanks Stefan, the qemu builds qemu - 1:6.2+dfsg-2ubuntu1~jammyppa8 and later have the xen supporting qemu built and a proper compat symlink now.

root@j:~# dpkg -L qemu-system-x86-xen | grep qemu-system-i386
/usr/bin/qemu-system-i386
/usr/share/man/man1/qemu-system-i386.1.gz
/usr/lib/xen/bin/qemu-system-i386
root@j:~# ll /usr/lib/xen/bin/qemu-system-i386
lrwxrwxrwx 1 root root 29 Jan 5 11:18 /usr/lib/xen/bin/qemu-system-i386 -> ../../../bin/qemu-system-i386*

I'd be happy about a re-test with that build.

[Stefan Bader (smb)]

Ok, with the updates in the PPA, I managed to bring up a jammy HVM

# virsh list
 Id Name State
-------------------------------
 0 Domain-0 running
 6 arg-jammy6401 running

There will maybe be some remaining problems with PV guests and some other odd things (though some I could certainly remember once I hit them). But at least it is overall in a working state.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christian Ehrhardt  (paelzer)]

Ok, that at least makes it better than the very-outdated 4.11 version.
Thanks for the testing smb!

Will probably upload this to Jammy after review of the MP.
https://code.launchpad.net/~paelzer/ubuntu/+source/xen/+git/xen/+merge/414651

[Belgarath (belgarath)]

Hi,

I upgraded working xen ubuntu 20.04 (up to date)
to xen ubuntu 22.04 (that fails as expected)

I then booted non-xen kernel
added repository and did apt full-upgrade

that still doesn't boot the xen kernel as Dom0.

installing "xen-hypervisor-4.16" resolved the issue
but if you have package "xen-hypervisor" installed you will expect that to point to the latest version so probably that needs to be updated both in PPA and 22.04

[Launchpad Janitor (janitor)]

This bug was fixed in the package xen - 4.16.0-1~ubuntu2

---------------
xen (4.16.0-1~ubuntu2) jammy; urgency=medium

  * Merge Debian experimental/salsa, among many other changes this fixes
    booting kernels with zstd compression (LP: #1956166).
    Remaining changes:
    - Recommend qemu-system-x86-xen
  * Dropped changes:
    - Additional patches to handle compiling with gcc10 [in upstream]
    - Select python2 for xen-init-* scripts [Debian is on python3 now]
    - Enforce python2 usage [Debian is on python3 now]
      - Build-depend on python2-dev.
      - Build using python2.
      - Build-depend on lmodern.
    - Handle config file moving between packages [no more needed]
  * Note changes already dropped in 4.11.4+24-gddaaccbbab-1ubuntu2
    - Set python2 for xen-init-name and xen-init-list scripts
      [in Debian]
    - Force fcf-protection off when using -mindirect-branch
      [fixed upstream]
    - Update: Building hypervisor with cf-protection enabled
      [fixed upstream]
    - Strip .note.gnu.property section for intermediate files
      [no more needed with the groovy toolchain]
    - Add transitional packages for upgrades
      [no more needed post focal]

xen (4.16.0-1~exp1) experimental; urgency=medium

  Significant changes:
  * Update to new upstream version 4.16.0. This also includes a security fix
    for the following issue, which was not applicable to Xen 4.14 yet:
    - certain VT-d IOMMUs may not work in shared page table mode
      XSA-390 CVE-2021-28710
  * No longer build any package for the i386 architecture. It was already not
    possible to use x86_32 hardware because the i386 packages already
    shipped a 64-bit hypervisor and PV shim. Running 32-bit utils with a
    64-bit hypervisor requires using a compatibility layer that is fragile and
    becomes harder to maintain and test upstream. This change ends the 'grace
    period' in which users should have moved to using a fully 64-bit dom0.
    - debian/{control,rules,salsa-ci.yml,xen-utils-V.install.vsn-in}: make the
      necessary changes
    - Remove the Recommends on libc6-xen, which already actually does not
      exist any more. (Closes: #992909)
    - Drop patch "tools/tests/x86_emulator: Pass -no-pie -fno-pic to gcc on
      x86_32" because it is not relevant any more.

  Changes related to upgrading to Xen 4.16:
  * debian/control: adjust to 4.16 [Maximilian Engelhardt]
  * Drop patches that have been applied upstream
  * Refresh remaining patches if needed
  * debian: follow upstream removal of '.sh' suffix in xl bash_completion file
    [Maximilian Engelhardt]
  * debian/control, debian/libxenstore*: ship a libxenstore4 package instead
    of libxenstore3.0, since upstream bumped the soname
    [Maximilian Engelhardt]

  Packaging minor fixes and improvements [Maximilian Engelhardt]:
  * debian/rules: set SOURCE_BASE_DIR to the top level build dir so that the
    "Display Debian package version in hypervisor log" patch can use it.
  * Add patch "xen/arch/x86: make objdump output user locale agnostic" to fix
    reproducable builds. This patch will also be sent upstream.
  * d/rules: remove reproducible=+fixfilepath from DEB_BUILD_MAINT_OP...

[Mauricio Faria de Oliveira (mfo)]

Hey Christian, Stefan,

Here's a MR [1] to add support for zstd compression to Focal's Xen 4.11.

It's a straightforward backport from Xen 4.15 (just too many additions
to implement zstd, and hook its functions in kernel decompression code).

The test packages in ppa:mfo/lp1956166 [2] can successfully boot a 5.15
kernel in Dom0, DomU/PV, and DomU/HVM (unchanged) modes, and continue to
boot the 5.4 kernel successfully (some level of 'no regression' testing).

All test steps are provided in the next comment.

Could you please review / provide feedback to the MR?

Thanks!
Mauricio

[1] https://code.launchpad.net/~mfo/ubuntu/+source/xen/+git/xen/+merge/426221
[2] https://launchpad.net/~mfo/+archive/ubuntu/lp1956166

[Mauricio Faria de Oliveira (mfo)]

Tests steps w/ a KVM guest from multipass with the libvirt driver
(for serial console) with original packages and patched packages
from ppa:mfo/lp1956166.

Host Setup:

$ sudo snap install multipass
$ sudo multipass set local.driver=libvirt
$ sudo snap connect multipass:libvirt

$ sudo apt install libvirt-daemon{,-system{,-systemd},-driver-qemu} qemu-system dnsmasq
$ sudo systemctl restart libvirtd.service
$ virsh net-start default # or reboot

Create a Focal VM for Xen, and log into it.

$ multipass launch -c 2 -d 8g -m 6g -n xen-focal focal
$ multipass shell xen-focal

Install Xen, configure serial console on grub/linux/xen:

$ sudo apt update
$ sudo apt install --yes xen-hypervisor-4.11-amd64

$ dpkg -s xen-hypervisor-4.11-amd64 | grep Version:
Version: 4.11.3+24-g14b62ab3e5-1ubuntu2

$ echo 'GRUB_CMDLINE_XEN_DEFAULT="com1=115200,8n1 console=com1 noreboot=true"' | sudo tee -a /etc/default/grub.d/xen.cfg

$ cat <<EOF | sudo tee /etc/default/grub.d/99-serial.cfg
GRUB_CMDLINE_LINUX_DEFAULT='console=ttyS0'
GRUB_TERMINAL=serial
GRUB_TIMEOUT=5
GRUB_TIMEOUT_STYLE=
GRUB_RECORDFAIL_TIMEOUT=
EOF

$ sudo update-grub

$ sudo passwd ubuntu # for serial console access
$ sudo reboot

Switch to serial console:

Check that Xen Dom0 works:

$ virsh console xen-focal

 |*Advanced options for Ubuntu GNU/Linux (with Xen hypervisor) |

 |*Xen hypervisor, version 4.11-amd64 |

 |*Ubuntu GNU/Linux, with Xen 4.11-amd64 and Linux 5.4.0-120-generic |

Loading Xen 4.11-amd64 ...
Loading Linux 5.4.0-120-generic ...
Loading initial ramdisk ...
(XEN) Xen version 4.11.4-pre (Ubuntu 4.11.3+24-g14b62ab3e5-1ubuntu2) ...
...
(XEN) Brought up 2 CPUs
(XEN) Dom0 has maximum 400 PIRQs
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x3a00000
...
[ 0.000000] Linux version 5.4.0-120-generic ...
...

$ sudo xl info | head -n4
host : xen-focal
release : 5.4.0-120-generic
version : #136-Ubuntu SMP Fri Jun 10 13:40:48 UTC 2022
machine : x86_64

Install HWE kernel:

$ sudo apt install --yes linux-image-generic-hwe-20.04-edge

$ ls -1 /boot/{vmlinuz,initrd.img}-5.15.0-*-generic
/boot/initrd.img-5.15.0-33-generic
/boot/vmlinuz-5.15.0-33-generic

Tests w/ the 5.15 kernel:
========================

DomU / HVM: (works)
---

cat <<EOF >xen-5.15-vm.hvm
name = "xen-5.15-vm.hvm"
type = "hvm"

kernel = "/boot/vmlinuz-5.15.0-33-generic"
ramdisk = "/boot/initrd.img-5.15.0-33-generic"
cmdline = "console=ttyS0 break=top"

vcpus = 1
memory = 1024
serial = [ "pty" ]
vnc = 0
EOF

$ sudo xl create -c xen-5.15-vm.hvm
Parsing config from xen-5.15-vm.hvm
<Wait until QEMU's CPU% goes down.>
...
[ 0.000000] Linux version 5.15.0-33-generic ...
...

(initramfs) uname -r
5.15.0-33-generic

(initramfs) dmesg | grep DMI:
[ 0.000000] DMI: Xen HVM domU, BIOS 4.11.4-pre 03/10/2020

Press ctrl+]

$ sudo xl destroy xen-5.15-vm.hvm

DomU / PV: (fails)
---

cat <<EOF >xen-5.15-vm.pv
name = "xen-5.15-vm.pv"
type = "pv"

kernel = "/boot/vmlinuz-5.15.0-33-generic"
ramdisk = "/boot/initrd.img-5.15.0-33-generic"
cmdline = "console=tty...

[Belgarath (belgarath)]

As far as using PV vs HVM I would say that most people who use(d) HVM migrated to KVM already.
PV is XEN distinct advantage over KVM as it improves performance even on machines with full virtualization support, especially for CPU and network-intensive loads.

[Mauricio Faria de Oliveira (mfo)]

Hi Belgarath,

Thanks for the feedback!

Just for reference in the comments, also noted in the description:
in this particular case, the same errors on DomU PV mode are seen
with the 5.4 and 5.15 kernels, so they're not tied to the patchset.

[Christian Ehrhardt  (paelzer)]

Reviewed and accepted from my POV based on [1][2] as presented by Mauricio.
Sponsored to focal-unapproved.
Up to the SRU team now to make the final decision.

[1] https://code.launchpad.net/~mfo/ubuntu/+source/xen/+git/xen/+merge/426221
[2] https://launchpad.net/~mfo/+archive/ubuntu/lp1956166

[Łukasz Zemczak (sil2100)]

Hello Belgarath, or anyone else affected,

Accepted xen into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/xen/4.11.3+24-g14b62ab3e5-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

I checked that indeed the actual changes to existing sources are rather minimal, so hopefully the regression potential is limited. Let's give this a try.

[Mauricio Faria de Oliveira (mfo)]

I see a build failure on armhf, which I'll check, and propose an incremental fix.

Sorry for the extra trouble, I forgot to enable all supported archs in the PPA build.

[Mauricio Faria de Oliveira (mfo)]

Hi Christian,

Could you please review the other MR [1] with an incremental fix?

The issue/fix is explained in the (new) patch header, but briefly,
we have to make the changes in lzo.c in patch 0001 conditional to
x86 (actual effect upstream/Xen 4.15), so that arm doesn't FTBFS.

I can confirm that the generated assembly instructions remain the same:
- on amd64 between the first and second MRs (ie, change is still active);
- on arm64 between the original package and the second MR (ie, no change).
(details in the MR)

Thanks!

[1] https://code.launchpad.net/~mfo/ubuntu/+source/xen/+git/xen/+merge/426503

[Mauricio Faria de Oliveira (mfo)]

P.S.: this time I enabled all architectures in the PPA [1],
and package v2 built successfully on amd64/arm64/armhf [2].

[1] https://launchpad.net/~mfo/+archive/ubuntu/lp1956166
[2] https://launchpad.net/~mfo/+archive/ubuntu/lp1956166/+sourcepub/13799054/+listing-archive-extra

[Christian Ehrhardt  (paelzer)]

Thanks for expanding the test build architectures Mauricio.
Xen is already rare on x86 nowadays, much more so on arm I'd think - I didn't think of arm either when reviewing :-/

The fixup LGTM, sponsored.

[Luís Infante da Câmara (luis220413)]

I am preparing a security update for Xen in Ubuntu 20.04 that includes the fixes for this bug.

[Mauricio Faria de Oliveira (mfo)]

Luís, thank you very much for the work on the security update in bug 1970507.

For now, I have to ask our security team (did it in that bug) to please try
and land these changes first, as that bug's changes are relatively large in
comparison -- and it'd be nice to have a way back/downgrade that still have
these fixes, in the inevitable risk of regressions, proportional to changes.

Thank again!

[Łukasz Zemczak (sil2100)]

Hello Belgarath, or anyone else affected,

Accepted xen into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/xen/4.11.3+24-g14b62ab3e5-1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauricio Faria de Oliveira (mfo)]

Verification done for focal-proposed; all good!

See test steps in comment #14 and results below.

cheers!

Test environment:
================

$ hostnamectl status | grep Virtualization:
    Virtualization: kvm

$ lsb_release -cs
focal

$ sudo apt install --yes \
  xen-hypervisor-4.11-amd64 \
  linux-image-generic-hwe-20.04-edge

$ rmadison -a source -s focal-proposed xen
 xen | 4.11.3+24-g14b62ab3e5-1ubuntu2.2 | focal-proposed/universe | source

$ dpkg -s xen-hypervisor-4.11-amd64 | grep Version:
Version: 4.11.3+24-g14b62ab3e5-1ubuntu2.2

$ dpkg -L xen-hypervisor-4.11-amd64 | grep ^/boot
/boot
/boot/xen-4.11-amd64.config
/boot/xen-4.11-amd64.efi
/boot/xen-4.11-amd64.gz

$ ls -1 /boot/{vmlinuz,initrd.img}-5.15.0-*-generic
/boot/initrd.img-5.15.0-41-generic
/boot/vmlinuz-5.15.0-41-generic

$ ls -1 /boot/{vmlinuz,initrd.img}-5.4.0-*-generic
/boot/initrd.img-5.4.0-122-generic
/boot/vmlinuz-5.4.0-122-generic

Tests with 5.15 kernel: (problem fixed)
======================

Dom0 / Host:
---

 |*Ubuntu GNU/Linux, with Xen 4.11-amd64 and Linux 5.15.0-41-generic |
...
Loading Xen 4.11-amd64 ...
Loading Linux 5.15.0-41-generic ...
Loading initial ramdisk ...
(XEN) Xen version 4.11.4-pre (Ubuntu 4.11.3+24-g14b62ab3e5-1ubuntu2.2) ...
...
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x4000000
...
about to get started...
[ 0.000000] Linux version 5.15.0-41-generic ...
...

$ sudo xl info | head -n4
host : xen-f
release : 5.15.0-41-generic
version : #44~20.04.1-Ubuntu SMP Fri Jun 24 13:27:29 UTC 2022
machine : x86_64

DomU / HVM:
---

$ sudo xl create -c xen-5.15-vm.hvm
...
(initramfs) uname -rv
5.15.0-41-generic #44~20.04.1-Ubuntu SMP Fri Jun 24 13:27:29 UTC 2022
(initramfs) dmesg | grep DMI:
[ 0.000000] DMI: Xen HVM domU, BIOS 4.11.4-pre 07/28/2022
(initramfs) poweroff
...

DomU / PV:
---

$ sudo xl create -c xen-5.15-vm.pv
...
[ 0.000000] Linux version 5.15.0-41-generic ...
...

Tests with 5.4 kernel: (no regressions)
=====================

Dom0 / Host:
---

 | Ubuntu GNU/Linux, with Xen 4.11.4-pre and Linux 5.4.0-122-generic |
...
Loading Xen 4.11-amd64 ...
Loading Linux 5.4.0-122-generic ...
Loading initial ramdisk ...
(XEN) Xen version 4.11.4-pre (Ubuntu 4.11.3+24-g14b62ab3e5-1ubuntu2.2) ...
...
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x3a00000
...
about to get started...
[ 0.000000] Linux version 5.4.0-122-generic ...

$ sudo xl info | head -n4
host : xen-f
release : 5.4.0-122-generic
version : #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022
machine : x86_64

DomU / HVM:
---

$ sudo xl create -c xen-5.4-vm.hvm
...
[ 0.000000] Linux version 5.4.0-122-generic ...
...
(initramfs) uname -r
5.4.0-122-generic
(initramfs) dmesg | grep DMI:
[ 0.000000] DMI: Xen HVM domU, BIOS 4.11.4-pre 07/28/2022
(initramfs) poweroff
...

$ sudo xl create -c xen-5.4-vm.pv
...
[ 0.000000] Linux version 5.4.0-122-generic ...
...

[Launchpad Janitor (janitor)]

This bug was fixed in the package xen - 4.11.3+24-g14b62ab3e5-1ubuntu2.2

---------------
xen (4.11.3+24-g14b62ab3e5-1ubuntu2.2) focal; urgency=medium

  * Fix FTBFS on armhf/arm64 due to missing <asm/unaligned.h>:
    - d/p/lp1956166-0006-fix-ftbfs-arm-lzo-unaligned.h.patch

xen (4.11.3+24-g14b62ab3e5-1ubuntu2.1) focal; urgency=medium

  * Add support for zstd compressed kernels for Dom0/DomU on x86 (LP: #1956166)
    - d/p/lp1956166-0001-introduce-unaligned.h.patch
    - d/p/lp1956166-0002-lib-introduce-xxhash.patch
    - d/p/lp1956166-0003-x86-Dom0-support-zstd-compressed-kernels.patch
    - d/p/lp1956166-0004-libxenguest-add-get_unaligned_le32.patch
    - d/p/lp1956166-0005-libxenguest-support-zstd-compressed-kernels.patch
    - d/control: add libzstd-dev as build-dep

 -- Mauricio Faria de Oliveira <email address hidden> Thu, 07 Jul 2022 13:53:37 -0300

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for xen has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Andrea C G Mennucci (am70xk)]

this bug is not solved.

Go to
https://launchpadlibrarian.net/619763840/buildlog_ubuntu-kinetic-amd64.xen_4.16.2-1_BUILDING.txt.gz

you will see the line : checking for libzstd... no

You need to add a build-dependency on libzstd-dev

regards. a.

[Mauricio Faria de Oliveira (mfo)]

Hi Andrea, thanks for spotting that; the build dep change isn't there yet. I'll fix it. Thanks!

[Luís Infante da Câmara (luis220413)]

Can anyone reopen the bug task for xen (Ubuntu), because this bug is not fixed in Kinetic?

[Andrea C G Mennucci (am70xk)]

thanks to you.

 I compiled all packages (adding libzstd-dev to the system ) using the sources https://salsa.debian.org/xen-team/debian-xen.git , and installed them, and this solves the problem. I use Ubuntu 22.04

 I wanted to open a new bug report, but I cannot: `ubuntu-bug` complains that the package is not the original package... and indeed they are not.

 Glad that you spotted my comment.

Hope that someone will fix this in Debian as well.

[Andrea C G Mennucci (am70xk)]

I add a comment.

The bug I experienced had a different effect than what reported above.

 The current XEN in Ubuntu 22.04 can boot a modern linux kernel (such as 5.15.0-46-generic) even w/o the ZSTD support. I believe that, at boot time, the kernel does self-decompress (that's what it has to do anyway, even when not using XEN...). So Dom0 was running fine.

What was happening to me was that afterwards it could not start containers: `xl create` would fail with the error «xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD decompress support unavailable»

The error disappeared after ZSTD decompressione was available in the XEN tools.

[HappyCamper (arno-cs)]

Hi

I still have this problem on Ubuntu 22.04.1 when creating a 22.04 guest

# xl create /etc/xen/stp0.cfg
Parsing config from /etc/xen/stp0.cfg
xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD decompress support unavailable
: Internal error
xc: error: panic: xg_dom_bzimageloader.c:772: xc_dom_probe_bzimage_kernel unable to ZSTD decompress kernel: Invalid kernel
xc: error: panic: xg_dom_core.c:690: xc_dom_find_loader: no loader found: Invalid kernel
libxl: error: libxl_dom.c:571:libxl__build_dom: xc_dom_parse_image failed
libxl: error: libxl_create.c:1611:domcreate_rebuild_done: Domain 1:cannot (re-)build domain: -3
libxl: error: libxl_domain.c:1183:libxl__destroy_domid: Domain 1:Non-existant domain
libxl: error: libxl_domain.c:1137:domain_destroy_callback: Domain 1:Unable to destroy guest
libxl: error: libxl_domain.c:1064:domain_destroy_cb: Domain 1:Destruction of domain failed

The host is running Xen 4.16 on 22.04.1:

root@host:~# dpkg -l | grep xen-hypervisor
ii xen-hypervisor-4.16-amd64 4.16.0-1~ubuntu2 amd64 Xen Hypervisor on AMD64
ii xen-hypervisor-common 4.16.0-1~ubuntu2 all Xen Hypervisor - common files
root@host:~# cat /etc/issue
Ubuntu 22.04.1 LTS \n \l

The guest has a 5.15 kernel:

# mount -o ro /dev/VolumeGroupXen/stp0-disk /mnt
# ls /mnt/boot/vmlinuz-5.15.0-48-generic
/mnt/boot/vmlinuz-5.15.0-48-generic

The stp0.cfg file boils down to:
bootloader = 'pygrub'
vcpus = '1'
memory = '1024'
type = 'pvh'
root = '/dev/xvda1 ro'
disk = [
                  'phy:/dev/VolumeGroupXen/stp0-disk,xvda1,w',
              ]
name = 'stp0'
dhcp = 'dhcp'
vif = [ 'mac=00:16:3E:EF:F3:BC,bridge=xenbr0' ]
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'

[Mauricio Faria de Oliveira (mfo)]

Hi @arno-cs,

Could you please test with the Xen packages from this PPA?
https://launchpad.net/~mfo/+archive/ubuntu/build

Thanks!

[Klaus Darilion (klausdarilion)]

I just tested your xen 4.16.0-1~ubuntu2.1 packages from https://launchpad.net/~mfo/+archive/ubuntu/build und confirm: With this packages now Ubuntu 22.04 can boot 22.04 PV-domUs using pygrub.

[HappyCamper (arno-cs)]

Hi Mauricio

I missed your reply, sorry. My colleague tested with your PPA and it works. Thanks and hope to see this in Ubuntu 22.04 soon!

       Arno

[Mauricio Faria de Oliveira (mfo)]

Thanks for testing, Klaus and Arno!

Let's use bug 1998026 for the libzstd-dev change on later releases
(after Focal).

Keeping this bug for the Dom0-only changes across releases,
and for DomU PV on Focal only. Added note in bug description.

(The reason is this bug has already been referenced in the Jammy
changelog, that has been Fix Released; if I undertand correctly,
for consistency/SRU machinery reasons, these can't be reutilized.)

[Mauricio Faria de Oliveira (mfo)]

The xen packages in jammy/kinetic-proposed
should have userspace zstd support enabled.
(bug 1998026).

P.S.: marking Ubuntu/devel as Fix Released.

[Mauricio Faria de Oliveira (mfo)]

The xen packages in jammy/kinetic-updates
now have userspace zstd support enabled
(bug 1998026).

xen (4.16.2-1ubuntu1) kinetic
xen (4.16.0-1~ubuntu2.1) jammy

Thanks, everyone!