Ubuntu
xen package

Ubuntu 22.04 doesn't boot with xen (DomU pv/pvh/pygrub)

Bug #1998026 reported by Mauricio Faria de Oliveira on 2022-11-26

This bug affects 1 person

	Status	Importance	Assigned to
xen (Ubuntu)	Fix Released	Undecided	Unassigned
Jammy	Fix Released	Medium	Mauricio Faria de Oliveira
Kinetic	Fix Released	Medium	Mauricio Faria de Oliveira

Bug Description

[Impact]

* Bug 1956166 (Ubuntu 22.04 doesn't boot with xen)
introduced a libzstd-dev build-dep on Focal but
not on later releases (oversight).

* This allows Focal to boot zstd kernels on Xen
   as Dom0 and DomU HVM & PV, but later releases
   cannot boot DomU PV modes as the userspace &
   tools code doesn't have zstd support enabled.

[Fix]

* The build-dep on libzstd-dev is applied in Debian
and Lunar, and still needed in Kinetic and Focal.

[Test Steps]

* Install and boot Xen as Dom0, and create/launch
a DomU PV domain with a zstd kernel.

  $ sudo apt install xen-system-amd64
  $ sudo reboot
  $ cat <<EOF >xen-5.19-vm.pv
  name = "xen-5.19-vm.pv"
  type = "pv"
  kernel = "/boot/vmlinuz-5.19.0-23-generic"
  ramdisk = "/boot/initrd.img-5.19.0-23-generic"
  cmdline = "console=ttyS0 break=top"
  vcpus = 1
  memory = 1024
  EOF

  $ sudo xl create -c xen-5.19-vm.pv
  Parsing config from xen-5.19-vm.pv
  xc: error: panic: xg_dom_bzimageloader.c:619:
  xc_try_zstd_decode: ZSTD decompress support unavailable

  (with fix)
  $ sudo xl create -c xen-5.19-vm.pv
  Parsing config from xen-5.19-vm.pv
  [ 0.000000] Linux version 5.19.0-23-generic ...
  ...

[Regression Potential]

* This enables code paths for zstd support,
which is used when zstd kernels are found,
thus isn't expected to break other formats.

* However, theoretically, regressions would
   probably manifest immediately on the boot
   on DomU PV domains of non-zstd images, as
   their kernel images would fail to load
   (this is fortunately easy to verify/test).

* This change has been available in Focal
since August/2022 with no bugs reported.

[Other Info]

* Fix available in Debian and Ubuntu Lunar.

$ git remote get-url origin
https://salsa.debian.org/xen-team/debian-xen.git

$ git show --oneline -s 8b790c177e1dd
8b790c177e1d debian/control: Add libzstd-dev as Build-Depends

$ git describe --contains 8b790c177e1dd
archive/debian/4.16.2-2~28

Tags:

Mauricio Faria de Oliveira (mfo) on 2022-11-26

Changed in xen (Ubuntu):
status:	New → Fix Released

Mauricio Faria de Oliveira (mfo) on 2022-11-26

Changed in xen (Ubuntu Jammy):
status:	New → In Progress
importance:	Undecided → Medium
assignee:	nobody → Mauricio Faria de Oliveira (mfo)
Changed in xen (Ubuntu Kinetic):
status:	New → In Progress
importance:	Undecided → Medium
assignee:	nobody → Mauricio Faria de Oliveira (mfo)

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2022-11-26:

Download full text (3.4 KiB)

Test Steps (details in bug 1956166 comment 14)

Looks good for std and non-zstd kernels on DomU PV mode;
still boots correctly with Dom0 mode; Kinetic and Jammy.

Uploading to K and J.

Kinetic:
---

zstd kernel:

cat <<EOF >xen-5.19-vm.pv
name = "xen-5.19-vm.pv"
type = "pv"
kernel = "/boot/vmlinuz-5.19.0-23-generic"
cmdline = "console=ttyS0"
vcpus = 1
memory = 1024
EOF

- before:

$ sudo xl create -c xen-5.19-vm.pv
Parsing config from xen-5.19-vm.pv
xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD decompress support unavailable
...

- after:

sudo add-apt-repository ppa:mfo/lp1998026
sudo apt install $(awk '/^Package:/ {pkg=$2} /^Source: xen$/ {print pkg}' /var/lib/dpkg/status)

$ sudo xl create -c xen-5.19-vm.pv
Parsing config from xen-5.19-vm.pv
[ 0.000000] Linux version 5.19.0-23-generic ...
...

non-zstd kernel:

wget https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/24655136/+files/linux-image-unsigned-5.4.0-133-generic_5.4.0-133.149_amd64.deb
dpkg-deb -x linux-image-unsigned-5.4.0-133-generic_5.4.0-133.149_amd64.deb kernel-5.4-deb

cat <<EOF >xen-5.4-vm.pv
name = "xen-5.4-vm.pv"
type = "pv"
kernel = "./kernel-5.4-deb/boot/vmlinuz-5.4.0-133-generic"
cmdline = "console=ttyS0"
vcpus = 1
memory = 1024
EOF

$ sudo xl create -c xen-5.4-vm.pv
Parsing config from xen-5.4-vm.pv
[ 0.000000] Linux version 5.4.0-133-generic ...
...

Dom0:

Loading Xen 4.16-amd64 ...
Loading Linux 5.19.0-23-generic ...
Loading initial ramdisk ...
(XEN) Xen version 4.16.2 (Ubuntu 4.16.2-1ubuntu1)
...

$ sudo xl info | head -n4
host : xen-zstd-k
release : 5.19.0-23-generic
version : #24-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct 14 15:39:57 UTC 2022
machine : x86_64

Jammy:
---

zstd kernel:

cat <<EOF >xen-5.15-vm.pv
name = "xen-5.15-vm.pv"
type = "pv"
kernel = "/boot/vmlinuz-5.15.0-53-generic"
cmdline = "console=ttyS0"
vcpus = 1
memory = 1024
EOF

- before:

$ sudo xl create -c xen-5.15-vm.pv
Parsing config from xen-5.15-vm.pv
xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD decompress support unavailable
...

- after:

sudo add-apt-repository ppa:mfo/lp1998026
sudo apt install $(awk '/^Package:/ {pkg=$2} /^Source: xen$/ {print pkg}' /var/lib/dpkg/status)

$ sudo xl create -c xen-5.15-vm.pv
Parsing config from xen-5.15-vm.pv
[ 0.000000] Linux version 5.15.0-53-generic ...
...

non-zstd kernel:

cat <<EOF >xen-5.4-vm.pv
name = "xen-5.4-vm.pv"
type = "pv"
kernel = "./kernel-5.4-deb/boot/vmlinuz-5.4.0-133-generic"
cmdline = "console=ttyS0"
vcpus = 1
memory = 1024
EOF

$ sudo xl create -c xen-5.4-vm.pv
Parsing config from xen-5.4-vm.pv
[ 0.000000] Linux version 5.4.0-133-generic ...
...

Dom0:

Loading Xen 4.16-amd64 ...
Loading Linux 5.15.0-53-generic ...
Loading initial ramdisk ...
(XEN) Xen version 4.1...

Test Steps (details in bug 1956166 comment 14)

Looks good for std and non-zstd kernels on DomU PV mode;
still boots correctly with Dom0 mode; Kinetic and Jammy.

Uploading to K and J.

Kinetic:
---

zstd kernel:

cat <<EOF >xen-5.19-vm.pv
	name = "xen-5.19-vm.pv"
	type = "pv"
	kernel = "/boot/vmlinuz-5.19.0-23-generic"
	cmdline = "console=ttyS0"
	vcpus = 1
	memory = 1024
	EOF

- before:

$ sudo xl create -c xen-5.19-vm.pv
	Parsing config from xen-5.19-vm.pv
	xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD decompress support unavailable
	...

- after:

sudo add-apt-repository ppa:mfo/lp1998026
	sudo apt install $(awk '/^Package:/ {pkg=$2} /^Source: xen$/ {print pkg}' /var/lib/dpkg/status)

$ sudo xl create -c xen-5.19-vm.pv
	Parsing config from xen-5.19-vm.pv
	[    0.000000] Linux version 5.19.0-23-generic ...
	...

non-zstd kernel:

wget https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/24655136/+files/linux-image-unsigned-5.4.0-133-generic_5.4.0-133.149_amd64.deb
	dpkg-deb -x linux-image-unsigned-5.4.0-133-generic_5.4.0-133.149_amd64.deb kernel-5.4-deb

cat <<EOF >xen-5.4-vm.pv
	name = "xen-5.4-vm.pv"
	type = "pv"
	kernel = "./kernel-5.4-deb/boot/vmlinuz-5.4.0-133-generic"
	cmdline = "console=ttyS0"
	vcpus = 1
	memory = 1024
	EOF

$ sudo xl create -c xen-5.4-vm.pv
	Parsing config from xen-5.4-vm.pv
	[    0.000000] Linux version 5.4.0-133-generic ...
	...

Dom0:

Loading Xen 4.16-amd64 ...
	Loading Linux 5.19.0-23-generic ...
	Loading initial ramdisk ...
	(XEN) Xen version 4.16.2 (Ubuntu 4.16.2-1ubuntu1)
	...

$ sudo xl info | head -n4
	host                   : xen-zstd-k
	release                : 5.19.0-23-generic
	version                : #24-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct 14 15:39:57 UTC 2022
	machine                : x86_64

Jammy:
---

zstd kernel:

cat <<EOF >xen-5.15-vm.pv
	name = "xen-5.15-vm.pv"
	type = "pv"
	kernel = "/boot/vmlinuz-5.15.0-53-generic"
	cmdline = "console=ttyS0"
	vcpus = 1
	memory = 1024
	EOF

- before:

$ sudo xl create -c xen-5.15-vm.pv
	Parsing config from xen-5.15-vm.pv
	xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD decompress support unavailable
	...
	
- after:

sudo add-apt-repository ppa:mfo/lp1998026
	sudo apt install $(awk '/^Package:/ {pkg=$2} /^Source: xen$/ {print pkg}' /var/lib/dpkg/status)

$ sudo xl create -c xen-5.15-vm.pv
	Parsing config from xen-5.15-vm.pv
	[    0.000000] Linux version 5.15.0-53-generic ...
	...

non-zstd kernel:
  
	wget https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/24655136/+files/linux-image-unsigned-5.4.0-133-generic_5.4.0-133.149_amd64.deb
	dpkg-deb -x linux-image-unsigned-5.4.0-133-generic_5.4.0-133.149_amd64.deb kernel-5.4-deb

cat <<EOF >xen-5.4-vm.pv
	name = "xen-5.4-vm.pv"
	type = "pv"
	kernel = "./kernel-5.4-deb/boot/vmlinuz-5.4.0-133-generic"
	cmdline = "console=ttyS0"
	vcpus = 1
	memory = 1024
	EOF

$ sudo xl create -c xen-5.4-vm.pv
	Parsing config from xen-5.4-vm.pv
	[    0.000000] Linux version 5.4.0-133-generic ...
	...
	
  Dom0:
  
	Loading Xen 4.16-amd64 ...
	Loading Linux 5.15.0-53-generic ...
	Loading initial ramdisk ...
	(XEN) Xen version 4.16.0 (Ubuntu 4.16.0-1~ubuntu2.1) ...
	...

$ sudo xl info | head -n4
	host                   : xen-zstd-j
	release                : 5.15.0-53-generic
	version                : #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022
	machine                : x86_64

summary:

- Ubuntu 22.04 doesn't boot with xen (pv/pvh/pygrub)
+ Ubuntu 22.04 doesn't boot with xen (DomU pv/pvh/pygrub)

Revision history for this message

Łukasz Zemczak (sil2100) wrote on 2022-12-02: Please test proposed package

Hello Mauricio, or anyone else affected,

Accepted xen into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/xen/4.16.2-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in xen (Ubuntu Kinetic):
status:	In Progress → Fix Committed
tags:	added: verification-needed verification-needed-kinetic
Changed in xen (Ubuntu Jammy):
status:	In Progress → Fix Committed
tags:	added: verification-needed-jammy

Revision history for this message

Łukasz Zemczak (sil2100) wrote on 2022-12-02:

Hello Mauricio, or anyone else affected,

Accepted xen into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/xen/4.16.0-1~ubuntu2.1 in a few hours, and then in the -proposed repository.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2022-12-05:

Verification done on kinetic.

...

$ lsb_release -cs
kinetic

$ sudo add-apt-repository -p proposed
$ sudo apt install xen-system-amd64 linux-modules-extra-$(uname -r)

$ dpkg -s xen-system-amd64 | grep Version:
Version: 4.16.2-1ubuntu1

# edit VM XML (don't use virtio at the moment)
...

Dom0:
----

|*Ubuntu GNU/Linux, with Xen hypervisor
Loading Xen 4.16-amd64 ...
Loading Linux 5.19.0-26-generic ...
Loading initial ramdisk ...
(XEN) Xen version 4.16.2 (Ubuntu 4.16.2-1ubuntu1) ...
...

$ sudo xl info | head -n4
host : xen-zstd-kinetic
release : 5.19.0-26-generic
version : #27-Ubuntu SMP PREEMPT_DYNAMIC Wed Nov 23 20:44:15 UTC 2022
machine : x86_64

DomU PV:
-------

zstd kernel:

$ cat <<EOF >xen-5.19-vm.pv
name = "xen-5.19-vm.pv"
type = "pv"
kernel = "/boot/vmlinuz-5.19.0-26-generic"
cmdline = "console=ttyS0"
vcpus = 1
memory = 1024
EOF

$ sudo xl create -c xen-5.19-vm.pv
Parsing config from xen-5.19-vm.pv
[ 0.000000] Linux version 5.19.0-26-generic ...
...

non-zstd kernel:

$ wget https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/24655136/+files/linux-image-unsigned-5.4.0-133-generic_5.4.0-133.149_amd64.deb
$ dpkg-deb -x linux-image-unsigned-5.4.0-133-generic_5.4.0-133.149_amd64.deb kernel-5.4-deb

$ cat <<EOF >xen-5.4-vm.pv
name = "xen-5.4-vm.pv"
type = "pv"
kernel = "./kernel-5.4-deb/boot/vmlinuz-5.4.0-133-generic"
cmdline = "console=ttyS0"
vcpus = 1
memory = 1024
EOF

$ sudo xl create -c xen-5.4-vm.pv
Parsing config from xen-5.4-vm.pv
[ 0.000000] Linux version 5.4.0-133-generic ...
...

tags:

added: verification-done-kinetic
removed: verification-needed-kinetic

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2022-12-05:

Verification done on jammy.

...

$ lsb_release -cs
jammy

$ sudo add-apt-repository -p proposed
$ sudo apt install xen-system-amd64

$ dpkg -s xen-system-amd64 | grep Version:
Version: 4.16.0-1~ubuntu2.1

...

Dom0:
----

|*Ubuntu GNU/Linux, with Xen hypervisor
Loading Xen 4.16-amd64 ...
Loading Linux 5.15.0-56-generic ...
Loading initial ramdisk ...
(XEN) Xen version 4.16.0 (Ubuntu 4.16.0-1~ubuntu2.1)
...

$ sudo xl info | head -n4
host : xen-zstd-jammy
release : 5.15.0-56-generic
version : #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022
machine : x86_64

DomU PV:
-------

zstd kernel:

$ cat <<EOF >xen-5.15-vm.pv
name = "xen-5.15-vm.pv"
type = "pv"
kernel = "/boot/vmlinuz-5.15.0-56-generic"
cmdline = "console=ttyS0"
vcpus = 1
memory = 1024
EOF

$ sudo xl create -c xen-5.15-vm.pv
Parsing config from xen-5.15-vm.pv
[ 0.000000] Linux version 5.15.0-56-generic ...
...

non-zstd kernel:

$ cat <<EOF >xen-5.4-vm.pv
name = "xen-5.4-vm.pv"
type = "pv"
kernel = "./kernel-5.4-deb/boot/vmlinuz-5.4.0-133-generic"
cmdline = "console=ttyS0"
vcpus = 1
memory = 1024
EOF

$ sudo xl create -c xen-5.4-vm.pv
Parsing config from xen-5.4-vm.pv
[ 0.000000] Linux version 5.4.0-133-generic ...
...

tags:	added: verification-done-jammy removed: verification-needed-jammy
tags:	added: verification-done removed: verification-needed

Revision history for this message

Launchpad Janitor (janitor) wrote on 2022-12-13:

This bug was fixed in the package xen - 4.16.2-1ubuntu1

---------------
xen (4.16.2-1ubuntu1) kinetic; urgency=medium

* debian/control: Add libzstd-dev as Build-Depends (LP: #1998026)
* debian/control: update-maintainer

-- Mauricio Faria de Oliveira <email address hidden> Sat, 26 Nov 2022 17:38:33 -0300

Changed in xen (Ubuntu Kinetic):
status:	Fix Committed → Fix Released

Revision history for this message

Brian Murray (brian-murray) wrote on 2022-12-13: Update Released

The verification of the Stable Release Update for xen has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2022-12-13:

This bug was fixed in the package xen - 4.16.0-1~ubuntu2.1

---------------
xen (4.16.0-1~ubuntu2.1) jammy; urgency=medium

* debian/control: Add libzstd-dev as Build-Depends (LP: #1998026)

-- Mauricio Faria de Oliveira <email address hidden> Sat, 26 Nov 2022 19:06:42 -0300

Changed in xen (Ubuntu Jammy):
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntuxen package

Ubuntu 22.04 doesn't boot with xen (DomU pv/pvh/pygrub)

Bug Description

Other bug subscribers

Remote bug watches

Ubuntu
xen package