ubuntu 20.04.1 installed with zfs root fails to boot after installing xen-hypervisor

[Todd]

After installing 20.04.1 using the ZFS root option I installed Xen. When it rebooted it failed in initramfs with the error "No pool imported". I was able to boot the previous kernel. When I examined the /boot/grub/grub.cfg I found what looked like invalid "root=ZFS=" options like this:

"root=ZFS=/ROOT/ubuntu_xxxxxx"

The previous kernel that boots looks like this:

"root=ZFS=rpool/ROOT/ubuntu_xxxxxx"

After looking in /etc/default/grub and /etc/grub.d I was unable to figure out why they were different and how I could change it.

As a test saved a copy and edited /boot/grub/grub.cfg adding the rpool to each "root=ZFS=" line where rpool was missing. This time I was able to boot the Xen kernel.

==============================
How to reproduce the problem

From scratch install ubuntu 20.04.1 desktop, select ZFS as root option, then install xen-hypervisor-4.11-amd64.
It will fail to reboot because the grub root=ZFS argument is wrong.

[Todd (codesleuth7)]

It looks like this is a problem with "grub-mkconfig".

[Todd (codesleuth7)]

This is looking like a problem with the grub package.

[Todd (codesleuth7)]

The xen postinst script is using the update-grub command, which runs grub-mkconfig. Running grub-mkconfig produced this output:
root@corona:~# grub-mkconfig -o /boot/grub/grub.cfg
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Sourcing file `/etc/default/grub.d/xen.cfg'
Including Xen overrides from /etc/default/grub.d/xen.cfg
WARNING: GRUB_DEFAULT changed to boot into Xen by default!
         Edit /etc/default/grub.d/xen.cfg to avoid this warning.
Generating grub configuration file ...
Found linux image: vmlinuz-5.4.0-52-generic in rpool/ROOT/ubuntu_69vvbr
Found initrd image: initrd.img-5.4.0-52-generic in rpool/ROOT/ubuntu_69vvbr
Found linux image: vmlinuz-5.4.0-51-generic in rpool/ROOT/ubuntu_69vvbr
Found initrd image: initrd.img-5.4.0-51-generic in rpool/ROOT/ubuntu_69vvbr
Found linux image: vmlinuz-5.4.0-42-generic in rpool/ROOT/ubuntu_69vvbr
Found initrd image: initrd.img-5.4.0-42-generic in rpool/ROOT/ubuntu_69vvbr
Found linux image: vmlinuz-5.4.0-47-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_4k3sjn
Found initrd image: initrd.img-5.4.0-47-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_4k3sjn
Found linux image: vmlinuz-5.4.0-45-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_4k3sjn
Found initrd image: initrd.img-5.4.0-45-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_4k3sjn
Found linux image: vmlinuz-5.4.0-42-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_4k3sjn
Found initrd image: initrd.img-5.4.0-42-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_4k3sjn
Found linux image: vmlinuz-5.4.0-47-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_7ur3n5
Found initrd image: initrd.img-5.4.0-47-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_7ur3n5
Found linux image: vmlinuz-5.4.0-45-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_7ur3n5
Found initrd image: initrd.img-5.4.0-45-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_7ur3n5
Found linux image: vmlinuz-5.4.0-42-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_7ur3n5
Found initrd image: initrd.img-5.4.0-42-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_7ur3n5
Found linux image: vmlinuz-5.4.0-48-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_l6bt3l
Found initrd image: initrd.img-5.4.0-48-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_l6bt3l
Found linux image: vmlinuz-5.4.0-47-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_l6bt3l
Found initrd image: initrd.img-5.4.0-47-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_l6bt3l
Found linux image: vmlinuz-5.4.0-45-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_l6bt3l
Found initrd image: initrd.img-5.4.0-45-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_l6bt3l
Found linux image: vmlinuz-5.4.0-42-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_l6bt3l
Found initrd image: initrd.img-5.4.0-42-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_l6bt3l
Found linux image: vmlinuz-5.4.0-48-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_3qeu07
Found initrd image: initrd.img-5.4.0-48-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_3qeu07
Found linux image: vmlinuz-5.4.0-47-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_3qeu07
Found initrd image: initrd.img-5.4.0-47-generic in rpool/ROOT/ubuntu_69vvbr@autozsys_3qeu07
Found linux image: vmlinuz-5.4.0-42-generic in rpool/ROOT/ubuntu_6...

[Todd (codesleuth7)]

This is my first bug report using bugs.launchpad.net and I did not know I could move the bug report to a different package. I have created a duplicate report, and don't see a way to close one of them. It was under the Xen package, but with more investigation it looks like it belongs in the grub2 package.

[Todd (codesleuth7)]

I have now seen this problem without xen-hypervisor installed. With a ZFS root it also happened after using grub-customizer, and after a kernel update.

[Todd (codesleuth7)]

I have two different Ubuntu 20.04.1 systems using ZFS on root. Only one has Xen installed, but both have had this same problem with rpool missing from the grub.cfg and fail to boot after update-grub/grub-mkconfig is run. Both systems share an error during /etc/grub.d/30_os-prober:

----- System one ----
### BEGIN /etc/grub.d/30_os-prober ###
device-mapper: reload ioctl on osprober-linux-nvme0n1p4 failed: Device or resource busy
Command failed.
### END /etc/grub.d/30_os-prober ###

----- System two ----
### BEGIN /etc/grub.d/30_os-prober ###
device-mapper: reload ioctl on osprober-linux-sda7 failed: Device or resource busy
Command failed.
### END /etc/grub.d/30_os-prober ###

I have uploaded the full log.

[Todd (codesleuth7)]

In line 137 of file "10_linux":

rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`

The command run returns this error:
root@stream:/etc/grub.d# /usr/sbin/grub-probe --device /dev/nvme0n1p4 --target=fs_label
/usr/sbin/grub-probe: error: unknown filesystem.

The result makes ${rpool} NULL, and sets ${LINUX_ROOT_DEVICE} to "ZFS=/ROOT/ubuntu_aaodxz"

This is clearly wrong and looks like the root cause of this issue.

[Todd (codesleuth7)]

The best solution may involve fixing grub-probe so it knows a ZFS filesystem.
The attached patch is a simple workaround that worked on my 3 test systems.

For example if the patch is downloaded to /tmp/grub2.patch

You can install the patch like this:

cd /etc/grub.d; sudo patch < /tmp/grub2.patch

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Test fix to work around grub-probe failure" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]