Ubuntu 22.04 boot stuck in initramfs, when installed with zfs encryption

[JaroAnto]

Hi,

I just installed the latest Ubuntu desktop from iso file ubuntu-21.10-desktop-amd64.iso inside of VMWare workstation. I chose ZFS and native ZFS encryption of the filesystem. The installation went fine. Everything worked as expected.

Then I upgraded the packages to the latest version via the Software updater. After reboot I'm stuck in the initramfs prompt. The following command fails:

mount -o zfsutil -t zfs rpool/ROOT/ubuntu_gtw63h /root//

Permission denied.

And the system never asks me for the password to unlock the root fs.

So, I'm guessing that there is something wrong with the new initramfs disk: initrd.img-5.13.0-27-generic

When I reboot and select the previous version in grub: initrd.img-5.13.0-27-generic, the system asks for the password and boots without a problem.

Thanks.

To summarize:

1. Installed new VM using the Ubuntu iso image. Chose ZFS native encryption. Minimal install.
2. As soon as the system came up, I hit the update/upgrade prompt. Rebooted and failed to boot the new version.

I didn't customize anything or installed anything extra.

root@ubud01:/var# lsb_release -rd
Description: Ubuntu 21.10
Release: 21.10

root@ubud01:/var# dpkg -l | grep zfs
ii libzfs4linux 2.0.6-1ubuntu2 amd64 OpenZFS filesystem library for Linux
ii zfs-initramfs 2.0.6-1ubuntu2 amd64 OpenZFS root filesystem capabilities for Linux - initramfs
ii zfs-zed 2.0.6-1ubuntu2 amd64 OpenZFS Event Daemon
ii zfsutils-linux 2.0.6-1ubuntu2 amd64 command-line tools to manage OpenZFS filesystems

root@ubud01:/var# dpkg -l | grep init
ii busybox-initramfs 1:1.30.1-6ubuntu3.1 amd64 Standalone shell setup for initramfs
ii cryptsetup-initramfs 2:2.3.6-0ubuntu1 all disk encryption support - initramfs integration
ii gnome-initial-setup 40.4-1ubuntu1 amd64 Initial GNOME system setup helper
ii init 1.60build1 amd64 metapackage ensuring an init system is installed
ii init-system-helpers 1.60build1 all helper tools for all init systems
ii initramfs-tools 0.140ubuntu6 all generic modular initramfs generator (automation)
ii initramfs-tools-bin 0.140ubuntu6 amd64 binaries used by initramfs-tools
ii initramfs-tools-core 0.140ubuntu6 all generic modular initramfs generator (core tools)
ii libatopology2:amd64 1.2.4-1.1ubuntu3.1 amd64 shared library for handling ALSA topology definitions
ii libklibc:amd64 2.0.8-6.1ubuntu2 amd64 minimal libc subset for use with initramfs
ii lsb-base 11.1.0ubuntu3 all Linux Standard Base init script functionality
ii ncurses-base 6.2+20201114-2build1 all basic terminal type definitions
ii sysvinit-utils 2.96-7ubuntu1 amd64 System-V-like utilities
ii xinit 1.4.1-0ubuntu3 amd64 X server initialisation tool
ii zfs-initramfs 2.0.6-1ubuntu2 amd64 OpenZFS root filesystem capabilities for Linux - initramfs

root@ubud01:/var# zfs list
NAME USED AVAIL REFER MOUNTPOINT
bpool 290M 542M 96K /boot
bpool/BOOT 289M 542M 96K none
bpool/BOOT/ubuntu_gtw63h 288M 542M 156M /boot
rpool 6.39G 10.6G 192K /
rpool/ROOT 5.64G 10.6G 192K none
rpool/ROOT/ubuntu_gtw63h 5.64G 10.6G 3.45G /
rpool/ROOT/ubuntu_gtw63h/srv 192K 10.6G 192K /srv
rpool/ROOT/ubuntu_gtw63h/usr 660K 10.6G 192K /usr
rpool/ROOT/ubuntu_gtw63h/usr/local 468K 10.6G 388K /usr/local
rpool/ROOT/ubuntu_gtw63h/var 1.53G 10.6G 192K /var
rpool/ROOT/ubuntu_gtw63h/var/games 272K 10.6G 192K /var/games
rpool/ROOT/ubuntu_gtw63h/var/lib 1.52G 10.6G 1.34G /var/lib
rpool/ROOT/ubuntu_gtw63h/var/lib/AccountsService 328K 10.6G 216K /var/lib/AccountsService
rpool/ROOT/ubuntu_gtw63h/var/lib/NetworkManager 404K 10.6G 228K /var/lib/NetworkManager
rpool/ROOT/ubuntu_gtw63h/var/lib/apt 88.1M 10.6G 82.1M /var/lib/apt
rpool/ROOT/ubuntu_gtw63h/var/lib/dpkg 61.7M 10.6G 48.0M /var/lib/dpkg
rpool/ROOT/ubuntu_gtw63h/var/log 4.84M 10.6G 3.57M /var/log
rpool/ROOT/ubuntu_gtw63h/var/mail 272K 10.6G 192K /var/mail
rpool/ROOT/ubuntu_gtw63h/var/snap 2.14M 10.6G 1.72M /var/snap
rpool/ROOT/ubuntu_gtw63h/var/spool 388K 10.6G 252K /var/spool
rpool/ROOT/ubuntu_gtw63h/var/www 272K 10.6G 192K /var/www
rpool/USERDATA 246M 10.6G 192K /
rpool/USERDATA/ja_1ocgwe 245M 10.6G 221M /home/ja
rpool/USERDATA/root_1ocgwe 424K 10.6G 272K /root
rpool/keystore 518M 11.0G 48.1M -
root@ubud01:/var# zfs get encryption rpool/ROOT/ubuntu_gtw63h
NAME PROPERTY VALUE SOURCE
rpool/ROOT/ubuntu_gtw63h encryption aes-256-gcm -

[JaroAnto (random-string)]

The original (working) disk is config-5.13.0-19-generic.

[JaroAnto (random-string)]

Heh, sorry, copy/paste issues: initrd.img-5.13.0-19-generic

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1959085/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[JaroAnto (random-string)]

I installed the podman package through apt. That triggered rebuild of the initramfs disk. And that surprisingly all fixed the boot problem.

[JaroAnto (random-string)]

I'm guessing that this might be the right package if you want to investigate this further.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Langasek (vorlon)]

Dimitri, can you take a look at this?

[Alan Baghumian (alanbach)]

I just found this, I encountered this exact same issue with Jammy Desktop installer last night and filed a bug here : #1960974

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alan Baghumian (alanbach)]

I confirmed this is happening. Here are two ways to reproduce:

1) Grabbed jammy-desktop-amd64.iso nightly build (Feb 2). Installing a system using this ISO, while choosing Encryption + ZFS lands you on this error.

2) Installed Impish using an ISO from October 12, choosing Encryption + ZFS. The system installs and boots up w/o any issues. Performed a dist-upgrade to Jammy, which resulted the system booting into initramfs with the exact same error.

[Dimitri John Ledkov (xnox)]

This is fixed release in pending images; which are failing installation due to other bugs.
Once those bugs are resolved, and a new image is promoted, it shouldn't experience this zfs issue.
So we are blocked on getting Ubuntu Desktop ISO passing the daily automated testing and getting promoted.

I will separately double check that the upgrade path works correctly.

[Dimitri John Ledkov (xnox)]

Ubuntu 22.04 daily images testing issues have been resolved and a new image promoted, jammy installs are now working correctly.

[Dimitri John Ledkov (xnox)]

On 21.10, zfs-linux packages 2.0.6-1ubuntu2 generate incorrect boot entries for snapshoted systems, please upgrade to 2.0.6-1ubuntu2.1. Snapshots created with the broken old version are not bootable.

[Alan Baghumian (alanbach)]

Thank you all for investigating and fixing this.