Upgrade with zfs-on-root failures

[Seth Arnold]

I followed rlaager's howto for installing Ubuntu 18.04 LTS with ZFS root (but used disco). I've also set up sanoid to perform periodic snapshots of bpool (which stores /boot).

The upgrade from disco to eoan didn't go well:

[.. trimmed to fit in the comment box; more context in attachment ..]

/ubuntu@autosnap_2020-01-24_04:00:05_frequently
Found initrd image: initrd.img-5.3.0-29-generic in rpool/ROOT/ubuntu@autosnap_2020-01-24_04:00:05_frequently
Found linux image: vmlinuz-5.0.0-39-generic in rpool/ROOT/ubuntu@autosnap_2020-01-24_04:15:05_frequently
Found initrd image: initrd.img-5.0.0-39-generic in rpool/ROOT/ubuntu@autosnap_2020-01-24_04:15:05_frequently
Found linux image: vmlinuz-5.0.0-40-generic in rpool/ROOT/ubuntu@autosnap_2020-01-24_04:15:05_frequently
Found initrd image: initrd.img-5.0.0-40-generic in rpool/ROOT/ubuntu@autosnap_2020-01-24_04:15:05_frequently
Found linux image: vmlinuz-5.3.0-29-generic in rpool/ROOT/ubuntu@autosnap_2020-01-24_04:15:05_frequently
Found initrd image: initrd.img-5.3.0-29-generic in rpool/ROOT/ubuntu@autosnap_2020-01-24_04:15:05_frequently
Adding boot menu entry for EFI firmware configuration
done
Processing triggers for dbus (1.12.14-1ubuntu2) ...
Processing triggers for initramfs-tools (0.133ubuntu10) ...
update-initramfs: Generating /boot/initrd.img-5.3.0-29-generic
cryptsetup: ERROR: Couldn't resolve device rpool/ROOT/ubuntu
cryptsetup: WARNING: Couldn't determine root device
Error 24 : Write error : cannot write compressed block
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.3.0-29-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Processing triggers for shim-signed (1.39+15+1533136590.3beb971-0ubuntu1) ...
Nothing to do.
Processing triggers for libgdk-pixbuf2.0-0:amd64 (2.40.0+dfsg-1build1) ...
Errors were encountered while processing:
 initramfs-tools
Exception during pm.DoInstall(): E:Sub-process /usr/bin/dpkg returned an error code (1)

Could not install the upgrades

The upgrade has aborted. Your system could be in an unusable state. A
recovery will run now (dpkg --configure -a).

Setting up initramfs-tools (0.133ubuntu10) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools (0.133ubuntu10) ...
update-initramfs: Generating /boot/initrd.img-5.3.0-29-generic
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/initramfs-tools.0.crash'
cryptsetup: ERROR: Couldn't resolve device rpool/ROOT/ubuntu
cryptsetup: WARNING: Couldn't determine root device
Error 24 : Write error : cannot write compressed block
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.3.0-29-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 initramfs-tools

Upgrade complete

The upgrade has completed but there were errors during the upgrade
process.

To continue please press [ENTER]

ProblemType: Bug
DistroRelease: Ubuntu 19.10
Package: ubuntu-release-upgrader-core 1:19.10.15.4
ProcVersionSignature: Ubuntu 5.0.0-40.44-generic 5.0.21
Uname: Linux 5.0.0-40-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu8.3
Architecture: amd64
CrashDB: ubuntu
Date: Fri Jan 24 06:02:32 2020
PackageArchitecture: all
ProcEnviron:
 TERM=rxvt-unicode-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: ubuntu-release-upgrader
UpgradeStatus: Upgraded to eoan on 2020-01-24 (0 days ago)
VarLogDistupgradeAptclonesystemstate.tar.gz:
 Error: command ['pkexec', 'cat', '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'] failed with exit code 127: polkit-agent-helper-1: error response to PolicyKit daemon: GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed: No session for cookie
 Error executing command as another user: Not authorized

 This incident has been reported.
VarLogDistupgradeXorgFixuplog:
 INFO:root:/usr/bin/do-release-upgrade running
 INFO:root:No xorg.conf, exiting

[Seth Arnold (seth-arnold)]

I'm surprised the snapshots showed up, I've got the snapdirs hidden:

$ zfs list -oname,snapdir
NAME SNAPDIR
bpool hidden
bpool/BOOT hidden
bpool/BOOT/ubuntu hidden
rpool hidden
rpool/ROOT hidden
rpool/ROOT/ubuntu hidden
rpool/home hidden
rpool/home/root hidden
rpool/home/sarnold hidden
rpool/swap -
rpool/tmp hidden
rpool/usr hidden
rpool/usr/local hidden
rpool/var hidden
rpool/var/cache hidden
rpool/var/lib hidden
rpool/var/lib/AccountsService hidden
rpool/var/lib/docker hidden
rpool/var/lib/nfs hidden
rpool/var/lib/schroot hidden
rpool/var/log hidden
rpool/var/mail hidden
rpool/var/snap hidden
rpool/var/spool hidden
rpool/var/tmp hidden
rpool/var/www hidden

[Seth Arnold (seth-arnold)]

I was able to fix this error:

 E: mkinitramfs failure cpio 141 lz4 -9 -l 24

by deleting the snapshots that sanoid made *during* the installation process:

sarnold@millbarge:/boot$ sudo zfs destroy bpool@autosnap_2020-01-24_04:00:05_hourly%autosnap_2020-01-24_08:45:05_frequently
[sudo] password for sarnold:
sarnold@millbarge:/boot$ sudo zfs destroy bpool/BOOT@autosnap_2020-01-24_04:00:05_hourly%autosnap_2020-01-24_08:45:05_frequently
sarnold@millbarge:/boot$ sudo zfs destroy bpool/BOOT/ubuntu@autosnap_2020-01-24_04:00:05_hourly%autosnap_2020-01-24_08:45:05_frequently
sarnold@millbarge:/boot$ df -h .
Filesystem Size Used Avail Use% Mounted on
bpool/BOOT/ubuntu 365M 156M 210M 43% /boot
sarnold@millbarge:/boot$ sudo dpkg --configure -a
Setting up initramfs-tools (0.133ubuntu10) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools (0.133ubuntu10) ...
update-initramfs: Generating /boot/initrd.img-5.3.0-29-generic
cryptsetup: ERROR: Couldn't resolve device rpool/ROOT/ubuntu
cryptsetup: WARNING: Couldn't determine root device
sarnold@millbarge:/boot$

Considering sanoid is not packaged in Ubuntu, there may not be anything that do-release-upgrade or initramfs-tools can do differently.

Thanks

[Seth Arnold (seth-arnold)]

My machine did not reboot successfully; because I followed The Guide, my system has a root password, and I was prompted for it at the systemd emergency shell. (Which is super-confusing, because I forgot I had set a password.)

The error message appeared to be along the lines of "bpool failed to import because it was most recently used on another machine". (This is a lie; rpool, on the same nvme, didn't say the same thing.)

I was able to proceed again after a 'zpool import -f bpool', reboot, and then things seemed to work well.

Thanks

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.