zfs-mount-generator results in ZFS Boot Stuck - Please Give Root Password or Press CTRL+D to Continue

[Stefano]

System Boot is Abnormal. Gives the message "Give root password for maintenance or press CTRL+D to continue".

I installed Ubuntu 23.10 on NVME using debootstrap (UEFI install).

I tried to re-generate the zfs cache and that seems to be set correctly in /etc/zfs/zfs-list.cache/rpool.

It is possible that the /root, /var/log, /home/ folders contained some data already. I tried to empty them and reboot, to make sure that ZFS would mount correctly.

Strangely enough zfs mount -a quickly finishes (without errors) while mount -a seems to enter an infinite loop.

The culprit issue seems to be (based on log files):
```
Nov 10 14:00:04 UbuntuWorkstation11 systemd[1]: Reloading requested from client PID 1743 (unit emergency.service)...
Nov 10 14:00:04 UbuntuWorkstation11 systemd[1]: Reloading...
Nov 10 14:00:04 UbuntuWorkstation11 (sd-execu[1747]: /usr/lib/systemd/system-generators/zfs-mount-generator.orig failed with exit status 1.
Nov 10 14:00:04 UbuntuWorkstation11 (sd-execu[1747]: /usr/lib/systemd/system-generators/zfs-mount-generator failed with exit status 1.
Nov 10 14:00:04 UbuntuWorkstation11 ERROR: zfs-mount-generator failed and you requested a revert:
Nov 10 14:00:04 UbuntuWorkstation11 systemd[1]: Reloading finished in 469 ms.
Nov 10 14:00:04 UbuntuWorkstation11 systemd[1]: Reloading requested from client PID 1807 (unit emergency.service)...
Nov 10 14:00:04 UbuntuWorkstation11 systemd[1]: Reloading...
Nov 10 14:00:04 UbuntuWorkstation11 (sd-execu[1811]: /usr/lib/systemd/system-generators/zfs-mount-generator failed with exit status 1.
Nov 10 14:00:04 UbuntuWorkstation11 (sd-execu[1811]: /usr/lib/systemd/system-generators/zfs-mount-generator.orig failed with exit status 1.
Nov 10 14:00:04 UbuntuWorkstation11 ERROR: zfs-mount-generator failed and you requested a revert:
Nov 10 14:00:05 UbuntuWorkstation11 systemd[1]: Reloading finished in 466 ms.
Nov 10 14:00:05 UbuntuWorkstation11 systemd[1]: Reloading requested from client PID 1733 (unit emergency.service)...
Nov 10 14:00:05 UbuntuWorkstation11 systemd[1]: Reloading...
Nov 10 14:00:05 UbuntuWorkstation11 (sd-execu[1873]: /usr/lib/systemd/system-generators/zfs-mount-generator failed with exit status 1.
Nov 10 14:00:05 UbuntuWorkstation11 (sd-execu[1873]: /usr/lib/systemd/system-generators/zfs-mount-generator.orig failed with exit status 1.
Nov 10 14:00:05 UbuntuWorkstation11 ERROR: zfs-mount-generator failed and you requested a revert:
Nov 10 14:00:05 UbuntuWorkstation11 systemd[1]: Reloading finished in 467 ms.
Nov 10 14:00:12 UbuntuWorkstation11 systemd[1]: Stopped target emergency.target - Emergency Mode.
```

It's difficult to know where the issue lies exactly, if the script/program doesn't return more informations on where it's actually failing.
```
root@UbuntuWorkstation11:/# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 7.30G 1.75T 96K none
rpool/ROOT 7.29G 1.75T 96K none
rpool/ROOT/ubuntu 7.29G 1.75T 6.81G /
rpool/ROOT/ubuntu/srv 96K 1.75T 96K /srv
rpool/ROOT/ubuntu/tmp 408K 1.75T 136K /tmp
rpool/ROOT/ubuntu/usr 424K 1.75T 96K /usr
rpool/ROOT/ubuntu/usr/local 328K 1.75T 192K /usr/local
rpool/ROOT/ubuntu/var 262M 1.75T 96K /var
rpool/ROOT/ubuntu/var/games 96K 1.75T 96K /var/games
rpool/ROOT/ubuntu/var/lib 259M 1.75T 56.4M /var/lib
rpool/ROOT/ubuntu/var/lib/AccountsService 224K 1.75T 96K /var/lib/AccountsService
rpool/ROOT/ubuntu/var/lib/NetworkManager 236K 1.75T 116K /var/lib/NetworkManager
rpool/ROOT/ubuntu/var/lib/apt 123M 1.75T 122M /var/lib/apt
rpool/ROOT/ubuntu/var/lib/dpkg 77.9M 1.75T 65.7M /var/lib/dpkg
rpool/ROOT/ubuntu/var/log 1.64M 1.75T 236K /var/log
rpool/ROOT/ubuntu/var/mail 96K 1.75T 96K /var/mail
rpool/ROOT/ubuntu/var/spool 264K 1.75T 144K /var/spool
rpool/ROOT/ubuntu/var/www 96K 1.75T 96K /var/www
rpool/USERDATA 524K 1.75T 96K none
rpool/USERDATA/root 276K 1.75T 140K /root
rpool/USERDATA/stefano 152K 1.75T 96K /home/stefano
```

The mounting order to /root, /home/stefano and /var/tmp is suspicious. And I have a feeling that ZFS keeps trying to automount since e.g. the journalctl log file that I created in /root/ quickly disappeared after creation.
```
root@UbuntuWorkstation11:/# cat /etc/fstab
# /boot Partition
UUID=8ce455bf-bc4e-4fe4-8a53-c488af3bb89c /boot ext4 auto 0 1

# /boot/efi Data
UUID=C39A-FFBB /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1

# Tools over NFS
nas.home:/export/tools /tools_nfs nfs rw,user=tools,auto,nfsvers=3 0 0
```

Tried ZFS 2.2.0 ("final") from mantic-proposed repository, but same issue happens. It seems zfs-mount-generator causes the boot process to fail, thus systemd requests Root Password or CTRL+D to continue.

```
root@UbuntuWorkstation02:/# zfs version
zfs-2.2.0-0ubuntu1~23.10
zfs-kmod-2.2.0-0ubuntu1~23.10
```

The bug is also reported upstream, however it's very likely to be Ubuntu-specific, since no other person complained about it.

Upstream bug report: https://github.com/openzfs/zfs/issues/15512

[Stefano (luckylinux777)]

Far from ideal, but if I boot into emergency mode, then run
```
zfs mount -a
systemctl restart networking
systemctl restart ssh
init 3
```

Then I get to a normal login prompt. Login as root & run /etc/init.d/lightdm restart and I can log in as a user in MATE.

Particularly the lines
```
zfs mount -a
init 3
```

Seem to be necessary.

And I don't know why but all datasets seem to mount correctly, EXCEPT:
```
/root
/home/stefano
/home/podman
```

[Stefano (luckylinux777)]

The issue could also be replicated on another system (512GB SSD, BIOS/legacy boot mode - not UEFI) which was upgraded from Ubuntu 23.04 to Ubuntu 23.10

[satmandu (satadru-umich)]

For what it is worth, I disabled these in debian/patches/series for my 2.2.0 based ppa at https://launchpad.net/~satadru-umich/+archive/ubuntu/zfs-experimental/ ...

#ubuntu/0001-Revert-etc-systemd-zfs-mount-generator-serialise-han.patch
#ubuntu/0002-Revert-etc-systemd-zfs-mount-generator-output-tweaks.patch
#ubuntu/0003-Revert-etc-systemd-zfs-mount-generator-rewrite-in-C.patch
#ubuntu/4000-zsys-support.patch
#ubuntu/4100-disable-bpool-upgrade.patch
#ubuntu/zfs-mount-container-start.patch
#ubuntu/4510-silently-ignore-modprobe-failure.patch
#ubuntu/4751-suppress-types.patch

[Stefano (luckylinux777)]

In my case I decided to give a try and install ZFS from source.

In that case the problem disappeared (plus I'm not stuck on a test version of ZFS that Ubuntu decided to ship for some reason).

Script in case anybody wants to replicate:

==============================================
#!/bin/bash
# Install Requirements
sudo apt-get --no-install-recommends install dkms
sudo apt-get install dh-dkms
sudo apt-get install aptitude libcurl4-openssl-dev libpam0g-dev lsb-release build-essential autoconf automake libtool libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev debhelper-compat dh-python po-debconf python3-all-dev python3-sphinx
sudo apt-get install build-essential autoconf automake libtool gawk fakeroot libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev debhelper-compat dh-python po-debconf python3-all-dev python3-sphinx
sudo apt-get install linux-headers-generic

# Define Desired Version
version="2.2.2"

cd /usr/src
mkdir -p zfs
cd zfs

# Save basedir
basedir=$(pwd)

# Use git and clone zfs-$version tag
#git clone https://github.com/openzfs/zfs.git --depth 1 --tag zfs-$version
#
# Use tar (working)
wget https://github.com/openzfs/zfs/archive/refs/tags/zfs-$version.tar.gz -O zfs-$version.tar.gz
mkdir -p zfs-$version
tar xvf zfs-$version.tar.gz -C zfs-$version --strip-components 1
#
# Use tar (currently broken archive)
#wget https://github.com/openzfs/zfs/releases/download/zfs-$version/zfs-$version.tar.gz -O zfs-$version.tar.gz
#tar xvf zfs-$version.tar.gz

# Change working direectory
cd zfs-$version

# Apply Patch in order to disable SIMD and Enable successfully ZFS Compile on ARM64
# NOT needed on AMD64
#wget https://raw.githubusercontent.com/chimera-linux/cports/master/main/zfs/patches/aarch64-disable-neon.patch -O aarch64-disable-neon.patch
#patch -p1 < aarch64-disable-neon.patch

sh autogen.sh
./configure
make -s -j$(nproc)
make native-deb
make native-deb-utils native-deb-dkms

# Select Subset of Packages to prevent installation of default linux-image and linux-headers
cd $basedir
mkdir -p selected-packages
mkdir -p selected-packages/$version
cd selected-packages/$version/
mv ../../openzfs-libnvpair3_$version*.deb ./
mv ../../openzfs-libpam-zfs_$version*.deb ./
mv ../../openzfs-libuutil3_$version*.deb ./
mv ../../openzfs-libzfs4_$version*.deb ./
mv ../../openzfs-libzpool5_$version*.deb ./
mv ../../openzfs-zfs-dkms_$version*.deb ./
mv ../../openzfs-zfs-initramfs_$version*.deb ./
mv ../../openzfs-zfs-zed_$version*.deb ./
mv ../../openzfs-zfsutils_$version*.deb ./

sudo apt-get install --fix-missing ./*.deb

==============================================