zfs not correctly imported at boot

[Panther]

On a fresh and up-to-date Ubuntu 20.04 amd64 installation I configured two encrypted partitions on the same hdd. On these I created a stripped zpool. After login I can import and mount the pool without problems, but the at-boot import fails after the first partitions is available and never tried again.

zpool version:
zfs-0.8.3-1ubuntu12.2
zfs-kmod-0.8.3-1ubuntu12.2
uname -a:
Linux hostname 5.4.0-40-generic #44-Ubuntu SMP Tue Jun 23 00:01:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
systemd --version
systemd 245 (245.4-4ubuntu3.2)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

Relevant logs:
Aug 17 07:12:25 hostname kernel: sd 1:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
Aug 17 07:12:25 hostname kernel: sd 1:0:0:0: [sdb] Write Protect is off
Aug 17 07:12:25 hostname kernel: sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Aug 17 07:12:25 hostname kernel: sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Aug 17 07:12:25 hostname kernel: sdb: sdb1 sdb2
Aug 17 07:12:25 hostname kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Aug 17 07:12:25 hostname systemd[1]: zfs-import-cache.service: Found ordering cycle on cryptsetup.target/start
Aug 17 07:12:25 hostname systemd[1]: zfs-import-cache.service: Found dependency on systemd-cryptsetup@vol\x2dswap_crypt.service/start
Aug 17 07:12:25 hostname systemd[1]: zfs-import-cache.service: Found dependency on systemd-random-seed.service/start
Aug 17 07:12:25 hostname systemd[1]: zfs-import-cache.service: Found dependency on zfs-mount.service/start
Aug 17 07:12:25 hostname systemd[1]: zfs-import-cache.service: Found dependency on zfs-import.target/start
Aug 17 07:12:25 hostname systemd[1]: zfs-import-cache.service: Found dependency on zfs-import-cache.service/start
Aug 17 07:12:25 hostname systemd[1]: zfs-import-cache.service: Job cryptsetup.target/start deleted to break ordering cycle starting with zfs-import-cache.service/start
Aug 17 07:12:25 hostname systemd[1]: cryptsetup.target: Found dependency on zfs-mount.service/start
Aug 17 07:12:25 hostname systemd[1]: cryptsetup.target: Found dependency on zfs-import.target/start
Aug 17 07:12:25 hostname systemd[1]: cryptsetup.target: Found dependency on zfs-import-cache.service/start
Aug 17 07:12:25 hostname systemd[1]: cryptsetup.target: Found dependency on zfs-mount.service/start
Aug 17 07:12:25 hostname systemd[1]: cryptsetup.target: Found dependency on zfs-import.target/start
Aug 17 07:12:25 hostname systemd[1]: cryptsetup.target: Found dependency on zfs-import-cache.service/start
Aug 17 07:12:26 hostname systemd[1]: cryptsetup.target: Found dependency on zfs-mount.service/start
Aug 17 07:12:26 hostname systemd[1]: cryptsetup.target: Found dependency on zfs-import.target/start
Aug 17 07:12:26 hostname systemd[1]: cryptsetup.target: Found dependency on zfs-import-cache.service/start
Aug 17 07:12:26 hostname systemd[1]: Starting Cryptography Setup for sdb1_crypt...
Aug 17 07:12:26 hostname systemd[1]: Starting Cryptography Setup for sdb2_crypt...
Aug 17 07:12:26 hostname systemd[1]: cryptsetup.target: Found dependency on zfs-mount.service/start
Aug 17 07:12:26 hostname systemd[1]: cryptsetup.target: Found dependency on zfs-import.target/start
Aug 17 07:12:26 hostname systemd[1]: cryptsetup.target: Found dependency on zfs-import-cache.service/start
Aug 17 07:12:32 hostname systemd[1]: Finished Cryptography Setup for sdb2_crypt.
Aug 17 07:12:32 hostname systemd[1]: Reached target Block Device Preparation for /dev/mapper/sdb2_crypt.
Aug 17 07:12:32 hostname zpool[1887]: cannot import 'sdb': no such pool or dataset
Aug 17 07:12:32 hostname zpool[1887]: Destroy and re-create the pool from
Aug 17 07:12:32 hostname zpool[1887]: a backup source.
Aug 17 07:12:32 hostname systemd[1]: zfs-import-cache.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 07:12:32 hostname systemd[1]: zfs-import-cache.service: Failed with result 'exit-code'.
Aug 17 07:12:34 hostname systemd[1]: Finished Cryptography Setup for sdb1_crypt.
Aug 17 07:12:34 hostname systemd[1]: Reached target Block Device Preparation for /dev/mapper/sdb1_crypt.

[Didier Roche-Tolomelli (didrocks)]

Please run apport-collect to attach logs so that we can debug your setting.
@baling: why subscribing zsys to this bu? There is no mention of zsys being used here, it seems directly a manual zfs setup.

[Panther (panther-physik)]

Yes, this ZFS is a manual setup. I tried to attach it to zfs-linux, but my browser failed me. @baling Thanks for moving the bug into the right direction.

A run of apport-collect fails, because
"Package zfs-linux not installed and no hook available, ignoring"
As zfs-linux is the source package, but not the installable package, this seems correct. Instead I did run "apport-bug --save foo zfs-dkms":

ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27.6
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 19 22:44:53 2020
Dependencies:
 adduser 3.118ubuntu2
 apt 2.0.2ubuntu0.1
 apt-utils 2.0.2ubuntu0.1
 binutils 2.34-6ubuntu1
 binutils-common 2.34-6ubuntu1
 binutils-x86-64-linux-gnu 2.34-6ubuntu1
 build-essential 12.8ubuntu1
 bzip2 1.0.8-2
 ca-certificates 20190110ubuntu1.1
 coreutils 8.30-3ubuntu2
 cpp 4:9.3.0-1ubuntu2
 cpp-9 9.3.0-10ubuntu2
 debconf 1.5.73
 debconf-i18n 1.5.73
 dirmngr 2.2.19-3ubuntu2
 distro-info-data 0.43ubuntu1.1
 dkms 2.8.1-5ubuntu1
 dpkg 1.19.7ubuntu3
 dpkg-dev 1.19.7ubuntu3
 fakeroot 1.24-1
 file 1:5.38-4
 g++ 4:9.3.0-1ubuntu2
 g++-9 9.3.0-10ubuntu2
 gcc 4:9.3.0-1ubuntu2
 gcc-10-base 10-20200411-0ubuntu1
 gcc-9 9.3.0-10ubuntu2
 gcc-9-base 9.3.0-10ubuntu2
 gnupg 2.2.19-3ubuntu2
 gnupg-l10n 2.2.19-3ubuntu2
 gnupg-utils 2.2.19-3ubuntu2
 gpg 2.2.19-3ubuntu2
 gpg-agent 2.2.19-3ubuntu2
 gpg-wks-client 2.2.19-3ubuntu2
 gpg-wks-server 2.2.19-3ubuntu2
 gpgconf 2.2.19-3ubuntu2
 gpgsm 2.2.19-3ubuntu2
 gpgv 2.2.19-3ubuntu2
 init-system-helpers 1.57
 kmod 27-1ubuntu2
 libacl1 2.2.53-6
 libalgorithm-diff-perl 1.19.03-2
 libalgorithm-diff-xs-perl 0.04-6
 libalgorithm-merge-perl 0.08-3
 libapt-pkg6.0 2.0.2ubuntu0.1
 libasan5 9.3.0-10ubuntu2
 libasn1-8-heimdal 7.7.0+dfsg-1ubuntu1
 libassuan0 2.5.3-7ubuntu2
 libatomic1 10-20200411-0ubuntu1
 libattr1 1:2.4.48-5
 libaudit-common 1:2.8.5-2ubuntu6
 libaudit1 1:2.8.5-2ubuntu6
 libbinutils 2.34-6ubuntu1
 libblkid1 2.34-0.1ubuntu9
 libbz2-1.0 1.0.8-2
 libc-dev-bin 2.31-0ubuntu9
 libc6 2.31-0ubuntu9
 libc6-dev 2.31-0ubuntu9
 libcap-ng0 0.7.9-2.1build1
 libcc1-0 10-20200411-0ubuntu1
 libcom-err2 1.45.5-2ubuntu1
 libcrypt-dev 1:4.4.10-10ubuntu4
 libcrypt1 1:4.4.10-10ubuntu4
 libctf-nobfd0 2.34-6ubuntu1
 libctf0 2.34-6ubuntu1
 libdb5.3 5.3.28+dfsg1-0.6ubuntu2
 libdpkg-perl 1.19.7ubuntu3
 libexpat1 2.2.9-1build1
 libfakeroot 1.24-1
 libffi7 3.3-4
 libfile-fcntllock-perl 0.22-3build4
 libgcc-9-dev 9.3.0-10ubuntu2
 libgcc-s1 10-20200411-0ubuntu1
 libgcrypt20 1.8.5-5ubuntu1
 libgdbm-compat4 1.18.1-5
 libgdbm6 1.18.1-5
 libgmp10 2:6.2.0+dfsg-4
 libgnutls30 3.6.13-2ubuntu1.2
 libgomp1 10-20200411-0ubuntu1
 libgpg-error0 1.37-1
 libgpm2 1.20.7-5
 libgssapi3-heimdal 7.7.0+dfsg-1ubuntu1
 libhcrypto4-heimdal 7.7.0+dfsg-1ubuntu1
 libheimbase1-heimdal 7.7.0+dfsg-1ubuntu1
 libheimntlm0-heimdal 7.7.0+dfsg-1ubuntu1
 libhogweed5 3.5.1+really3.5.1-2
 libhx509-5-heimdal 7.7.0+dfsg-1ubuntu1
 libidn2-0 2.2.0-2
 libisl22 0.22.1-1
 libitm1 10-20200411-0ubuntu1
 libkmod2 27-1ubuntu2
 libkrb5-26-heimdal 7.7.0+dfsg-1ubuntu1
 libksba8 1.3.5-2
 libldap-2.4-2 2.4.49+dfsg-2ubuntu1.3
 libldap-common 2.4.49+dfsg-2ubuntu1.3
 liblocale-gettext-perl 1.07-4
 liblsan0 10-20200411-0ubuntu1
 liblz4-1 ...

[Andrea Righi (arighi)]

I'm trying to reproduce the problem, but the zpool is always imported correctly for me also at boot. How does your /etc/crypttab look like? And zpool list -v? Thanks.

[Panther (panther-physik)]

Here are
zpool list -v:
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
sdb 1,81T 1,41T 415G - - 0% 77% 1.00x ONLINE -
  sdb1_crypt 928G 722G 206G - - 0% 77,8% - ONLINE
  sdb2_crypt 928G 719G 209G - - 0% 77,4% - ONLINE

and

cat /etc/crypttab:
vol-root_crypt UUID=$UUID1 none luks,discard
vol-swap_crypt /dev/mapper/nvme-swap /dev/urandom cipher=aes-xts-plain64,size=256,swap,discard
sdb1_crypt UUID=$UUID2 /root/sdb.key luks
sdb2_crypt UUID=$UUID3 /root/sdb.key luks

When I created the bug report I had the additional line
vol-tmp_crypt /dev/mapper/vol-tmp /dev/urandom cipher=aes-xts-plain64,size=256,tmp,discard
in my config, as reported in https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1891858

Lately I upgraded the CPU/Mainbord/Memory with more compute power. Instead of around 7200, I now have around 15000 CPUMark points. With the slower system, mounting did not work for weeks of daily booting. Since the upgrade the problem vanished.
May your system is to fast? Compare it with this https://www.cpubenchmark.net/cpu_lookup.php?cpu=Intel+Core+i7-4790+%40+3.60GHz&id=2226

[Panther (panther-physik)]

Just a correction to the previous comment. The problem did not vanish, but after 20 times booting the system, the mounting failed once.

[Dan Streetman (ddstreet)]

This might be a duplicate of bug 1875577