/dev/zvol links not created for encrypted zvols, zfs-volume-wait Timed out waiting on zvol links

[David]

Fresh installation of 20.04.4 with installer created zfs native encrypted root
zfsutils-linux: 0.8.3-1ubuntu12.13

On reboot, zvol /dev/zvol entries are not being created. This seems to effect different zvols each reboot.

Example logs:
```
-- A start job for unit zsys-gc.service has finished successfully.
--
-- The job identifier is 3139.
Apr 19 06:43:49 monolith zvol_wait[74768]: cannot open 'rpool/export/vault/block': dataset does not exist
Apr 19 06:43:50 monolith zvol_wait[74786]: cannot open 'rpool/export/vault/block': dataset does not exist
Apr 19 06:43:51 monolith zvol_wait[74805]: cannot open 'rpool/export/vault/block': dataset does not exist
Apr 19 06:43:52 monolith zvol_wait[74823]: cannot open 'rpool/export/vault/block': dataset does not exist
Apr 19 06:43:53 monolith zvol_wait[74932]: cannot open 'rpool/export/vault/block': dataset does not exist
Apr 19 06:43:54 monolith zvol_wait[74950]: cannot open 'rpool/export/vault/block': dataset does not exist
Apr 19 06:43:55 monolith zvol_wait[74969]: cannot open 'rpool/export/vault/block': dataset does not exist
Apr 19 06:43:56 monolith zvol_wait[74987]: cannot open 'rpool/export/vault/block': dataset does not exist
Apr 19 06:43:57 monolith zvol_wait[75006]: cannot open 'rpool/export/vault/block': dataset does not exist
Apr 19 06:43:58 monolith zvol_wait[4440]: Still waiting on 1 zvol links ...
Apr 19 06:43:58 monolith zvol_wait[4440]: No progress since last loop.
Apr 19 06:43:58 monolith zvol_wait[4440]: Checking if any zvols were deleted.
Apr 19 06:43:58 monolith zvol_wait[4440]: Remaining zvols:
Apr 19 06:43:58 monolith zvol_wait[4440]: rpool/libvirt_zvols/kube-04-disk0
Apr 19 06:43:58 monolith zvol_wait[4440]: Timed out waiting on zvol links
Apr 19 06:43:58 monolith systemd[1]: zfs-volume-wait.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit zfs-volume-wait.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Apr 19 06:43:58 monolith systemd[1]: zfs-volume-wait.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit zfs-volume-wait.service has entered the 'failed' state with result 'exit-code'.
Apr 19 06:43:58 monolith systemd[1]: Failed to start Wait for ZFS Volume (zvol) links in /dev.
-- Subject: A start job for unit zfs-volume-wait.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit zfs-volume-wait.service has finished with a failure.
--
-- The job identifier is 226 and the job result is failed.
Apr 19 06:43:58 monolith systemd[1]: Dependency failed for ZFS volumes are ready.
-- Subject: A start job for unit zfs-volumes.target has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit zfs-volumes.target has finished with a failure.
```
I believe this is related to changes made here:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1888405

[David (dprude)]

It looks as if someone has filed a similar bug for debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=997980

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alan Baghumian (alanbach)]

This is still happening, any progress on the fix?

Mar 26 19:41:36 juju-868061-case00356962-2 zvol_wait[123774]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:36 juju-868061-case00356962-2 sudo[120459]: pam_unix(sudo:session): session closed for user root
Mar 26 19:41:37 juju-868061-case00356962-2 zvol_wait[123809]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:38 juju-868061-case00356962-2 zvol_wait[123818]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:39 juju-868061-case00356962-2 zvol_wait[123827]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:40 juju-868061-case00356962-2 zvol_wait[122728]: Still waiting on 1 zvol links ...
Mar 26 19:41:40 juju-868061-case00356962-2 zvol_wait[122728]: No progress since last loop.
Mar 26 19:41:40 juju-868061-case00356962-2 zvol_wait[122728]: Checking if any zvols were deleted.
Mar 26 19:41:40 juju-868061-case00356962-2 zvol_wait[122728]: Remaining zvols:
Mar 26 19:41:40 juju-868061-case00356962-2 zvol_wait[122728]: local/virtual-machines/u3.block
Mar 26 19:41:40 juju-868061-case00356962-2 zvol_wait[123846]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:41 juju-868061-case00356962-2 zvol_wait[123855]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:42 juju-868061-case00356962-2 zvol_wait[123864]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:43 juju-868061-case00356962-2 zvol_wait[123873]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:44 juju-868061-case00356962-2 zvol_wait[123882]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:45 juju-868061-case00356962-2 zvol_wait[123891]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:46 juju-868061-case00356962-2 zvol_wait[123901]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:47 juju-868061-case00356962-2 zvol_wait[123910]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:48 juju-868061-case00356962-2 zvol_wait[123919]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:49 juju-868061-case00356962-2 zvol_wait[123928]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:50 juju-868061-case00356962-2 zvol_wait[123937]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:51 juju-868061-case00356962-2 zvol_wait[123960]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:52 juju-868061-case00356962-2 zvol_wait[123969]: cannot open 'rpool/export/vault/block': dataset does not exist
Mar 26 19:41:53 juju-868061-case00356962-2 zvol_wait[123979]: cannot open 'rpool/export/vault/block': dataset does not exist

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal

$ dpkg -l | grep zfs
ii libzfs2linux 0.8.3-1ubuntu12.14 amd64 OpenZFS filesystem library for Linux
iU zfs-zed 0.8.3-1ubuntu12.14 ...

[homerlinux (mibanezm)]

May be, by any chance, that your zvols contain LVMs?

[David (dprude)]

In my case, they did not.

On Tue, Apr 25, 2023 at 4:01 AM homerlinux <email address hidden>
wrote:

> May be, by any chance, that your zvols contain LVMs?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1969457
>
> Title:
> /dev/zvol links not created for encrypted zvols, zfs-volume-wait Timed
> out waiting on zvol links
>
> Status in zfs-linux package in Ubuntu:
> Confirmed
>
> Bug description:
> Fresh installation of 20.04.4 with installer created zfs native
> encrypted root
> zfsutils-linux: 0.8.3-1ubuntu12.13
>
> On reboot, zvol /dev/zvol entries are not being created. This seems to
> effect different zvols each reboot.
>
> Example logs:
> ```
> -- A start job for unit zsys-gc.service has finished successfully.
> --
> -- The job identifier is 3139.
> Apr 19 06:43:49 monolith zvol_wait[74768]: cannot open
> 'rpool/export/vault/block': dataset does not exist
> Apr 19 06:43:50 monolith zvol_wait[74786]: cannot open
> 'rpool/export/vault/block': dataset does not exist
> Apr 19 06:43:51 monolith zvol_wait[74805]: cannot open
> 'rpool/export/vault/block': dataset does not exist
> Apr 19 06:43:52 monolith zvol_wait[74823]: cannot open
> 'rpool/export/vault/block': dataset does not exist
> Apr 19 06:43:53 monolith zvol_wait[74932]: cannot open
> 'rpool/export/vault/block': dataset does not exist
> Apr 19 06:43:54 monolith zvol_wait[74950]: cannot open
> 'rpool/export/vault/block': dataset does not exist
> Apr 19 06:43:55 monolith zvol_wait[74969]: cannot open
> 'rpool/export/vault/block': dataset does not exist
> Apr 19 06:43:56 monolith zvol_wait[74987]: cannot open
> 'rpool/export/vault/block': dataset does not exist
> Apr 19 06:43:57 monolith zvol_wait[75006]: cannot open
> 'rpool/export/vault/block': dataset does not exist
> Apr 19 06:43:58 monolith zvol_wait[4440]: Still waiting on 1 zvol links
> ...
> Apr 19 06:43:58 monolith zvol_wait[4440]: No progress since last loop.
> Apr 19 06:43:58 monolith zvol_wait[4440]: Checking if any zvols were
> deleted.
> Apr 19 06:43:58 monolith zvol_wait[4440]: Remaining zvols:
> Apr 19 06:43:58 monolith zvol_wait[4440]:
> rpool/libvirt_zvols/kube-04-disk0
> Apr 19 06:43:58 monolith zvol_wait[4440]: Timed out waiting on zvol links
> Apr 19 06:43:58 monolith systemd[1]: zfs-volume-wait.service: Main
> process exited, code=exited, status=1/FAILURE
> -- Subject: Unit process exited
> -- Defined-By: systemd
> -- Support: http://www.ubuntu.com/support
> --
> -- An ExecStart= process belonging to unit zfs-volume-wait.service has
> exited.
> --
> -- The process' exit code is 'exited' and its exit status is 1.
> Apr 19 06:43:58 monolith systemd[1]: zfs-volume-wait.service: Failed
> with result 'exit-code'.
> -- Subject: Unit failed
> -- Defined-By: systemd
> -- Support: http://www.ubuntu.com/support
> --
> -- The unit zfs-volume-wait.service has entered the 'failed' state with
> result 'exit-code'.
> Apr 19 06:43:58 monolith systemd[1]: Failed to start Wait for ZFS Volume
> (zvol) links in /dev.
> -- Subject: A start job for unit zfs-volume-wait.s...