ZoL: wrong import order prevents boot

[Jens Elkner]

I've the following zfs:

# zfs list -r rpool/VARSHARE
NAME USED AVAIL REFER MOUNTPOINT
rpool/VARSHARE 114K 165G 30K /var/share
rpool/VARSHARE/lxc 84K 165G 19K /var/share/lxc
rpool/VARSHARE/lxc/xenial 65K 165G 19K /var/share/lxc/xenial
rpool/VARSHARE/lxc/xenial/pkg 19K 165G 19K /var/share/lxc/xenial/pkg
rpool/VARSHARE/lxc/xenial/rootfs-amd64 27K 165G 27K /var/share/lxc/xenial/rootfs-amd64

On boot, we see

         Starting Mount ZFS filesystems...
[FAILED] Failed to start Mount ZFS filesystems.
See 'systemctl status zfs-mount.service' for details.
Welcome to emergPress Enter for maintenance
(or press Control-D to continue):

# df -h /var/share
rpool/VARSHARE/lxc 165G 0 165G 0% /var/share/lxc
rpool/VARSHARE/lxc/xenial 165G 0 165G 0% /var/share/lxc/xenial
rpool/VARSHARE/lxc/xenial/pkg 165G 0 165G 0% /var/share/lxc/xenial/pkg
rpool/VARSHARE/lxc/xenial/rootfs-amd64 165G 0 165G 0% /var/share/lxc/xenial/rootfs-amd64

Obviously rpool/VARSHARE - the parent of rpool/VARSHARE/lxc - was not mounted, even so canmount property is for all set to on, rpool/VARSHARE's mountpoint to /var/share and rpool/VARSHARE/lxc children inherit their mountpoint.

# systemctl status zfs-mount.service
● zfs-mount.service - Mount ZFS filesystems
   Loaded: loaded (/lib/systemd/system/zfs-mount.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2017-05-28 04:51:46 CEST; 13min ago
  Process: 6935 ExecStart=/sbin/zfs mount -a (code=exited, status=1/FAILURE)
 Main PID: 6935 (code=exited, status=1/FAILURE)

May 28 04:51:45 ares systemd[1]: Starting Mount ZFS filesystems...
May 28 04:51:45 ares zfs[6935]: cannot mount '/var/share': directory is not empty
May 28 04:51:46 ares systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
May 28 04:51:46 ares systemd[1]: Failed to start Mount ZFS filesystems.
May 28 04:51:46 ares systemd[1]: zfs-mount.service: Unit entered failed state.
May 28 04:51:46 ares systemd[1]: zfs-mount.service: Failed with result 'exit-code'.

So 'zfs mount ...' seems to be severely buggy.

[Richard Laager (rlaager)]

This is the problem:
'/var/share': directory is not empty

Figure out what is in there and deal with this appropriately.

I don't personally love that ZFS refuses to mount on non-empty directories, but most of the time, the fact that the directory is non-empty is the real problem.

[Jens Elkner (jelmd)]

Please read carefully before answering: As already said (and the given output clearly shows the problem beside being mentioned in the subject): zfs screwed up the mount order and mounted rpool/VARSHARE/lxc _before_ rpool/VARSHARE. And when /var/share/lxc gets mounted, obviously /var/share is not empty and thus zfs,mount shot itself into the feet ...

[Richard Laager (rlaager)]

The output only shows the results, not the order of mount attempts. It may be the case that there is an ordering bug here. But we need to rule out the other case first. It could easily be the case that the directory is non-empty, so the /var/share mount failed even though it was properly attempted first.

Try manually unmounting everything under /var/share. Then rmdir the empty lxc directory. Once you are certain that /var/share is empty, re-run zfs mount -a.

[Jens Elkner (jelmd)]

It might be also a result of the bogus mount.zfs, which requires POSIX incorrect arguments but honors POSIXLY_CORRECT env var!

Had a similar zfs mount order problem with a rpool/local/home/{A,B,C} tree (was created using -p option) ...

[Launchpad Janitor (janitor)]

[Expired for zfs-linux (Ubuntu) because there has been no activity for 60 days.]