Ubuntu
zfs-linux package

  1. Bug #1837717
  2. Activity log

Activity log for bug #1837717

Date Who What changed Old value New value Message
2019-07-24 10:31:23 Didier Roche-Tolomelli bug added bug
2019-07-24 10:35:48 Jean-Baptiste Lallement bug added subscriber Jean-Baptiste Lallement
2019-07-24 11:02:17 Didier Roche-Tolomelli description # zpool create -o ashift=12 -O atime=off -O canmount=off -O normalization=formD -O mountpoint=/ -R /mnt rpool /dev/vda2 # zfs create rpool/ROOT -o canmount=off -o mountpoint=none # zfs create rpool/ROOT/ubuntu_123456 -o mountpoint=/ # zfs create rpool/ROOT/ubuntu_123456/var # zfs create rpool/ROOT/ubuntu_123456/var/lib -o canmount=off # zfs create rpool/var -o canmount=off # zfs create rpool/var/lib # zfs create rpool/var/games # zfs create rpool/ROOT/ubuntu_123456/var/lib/apt Zfs mount is what we expect (5 datasets mounted): # zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/var/lib /mnt/var/lib rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt # zfs umount -a Everything unmounted as expected: # find /mnt/ /mnt/ However, zfs mount -a doesn't mount everything in the correct order reliably: # zfs mount -a cannot mount '/mnt': directory is not empty -> In that case, rpool/ROOT/ubuntu_123456 wasn't mounted: # zfs mount rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/lib /mnt/var/lib rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt # find /mnt/ /mnt/ /mnt/var /mnt/var/lib /mnt/var/lib/apt /mnt/var/games # zfs umount -a # find /mnt/ /mnt/ -> Everything was umounted, let's try to remount all again: # zfs mount -a cannot mount '/mnt/var/lib': failed to create mountpoint # zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt -> This time, rpool/ROOT/ubuntu_123456 was mounted, but not rpool/var/lib (before rpool/ROOT/ubuntu_123456/var/lib/apt) Tested as well with zfs 0.7: tried to zfs mount -a && zfs umount -a in loop, no issue: all datasets are mounted in the correct order reliably. Note that it seems to be related to the version of zfs we created a pool with: - Try zfs mount -a on zfs 0.7 with a pool/datasets created under zfs 0.8: the ordering issues occur - Try zfs mount -a on zfs 0.8 with a pool/datasets created under zfs 0.7: no ordering issue. # zpool create -o ashift=12 -O atime=off -O canmount=off -O normalization=formD -O mountpoint=/ -R /mnt rpool /dev/vda2 # zfs create rpool/ROOT -o canmount=off -o mountpoint=none # zfs create rpool/ROOT/ubuntu_123456 -o mountpoint=/ # zfs create rpool/ROOT/ubuntu_123456/var # zfs create rpool/ROOT/ubuntu_123456/var/lib -o canmount=off # zfs create rpool/var -o canmount=off # zfs create rpool/var/lib # zfs create rpool/var/games # zfs create rpool/ROOT/ubuntu_123456/var/lib/apt Zfs mount is what we expect (5 datasets mounted): # zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/var/lib /mnt/var/lib rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt # zfs umount -a Everything unmounted as expected: # find /mnt/ /mnt/ However, zfs mount -a doesn't mount everything in the correct order reliably: # zfs mount -a cannot mount '/mnt': directory is not empty -> In that case, rpool/ROOT/ubuntu_123456 wasn't mounted: # zfs mount rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/lib /mnt/var/lib rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt # find /mnt/ /mnt/ /mnt/var /mnt/var/lib /mnt/var/lib/apt /mnt/var/games # zfs umount -a # find /mnt/ /mnt/ -> Everything was umounted, let's try to remount all again: # zfs mount -a cannot mount '/mnt/var/lib': failed to create mountpoint # zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt -> This time, rpool/ROOT/ubuntu_123456 was mounted, but not rpool/var/lib (before rpool/ROOT/ubuntu_123456/var/lib/apt) Note: the same ordering issue can happen on zfs umount -a. Tested as well with zfs 0.7: tried to zfs mount -a && zfs umount -a in loop, no issue: all datasets are mounted in the correct order reliably. Note that it seems to be slightly related to the version of zfs we created a pool with: - Try zfs mount -a on zfs 0.7 with a pool/datasets created under zfs 0.8: the ordering issues still happens. - However, the contrary isn't try: try zfs mount -a on zfs 0.8 with a pool/datasets created under zfs 0.7: there can be some ordering issues. There is nothing specific in the journal log: juil. 24 10:59:27 ubuntu kernel: ZFS: Loaded module v0.8.1-1ubuntu5, ZFS pool version 5000, ZFS filesystem version 5 juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1116]: mnt-var.mount: Succeeded.
2019-07-24 11:11:02 Didier Roche-Tolomelli description # zpool create -o ashift=12 -O atime=off -O canmount=off -O normalization=formD -O mountpoint=/ -R /mnt rpool /dev/vda2 # zfs create rpool/ROOT -o canmount=off -o mountpoint=none # zfs create rpool/ROOT/ubuntu_123456 -o mountpoint=/ # zfs create rpool/ROOT/ubuntu_123456/var # zfs create rpool/ROOT/ubuntu_123456/var/lib -o canmount=off # zfs create rpool/var -o canmount=off # zfs create rpool/var/lib # zfs create rpool/var/games # zfs create rpool/ROOT/ubuntu_123456/var/lib/apt Zfs mount is what we expect (5 datasets mounted): # zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/var/lib /mnt/var/lib rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt # zfs umount -a Everything unmounted as expected: # find /mnt/ /mnt/ However, zfs mount -a doesn't mount everything in the correct order reliably: # zfs mount -a cannot mount '/mnt': directory is not empty -> In that case, rpool/ROOT/ubuntu_123456 wasn't mounted: # zfs mount rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/lib /mnt/var/lib rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt # find /mnt/ /mnt/ /mnt/var /mnt/var/lib /mnt/var/lib/apt /mnt/var/games # zfs umount -a # find /mnt/ /mnt/ -> Everything was umounted, let's try to remount all again: # zfs mount -a cannot mount '/mnt/var/lib': failed to create mountpoint # zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt -> This time, rpool/ROOT/ubuntu_123456 was mounted, but not rpool/var/lib (before rpool/ROOT/ubuntu_123456/var/lib/apt) Note: the same ordering issue can happen on zfs umount -a. Tested as well with zfs 0.7: tried to zfs mount -a && zfs umount -a in loop, no issue: all datasets are mounted in the correct order reliably. Note that it seems to be slightly related to the version of zfs we created a pool with: - Try zfs mount -a on zfs 0.7 with a pool/datasets created under zfs 0.8: the ordering issues still happens. - However, the contrary isn't try: try zfs mount -a on zfs 0.8 with a pool/datasets created under zfs 0.7: there can be some ordering issues. There is nothing specific in the journal log: juil. 24 10:59:27 ubuntu kernel: ZFS: Loaded module v0.8.1-1ubuntu5, ZFS pool version 5000, ZFS filesystem version 5 juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1116]: mnt-var.mount: Succeeded. # zpool create -o ashift=12 -O atime=off -O canmount=off -O normalization=formD -O mountpoint=/ -R /mnt rpool /dev/vda2 # zfs create rpool/ROOT -o canmount=off -o mountpoint=none # zfs create rpool/ROOT/ubuntu_123456 -o mountpoint=/ # zfs create rpool/ROOT/ubuntu_123456/var # zfs create rpool/ROOT/ubuntu_123456/var/lib -o canmount=off # zfs create rpool/var -o canmount=off # zfs create rpool/var/lib # zfs create rpool/var/games # zfs create rpool/ROOT/ubuntu_123456/var/lib/apt Zfs mount is what we expect (5 datasets mounted): # zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/var/lib /mnt/var/lib rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt # zfs umount -a Everything unmounted as expected: # find /mnt/ /mnt/ However, zfs mount -a doesn't mount everything in the correct order reliably: # zfs mount -a cannot mount '/mnt': directory is not empty -> In that case, rpool/ROOT/ubuntu_123456 wasn't mounted: # zfs mount rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/lib /mnt/var/lib rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt # find /mnt/ /mnt/ /mnt/var /mnt/var/lib /mnt/var/lib/apt /mnt/var/games # zfs umount -a # find /mnt/ /mnt/ -> Everything was umounted, let's try to remount all again: # zfs mount -a cannot mount '/mnt/var/lib': failed to create mountpoint # zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt -> This time, rpool/ROOT/ubuntu_123456 was mounted, but not rpool/var/lib (before rpool/ROOT/ubuntu_123456/var/lib/apt) Note: the same ordering issue can happen on zfs umount -a. Tested as well with zfs 0.7: tried to zfs mount -a && zfs umount -a in loop, no issue: all datasets are mounted in the correct order reliably. Note that it seems to be slightly related to the version of zfs we created a pool with: - Try zfs mount -a on zfs 0.7 with a pool/datasets created under zfs 0.8: the ordering issues still happens. - However, the contrary isn't try: try zfs mount -a on zfs 0.8 with a pool/datasets created under zfs 0.7: there can be some ordering issues. There is nothing specific in the journal log: juil. 24 10:59:27 ubuntu kernel: ZFS: Loaded module v0.8.1-1ubuntu5, ZFS pool version 5000, ZFS filesystem version 5 juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1116]: mnt-var.mount: Succeeded. Note that pools created with 0.8-3 from debian has the same issues.
2019-07-24 11:16:43 Didier Roche-Tolomelli description # zpool create -o ashift=12 -O atime=off -O canmount=off -O normalization=formD -O mountpoint=/ -R /mnt rpool /dev/vda2 # zfs create rpool/ROOT -o canmount=off -o mountpoint=none # zfs create rpool/ROOT/ubuntu_123456 -o mountpoint=/ # zfs create rpool/ROOT/ubuntu_123456/var # zfs create rpool/ROOT/ubuntu_123456/var/lib -o canmount=off # zfs create rpool/var -o canmount=off # zfs create rpool/var/lib # zfs create rpool/var/games # zfs create rpool/ROOT/ubuntu_123456/var/lib/apt Zfs mount is what we expect (5 datasets mounted): # zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/var/lib /mnt/var/lib rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt # zfs umount -a Everything unmounted as expected: # find /mnt/ /mnt/ However, zfs mount -a doesn't mount everything in the correct order reliably: # zfs mount -a cannot mount '/mnt': directory is not empty -> In that case, rpool/ROOT/ubuntu_123456 wasn't mounted: # zfs mount rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/lib /mnt/var/lib rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt # find /mnt/ /mnt/ /mnt/var /mnt/var/lib /mnt/var/lib/apt /mnt/var/games # zfs umount -a # find /mnt/ /mnt/ -> Everything was umounted, let's try to remount all again: # zfs mount -a cannot mount '/mnt/var/lib': failed to create mountpoint # zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt -> This time, rpool/ROOT/ubuntu_123456 was mounted, but not rpool/var/lib (before rpool/ROOT/ubuntu_123456/var/lib/apt) Note: the same ordering issue can happen on zfs umount -a. Tested as well with zfs 0.7: tried to zfs mount -a && zfs umount -a in loop, no issue: all datasets are mounted in the correct order reliably. Note that it seems to be slightly related to the version of zfs we created a pool with: - Try zfs mount -a on zfs 0.7 with a pool/datasets created under zfs 0.8: the ordering issues still happens. - However, the contrary isn't try: try zfs mount -a on zfs 0.8 with a pool/datasets created under zfs 0.7: there can be some ordering issues. There is nothing specific in the journal log: juil. 24 10:59:27 ubuntu kernel: ZFS: Loaded module v0.8.1-1ubuntu5, ZFS pool version 5000, ZFS filesystem version 5 juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1116]: mnt-var.mount: Succeeded. Note that pools created with 0.8-3 from debian has the same issues. Update: I was able to reproduce it with a simpler schema (/ isn't always mounted before /var). This is to mimick the official zol guide with zfs on root: https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS zpool create -o ashift=12 -O atime=off -O canmount=off -O normalization=formD -O mountpoint=/ -R /mnt rpool /dev/vda2 zfs create rpool/ROOT -o canmount=off -o mountpoint=none zfs create rpool/ROOT/ubuntu_123456 -o mountpoint=/ zfs create rpool/ROOT/ubuntu_123456/var zpool create -o ashift=12 -O atime=off -O canmount=off -O normalization=formD -O mountpoint=/ -R /mnt rpool /dev/vda2 zfs create rpool/ROOT -o canmount=off -o mountpoint=none zfs create rpool/ROOT/ubuntu_123456 -o mountpoint=/ zfs create rpool/ROOT/ubuntu_123456/var zfs create rpool/ROOT/ubuntu_123456/var/lib -o canmount=off zfs create rpool/var -o canmount=off zfs create rpool/var/lib zfs create rpool/var/games zfs create rpool/ROOT/ubuntu_123456/var/lib/apt # Zfs mount is what we expect (5 datasets mounted): zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/var/lib /mnt/var/lib rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt zfs umount -a # Everything unmounted as expected: find /mnt/ /mnt/ # However, zfs mount -a doesn't mount everything in the correct order reliably: zfs mount -a cannot mount '/mnt': directory is not empty # In that case, rpool/ROOT/ubuntu_123456 wasn't mounted: zfs mount rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/lib /mnt/var/lib rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt find /mnt/ /mnt/ /mnt/var /mnt/var/lib /mnt/var/lib/apt /mnt/var/games zfs umount -a find /mnt/ /mnt/ # Everything was umounted, let's try to remount all again: zfs mount -a cannot mount '/mnt/var/lib': failed to create mountpoint zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt #This time, rpool/ROOT/ubuntu_123456 was mounted, but not rpool/var/lib (before rpool/ROOT/ubuntu_123456/var/lib/apt) Note: the same ordering issue can happen on zfs umount -a. Tested as well with zfs 0.7: tried to zfs mount -a && zfs umount -a in loop, no issue: all datasets are mounted in the correct order reliably. Note that it seems to be slightly related to the version of zfs we created a pool with: - Try zfs mount -a on zfs 0.7 with a pool/datasets created under zfs 0.8: the ordering issues still happens. - However, the contrary isn't try: try zfs mount -a on zfs 0.8 with a pool/datasets created under zfs 0.7: there can be some ordering issues. There is nothing specific in the journal log: juil. 24 10:59:27 ubuntu kernel: ZFS: Loaded module v0.8.1-1ubuntu5, ZFS pool version 5000, ZFS filesystem version 5 juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1116]: mnt-var.mount: Succeeded. Note that pools created with 0.8-3 from debian has the same issues.
2019-07-24 11:17:34 Didier Roche-Tolomelli description Update: I was able to reproduce it with a simpler schema (/ isn't always mounted before /var). This is to mimick the official zol guide with zfs on root: https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS zpool create -o ashift=12 -O atime=off -O canmount=off -O normalization=formD -O mountpoint=/ -R /mnt rpool /dev/vda2 zfs create rpool/ROOT -o canmount=off -o mountpoint=none zfs create rpool/ROOT/ubuntu_123456 -o mountpoint=/ zfs create rpool/ROOT/ubuntu_123456/var zpool create -o ashift=12 -O atime=off -O canmount=off -O normalization=formD -O mountpoint=/ -R /mnt rpool /dev/vda2 zfs create rpool/ROOT -o canmount=off -o mountpoint=none zfs create rpool/ROOT/ubuntu_123456 -o mountpoint=/ zfs create rpool/ROOT/ubuntu_123456/var zfs create rpool/ROOT/ubuntu_123456/var/lib -o canmount=off zfs create rpool/var -o canmount=off zfs create rpool/var/lib zfs create rpool/var/games zfs create rpool/ROOT/ubuntu_123456/var/lib/apt # Zfs mount is what we expect (5 datasets mounted): zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/var/lib /mnt/var/lib rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt zfs umount -a # Everything unmounted as expected: find /mnt/ /mnt/ # However, zfs mount -a doesn't mount everything in the correct order reliably: zfs mount -a cannot mount '/mnt': directory is not empty # In that case, rpool/ROOT/ubuntu_123456 wasn't mounted: zfs mount rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/lib /mnt/var/lib rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt find /mnt/ /mnt/ /mnt/var /mnt/var/lib /mnt/var/lib/apt /mnt/var/games zfs umount -a find /mnt/ /mnt/ # Everything was umounted, let's try to remount all again: zfs mount -a cannot mount '/mnt/var/lib': failed to create mountpoint zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt #This time, rpool/ROOT/ubuntu_123456 was mounted, but not rpool/var/lib (before rpool/ROOT/ubuntu_123456/var/lib/apt) Note: the same ordering issue can happen on zfs umount -a. Tested as well with zfs 0.7: tried to zfs mount -a && zfs umount -a in loop, no issue: all datasets are mounted in the correct order reliably. Note that it seems to be slightly related to the version of zfs we created a pool with: - Try zfs mount -a on zfs 0.7 with a pool/datasets created under zfs 0.8: the ordering issues still happens. - However, the contrary isn't try: try zfs mount -a on zfs 0.8 with a pool/datasets created under zfs 0.7: there can be some ordering issues. There is nothing specific in the journal log: juil. 24 10:59:27 ubuntu kernel: ZFS: Loaded module v0.8.1-1ubuntu5, ZFS pool version 5000, ZFS filesystem version 5 juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1116]: mnt-var.mount: Succeeded. Note that pools created with 0.8-3 from debian has the same issues. Update: I was able to reproduce it with a simpler schema (/ isn't always mounted before /var). This is to mimick the official zol guide with zfs on root: https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS $ zpool create -o ashift=12 -O atime=off -O canmount=off -O normalization=formD -O mountpoint=/ -R /mnt rpool /dev/vda2 $ zfs create rpool/ROOT -o canmount=off -o mountpoint=none $ zfs create rpool/ROOT/ubuntu_123456 -o mountpoint=/ $ zfs create rpool/ROOT/ubuntu_123456/var $ zpool create -o ashift=12 -O atime=off -O canmount=off -O normalization=formD -O mountpoint=/ -R /mnt rpool /dev/vda2 $ zfs create rpool/ROOT -o canmount=off -o mountpoint=none $ zfs create rpool/ROOT/ubuntu_123456 -o mountpoint=/ $ zfs create rpool/ROOT/ubuntu_123456/var $ zfs create rpool/ROOT/ubuntu_123456/var/lib -o canmount=off $ zfs create rpool/var -o canmount=off $ zfs create rpool/var/lib $ zfs create rpool/var/games $ zfs create rpool/ROOT/ubuntu_123456/var/lib/apt # Zfs mount is what we expect (5 datasets mounted): $ zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/var/lib /mnt/var/lib rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt $ zfs umount -a # Everything unmounted as expected: $ find /mnt/ /mnt/ # However, zfs mount -a doesn't mount everything in the correct order reliably: $ zfs mount -a cannot mount '/mnt': directory is not empty # In that case, rpool/ROOT/ubuntu_123456 wasn't mounted: $ zfs mount rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/lib /mnt/var/lib rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt $ find /mnt/ /mnt/ /mnt/var /mnt/var/lib /mnt/var/lib/apt /mnt/var/games $ zfs umount -a $ find /mnt/ /mnt/ # Everything was umounted, let's try to remount all again: $ zfs mount -a cannot mount '/mnt/var/lib': failed to create mountpoint $ zfs mount rpool/ROOT/ubuntu_123456 /mnt rpool/ROOT/ubuntu_123456/var /mnt/var rpool/var/games /mnt/var/games rpool/ROOT/ubuntu_123456/var/lib/apt /mnt/var/lib/apt #This time, rpool/ROOT/ubuntu_123456 was mounted, but not rpool/var/lib (before rpool/ROOT/ubuntu_123456/var/lib/apt) Note: the same ordering issue can happen on zfs umount -a. Tested as well with zfs 0.7: tried to zfs mount -a && zfs umount -a in loop, no issue: all datasets are mounted in the correct order reliably. Note that it seems to be slightly related to the version of zfs we created a pool with: - Try zfs mount -a on zfs 0.7 with a pool/datasets created under zfs 0.8: the ordering issues still happens. - However, the contrary isn't try: try zfs mount -a on zfs 0.8 with a pool/datasets created under zfs 0.7: there can be some ordering issues. There is nothing specific in the journal log: juil. 24 10:59:27 ubuntu kernel: ZFS: Loaded module v0.8.1-1ubuntu5, ZFS pool version 5000, ZFS filesystem version 5 juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:39 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 10:59:42 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1116]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:45 ubuntu systemd[1]: mnt-var-lib.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 10:59:46 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-lib-apt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt-var-games.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1]: mnt.mount: Succeeded. juil. 24 11:01:06 ubuntu systemd[1116]: mnt.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1]: mnt-var.mount: Succeeded. juil. 24 11:01:08 ubuntu systemd[1116]: mnt-var.mount: Succeeded. Note that pools created with 0.8-3 from debian has the same issues.
2019-07-24 11:25:02 Didier Roche-Tolomelli bug watch added https://github.com/zfsonlinux/zfs/issues/8833
2019-07-24 11:25:02 Didier Roche-Tolomelli bug task added zfs
2019-07-31 11:57:55 Launchpad Janitor zfs-linux (Ubuntu): status New Fix Released
2019-09-15 09:16:42 Bug Watch Updater zfs: status Unknown Fix Released