10_linuxzfs fails if a non-root filesystem has a mountpoint of /

[Jason S]

For ~4 years, this server ran Ubuntu-18.04 with the OS on mirrored SSDs and all persistent data in a zpool on separate disks. Last week I upgraded it. Installation of Ubuntu-22.04 ZFS-root on a new SSD ran flawlessly. But after importing the persistent data pool, update-grub (specifically 10_linux_zfs) found the valid kernels but wouldn't create grub entries for them so /boot/grub/grub.cfg only contained memtest86. 10_linux_zfs's section of grub-mkconfig output is below.

The problem turned out to be a FreeBSD backup I'd zfs-send/rec'd to the Ubuntu system 2 years ago. At the time, I added a prefix ("zmule") to all of the FreeBSD filesystems' mountpoints and turned off "canmount". Somehow I missed the *actual* root filesystem so it was still set to mount on '/' with "canmount" enabled - though Ubuntu22.04 wouldn't mount it on top of /, so I never noticed it. However, that's what was confusing /etc/grub.d/10_linux_zfs even though 18.04 was fine with it. Once I changed FreeBSD's root mountpoint to not '/', update-grub ran as expected.

I know this is a strange, unusual situation that shouldn't happen, but a missing /etc/os-release file in one root(ish) filesystem shouldn't block it from creating entries for valid ones. If you'd like me to break it again and run 10_linux_zfs with 'set -x' or similar, I'll be happy to do that. (I'm a Linux SRE professionally so breaking stuff is all in a days work.)

One more note: the bug submission wouldn't let me use "grub-common" as the package, so I changed that to "grub2" even though, as you can see below, that's not what's installed.

===================================================================
More detail:

$ lsb_release -rd
Description: Ubuntu 22.04 LTS
Release: 22.04
$ apt-cache policy grub-common
grub-common:
  Installed: 2.06-2ubuntu7
  Candidate: 2.06-2ubuntu7
  Version table:
 *** 2.06-2ubuntu7 500
        500 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
$ apt-cache policy grub2
grub2:
  Installed: (none)
  Candidate: 2.06-2ubuntu7
  Version table:
     2.06-2ubuntu7 500
        500 http://us.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages

-----------------

### BEGIN /etc/grub.d/10_linux_zfs ###
Found linux image: vmlinuz-5.15.0-27-generic in rpool/ROOT/ubuntu_lva06t
Found initrd image: initrd.img-5.15.0-27-generic in rpool/ROOT/ubuntu_lva06t
Found linux image: vmlinuz-5.15.0-25-generic in rpool/ROOT/ubuntu_lva06t
Found initrd image: initrd.img-5.15.0-25-generic in rpool/ROOT/ubuntu_lva06t
/etc/grub.d/10_linux_zfs: 404: .: cannot open /tmp/zfsmnt.Z7nWCz/etc/os-release: No such file
-----------------

$ zfs get mountpoint persist/backup/mule-zfs/ROOT/default
NAME PROPERTY VALUE SOURCE
persist/backup/mule-zfs/ROOT/default mountpoint / local
-----------------

# oops! This fixed it:
$ sudo zfs set mountpoint=zmule persist/backup/mule-zfs/ROOT/default
$ zfs get mountpoint persist/backup/mule-zfs/ROOT/default
NAME PROPERTY VALUE SOURCE
persist/backup/mule-zfs/ROOT/default mountpoint /zmule local