update-grub 10_linux_zfs fails when /usr is a separate filesystem

[Cory Coager]

Description: Ubuntu 20.04.1 LTS
Release: 20.04
grub-common: 2.04-1ubuntu26.4

I expect grub-update to create grub.cfg with Linux entries for ZFS. Instead I am left with an empty list and an unbootable OS.

The output from update-grub:
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Sourcing file `/etc/default/grub.d/kdump-tools.cfg'
Generating grub configuration file ...
/etc/grub.d/10_linux_zfs: 404: .: Can't open /tmp/zfsmnt.stGSQC/etc/os-release
Found memtest86+ image: /memtest86+.elf
Found memtest86+ image: /memtest86+.bin
done

This is failing because I have a separate ZFS filesystem for /usr. The 10_linux_zfs script is gathering information from /etc/os-release which is a symlink to ../usr/lib/os-release.

The first issue I see in 10_linux_zfs is under function get_dataset_info() at line 382:
mount -o noatime,zfsutil -t zfs "${base_dataset}" "${mntdir}"

Since /usr is under a separate filesystem, the symlink would be broken and the script fails. This should mount /usr as well if it is a separate filesystem.

The next issue I see is when the script iterates the zsys snapshots at line 514:
for snapshot_dataset in $(zfs list -H -o name -t snapshot "${dataset}"); do
    boot_list="${boot_list}$(get_dataset_info ${snapshot_dataset} ${mntdir})\n"
done

You would get a similar error message about os-release.
/etc/grub.d/10_linux_zfs: 410: .: Can't open /.zfs/snapshot/autozsys_oiimb5/etc/os-release

Again, the script should consider reading the /usr snapshot as well if it is a separate filesystem.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dimitri John Ledkov (xnox)]

So, /etc/os-release is an obsolete file location, which is normally a symlink to /usr/lib/os-release.

When mounting snapshots, the grub script doesn't parse /etc/fstab; to mount the snapshots of where the /usr is, and mount that.....

However, I don't know how having /usr as a subvolumen (if that is the right term) even supported or usable with zsys.

What do you mean by /usr is a separate filesystem? which filesystem is it?

We only support zsys when everything is managed by it.

[Cory Coager (ccoager)]

/usr is a separate ZFS file system, separate from /. My system boots and operates normally. The only issue is with the grub script, /etc/grub.d/10_linux_zfs. I had to do some hacking of this script for it to parse the Linux images properly.

If you think /usr is not supported as a separate ZFS file system, can you point me to the documentation that states so?

[Julian Andres Klode (juliank)]

I think the question is whether there is documentation that states that zsys supports split /usr, and the answer for that is no.

Also, um, the installer does not support installing like this - there's only one experimental ZFS option, on desktop anyway.

So this bug in essence will answer the question whether this is something we want to support or not.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[KAMI (kami911)]

I can confirm, I also has this bug:
/etc/grub.d/10_linux_zfs: 404: .: Can't open /tmp/zfsmnt.stGSQC/etc/os-release

I mounted the root partition under /target when I hit into this problem. I simply modified the 404 line to use /etc.

[Tessa (unit3)]

I think it's an important use case to support, since it's more than likely going to be needed once Ubuntu supports ZFS on server installs. the zfs tools and scripts should definitely be bulletproof'd to handle any zfs fs configuration, especially around something like this where it should be easy to fix.

[Shayne Underwood (libertyspike138)]

I too am experiencing this on 22.04 but don't quite understand how to resolve it. https://ubuntuforums.org/showthread.php?t=2486982

[Shayne Underwood (libertyspike138)]

I have resolved the issue in my case. This is a server and besides Ubuntu I also have TrueNAS installed to another disk as a backup to get to my files if for some reason I can't get in. What I have discovered is that anytime the disk containing TrueNAS is connected the GRUB scripts for ZFS take a dump and leave the "/etc/grub.d/10_linux_zfs: 404: .: cannot open /tmp/zfsmnt.dCD7rF/etc/os-release: No such file " error. I initially thought that maybe grub-customizer had botched it so I backed up my home and reinstalled the OS leaving out grub-customizer and manually editing my GRUB. I was having a black screen GRUB issue before enabling console mode and when I updated GRUB I noticed the script took a dump again so I thought to eject the disk with TrueNAS and try to update GRUB again so I'm not left in an unbootable state and it solved the issue. So now I have just disabled the port for my TrueNAS backup inside the BIOS and all seems well. The reason that rpool was missing in the grub.cfg is because it was adding the entries under linux instead of linux_zfs scripts. If anybody else has this issue pay attention to the scripts in /etc/grub.d/ to find out where your entries are being generated and disable or remove any other ZFS / Solaris type disks that may be conflicting. Other than that I'm not sure how else to leave the TrueNAS drive enabled and not have GRUB take a dump but in my case it is no big deal considering it is just a backup OS that I can leave disabled.