21.10 beta, errors in 10-linux and 10_linux_zfs

[Mason Loring Bliss]

In a custom install of Ubuntu 21.10 beta, both hardware and VM installs
suffer from a bug in the grub.d/10_linux and 10_linux_zfs scripts. (For
comparison, Debian Bullseye, running a similar version of grub, doesn't
have this issue.)

Unique to Ubuntu, there's this block in 10_linux:

    xzfs)
        # We have a more specialized ZFS handler, with multiple system in
        # 10_linux_zfs.
        if [ -e "`dirname $(readlink -f $0)`/10_linux_zfs" ]; then
          exit 0
        fi

This looks at the root filesystem, and if it's ZFS, it shunts kernel
discovery and entry population off to 10_linux_zfs. This subsequent
script assumes that the default/automated Ubuntu ZFS layout is in
effect, and if it's not, the end result is that 10_linux doesn't add an
entry because there is ZFS present, and 10_linux_zfs doesn't add a
kernel stanza either, evidently because /boot isn't in a pool. (I
haven't tracked the logic in 10_linux_zfs fully but given time pressure
I wanted to get this bug in so someone could look at it.) With this
combination of events, the resulting grub.cfg has no kernel stanzas at
all, leaving the user at a grub> prompt. Manual configuration and
booting from the prompt works from this point, but it's obviously not
ideal.

In testing, commenting out the "exit" in the code block noted above
resulted in correct stanzas being generated, in this case with /boot
being on ext4 atop MD-RAID1. Rather than exiting if the root is on ZFS,
correct behaviour would occur in more cases if we check for /boot being
on ZFS or not. A simple check (untested) might be:

        if ! grep -q '[[:space:]]/boot[[:space:]]' /etc/fstab; then
          exit 0
        fi

This doesn't check for 10_linux_zfs existing, but that check is perhaps
redundant given that the file is installed alongside 10_linux and thus
will always exist, as packaged. This instead checks to see if there's a
/boot in /etc/fstab, which if present should indicate that /boot is not
going to be found on a ZFS dataset. (Certainly, traditional filesystems
can exist on zvols and legacy-mount datasets exist, and both of these
can appear in fstab, but neither of those is possible for a working
/boot.)

[Didier Roche-Tolomelli (didrocks)]

Thanks for reporting this bug and help making ubuntu better.

10_linux_zfs is supposed to be able to track ext4 /boot and handling it correctly (via looking at fstab and others).

What would be interesting is to print your fstab, if we miss anything by any chance. Also, can you set -x on top of 10_linux_zfs and print the logs here?

[Mason Loring Bliss (y-mason)]

Sure. This is a mode I've been using lately where I'm using legacy
mountpoints on datasets out of fstab. I suspect this would do the same
thing with a traditional inherited mount hierarchy.

# cat /etc/fstab
tank/zroot / zfs defaults 0 0
tank/home /home zfs defaults 0 0
tank/usr/src /usr/src zfs defaults 0 0
tank/var/mail /var/mail zfs defaults 0 0
tank/home/mason /home/mason zfs defaults 0 0
tank/var/log /var/log zfs defaults 0 0
tank/tmp /tmp zfs defaults 0 0
/dev/md0 /boot ext4 defaults 0 1
/dev/sda2 /boot/efi0 vfat defaults 0 1
/dev/sdb2 /boot/efi1 vfat defaults 0 1
/dev/mapper/swap none swap sw 0 0

# update-grub
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
+ set -e
+ prefix=/usr
+ datarootdir=/usr/share
+ ubuntu_recovery=1
+ quiet_boot=1
+ quick_boot=1
+ gfxpayload_dynamic=1
+ vt_handoff=1
+ . /usr/share/grub/grub-mkconfig_lib
+ prefix=/usr
+ exec_prefix=/usr
+ datarootdir=/usr/share
+ datadir=/usr/share
+ bindir=/usr/bin
+ sbindir=/usr/sbin
+ [ x/usr/share/grub = x ]
+ test x = x
+ grub_probe=/usr/sbin/grub-probe
+ test x = x
+ grub_file=/usr/bin/grub-file
+ test x = x
+ grub_mkrelpath=/usr/bin/grub-mkrelpath
+ which gettext
+ :
+ grub_tab=
+ export TEXTDOMAIN=grub
+ export TEXTDOMAINDIR=/usr/share/locale
+ set -u
+ which zfs
+
+ imported_pools=
+ mktemp -d /tmp/zfsmnt.XXXXXX
+ MNTDIR=/tmp/zfsmnt.TssEDF
+ mktemp -d /tmp/zfstmp.XXXXXX
+ ZFSTMP=/tmp/zfstmp.v6r2Ln
+ uname -m
+ machine=x86_64
+ GENKERNEL_ARCH=x86_64
+ RC=0
+ trap on_exit EXIT INT QUIT ABRT PIPE TERM
+ GRUB_LINUX_ZFS_TEST=
+ import_pools
+ zpool list
+ awk {if (NR>1) print $1}
+ local initial_pools=tank
+ local all_pools=
+ local imported_pools=
+ local err=
+ set +e
+ zpool import -f -a -o cachefile=none -o readonly=on -N
+ err=no pools available to import
+ [ 0 -ne 0 ]
+ set -e
+ zpool list
+ awk {if (NR>1) print $1}
+ all_pools=tank
+ echo tank
+ grep -wq tank
+ continue
+ echo
+ imported_pools=
+ bootlist /tmp/zfsmnt.TssEDF
+ local mntdir=/tmp/zfsmnt.TssEDF
+ local boot_list=
+ get_root_datasets
+ zpool list
+ awk {if (NR>1) print $1}
+ local pools=tank
+ zpool get -H altroot tank
+ awk {print $3}
+ local rel_pool_root=-
+ [ - = - ]
+ rel_pool_root=/
+ zfs list -H -o name,canmount,mountpoint -t filesystem
+ awk {print $1}
+ grep -E ^tank(\s|/[[:print:]]*\s)(on|noauto)\s/$
+ echo
+ boot_list=
+ generate_grub_menu_metadata
+ local bootlist=
+ get_machines_sorted
+ local bootlist=
+ echo
+ + awk {print $3}
sort -u
+ local machineids=
+ + sort -nr
awk {print $2}
+ menu_metadata=
+ generate_grub_menu
+ local menu_metadata=
+ local last_section=
+ local main_dataset_name=
+ local main_dataset=
+ local have_zsys=
+ [ -z ]
+ return
+ grub_menu=
+ [ -n ]
+ on_exit
+ mountpoint -q /tmp/zfsmnt.TssEDF
+ true
+ rmdir /tmp/zfsmnt.TssEDF
+ rm -rf /tmp/zfstmp.v6r2Ln
+ exit 0
done

To reiterate, though, 10_linux does the right thing if it's allowed to
probe for entries. Maybe a good solution would be to let it, and save a
few lines and some complexity in 10_linux_zfs.

[Mason Loring Bliss (y-mason)]

A quick test shows the issue not cropping up if I use an install with
inherited mountpoints in a more standard hierarchy. I haven't checked
out what's different.

tank/var/log /var/log zfs defaults 0 0
tank/tmp /tmp zfs defaults 0 0
/dev/md0 /boot ext4 defaults 0 1
/dev/mapper/swap none swap sw 0 0

Realistically, this probably makes it a not-super-high-priority bug
given how rare legacy mountpoints are and the fact that probably 25%
of the world population using them (me) has a workaround.

Ironically, the issue noted in 1945873 didn't show up with this build
either, for reasons that I haven't yet sussed out.

[Didier Roche-Tolomelli (didrocks)]

Thanks for getting back.

Indeed, as I told you, manual mount on /boot is supported (and backed by a large testsuite).

I’m puzzled with your failing case: as you can see, we couldn’t import any pools in the script:
+ zpool import -f -a -o cachefile=none -o readonly=on -N
+ err=no pools available to import
Which is then why we couldn’t scan anything… it would be interesting to understand why this is the case…

[Mason Loring Bliss (y-mason)]

Didier,

That part didn't strike me as exceptional because the pool's already
mounted, since we're running update-grub from the running system. It's
not available to be listed or imported again.

I'll want to read 10_linux_zfs in depth to see what it's doing, but if
it's depending on a list to come back from 'zpool import' it's not
going to get one in circumstances where the pool's already imported,
unless there's some critical concept confusing me.

[Didier Roche-Tolomelli (didrocks)]

Actually, I got caught by "imported_pools_ vs "all_pools". So yeah, this is expected (this list is to cleanup, removing any temporary imported pool by the script to list all systems available on grub).

The issues is below, (I don’t have time to debug it fully right now), but in this part of set -x I believe:
+ zfs list -H -o name,canmount,mountpoint -t filesystem
+ awk {print $1}
+ grep -E ^tank(\s|/[[:print:]]*\s)(on|noauto)\s/$
+ echo
+ boot_list=
+ generate_grub_menu_metadata
+ local bootlist=
+ get_machines_sorted
+ local bootlist=
+ echo
+ + awk {print $3}
sort -u

which is triggered by that:
    for dataset in $(get_root_datasets); do
        # get information from current root dataset
        boot_list="${boot_list}$(get_dataset_info "${dataset}" ${mntdir})\n"

        # get information from snapshots of this root dataset
        snapshots="$(zfs list -H -o name -t snapshot "${dataset}"|while read snapshot_dataset; do
            get_dataset_info "${snapshot_dataset}" ${mntdir}
        done)"
        [ -n "${snapshots}" ] && boot_list="${boot_list}${snapshots}\n"
    done

[Mason Loring Bliss (y-mason)]

I haven't had a chance to dig deeper, but I just noticed this same issue
in Focal Fossa.

If I get a chance to debug this I'll submit a patch here. I might get a
chance over the next week, during Thanksgiving break.

[Mason Loring Bliss (y-mason)]

A quick update, I might get a chance to dig into this again. I recently
noted that the issue persists in 22.04, via the beta installer.