ZFS initramfs mounts dataset explicitly set not to be mounted, causing boot process to fail

[Sam Van den Eynde]

Per https://github.com/zfsonlinux/pkg-zfs/issues/221: the initramfs zfs script might overrule canmount and mountpoint options for a dataset, causing other mount operations and with them the boot process to fail.

Experienced this with Ubuntu Zesty. Xenial seems to ship with a different zfs script for the initrd.

Work around when it happens: unmount the dataset that should not be mounted, and exit the initramfs rescue prompt to resume booting.

[Richard Laager (rlaager)]

On 04/22/2017 12:36 PM, Sam Van den Eynde wrote:
> Experienced this with Ubuntu Zesty. Xenial seems to ship with a
> different zfs script for the initrd.

Who completely replaced the zfs-initramfs script?

Was there a particular reason for this massive change, and was it
discussed anywhere?

This change has caused many regressions, of varying importance.

I realize I don't "own" this code, but as someone who is very involved
in root-on-ZFS installs in Ubuntu, I'm quite annoyed that nobody asked
me if this was a good idea or mentioned it was happening so I might
review it and offer some feedback.

Aside from the issue that Sam Van den Eynde is having [0], this also
reverted a number of changes I made, including but not limited to: the
support I added for rootdelay= and the code to set elevator=noop [1].

[0] https://bugs.launchpad.net/bugs/1685528
    https://github.com/zfsonlinux/pkg-zfs/issues/221

[1] https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1550301

--
Richard

[Petter Reinholdtsen (pere-hungry)]

[Richard Laager]
> Who completely replaced the zfs-initramfs script?

You can find out who commited what in the Debian package by looking in
the package maintenance git repository available from
<URL: http://anonscm.debian.org/gitweb/?p=pkg-zfsonlinux/zfs.git >.

There is also the <email address hidden> mailing
list and the #debian-zfs IRC channel on irc.debian.org. I believe we
would all be very happy to see more people there. This is where the
package maintenance is being coordinated and discussed in Debian.

> Was there a particular reason for this massive change, and was it
> discussed anywhere?

As far as I can see from the git repository, there was no 'change',
there was an 'addition', where to fix BTS report #848157 a file was
added to /usr/share/initramfs-tools/conf.d/.

--
Happy hacking
Petter Reinholdtsen

[Richard Laager (rlaager)]

On 04/24/2017 12:02 AM, Petter Reinholdtsen wrote:
> [Richard Laager]
>> Who completely replaced the zfs-initramfs script?
>
> You can find out who commited what in the Debian package by looking in
> the package maintenance git repository available from
> <URL: http://anonscm.debian.org/gitweb/?p=pkg-zfsonlinux/zfs.git >.

I will take a look.

> There is also the <email address hidden> mailing
> list and the #debian-zfs IRC channel on irc.debian.org. I believe we
> would all be very happy to see more people there. This is where the
> package maintenance is being coordinated and discussed in Debian.

I have been subscribed to the mailing list for some time. I have joined
the IRC room now.

Are you saying it was discussed in one of these places?

>> Was there a particular reason for this massive change, and was it
>> discussed anywhere?
>
> As far as I can see from the git repository, there was no 'change',
> there was an 'addition', where to fix BTS report #848157 a file was
> added to /usr/share/initramfs-tools/conf.d/.

/usr/share/initramfs-tools/scripts/zfs was significantly changed in
Zesty from what it is in Xenial:
1 file changed, 936 insertions(+), 89 deletions(-)

--
Richard

[Petter Reinholdtsen (pere-hungry)]

[Richard Laager]
> I will take a look.

If you are not already aware of it, it is useful to know about the
summary page and related links,
<URL: https://qa.debian.org/developer.php?email=pkg-zfsonlinux-devel%40lists.alioth.debian.org >.

> I have been subscribed to the mailing list for some time. I have joined
> the IRC room now.

Welcome. :)

> Are you saying it was discussed in one of these places?

At least I am happy with the amount of information flowing in these
channels. All commits are automatically send to the list, for those
with time and interest to review, and all bug reports and discussions
are sent there too. But most important, everyone in the group seem to
welcome contributions from interested and skilled people. The team is
short on man-power, as is most teams of course, but everyone is doing as
good as we can.

My main problem is personal, I lack the required spare time to
contribute technically, and so far has mostly helped by guiding the
package through the NEW queue and commiting a few translations. :)

> /usr/share/initramfs-tools/scripts/zfs was significantly changed in
> Zesty from what it is in Xenial:
> 1 file changed, 936 insertions(+), 89 deletions(-)

Aha. I misunderstood what to look for in git. I do not know anything
about that file, unfortunately. :)

--
Happy hacking
Petter Reinholdtsen

[Aron Xu (happyaron)]

> On Apr 24, 2017, at 15:09, Petter Reinholdtsen <email address hidden> wrote:
>
> [Richard Laager]
>> I will take a look.
>
> If you are not already aware of it, it is useful to know about the
> summary page and related links,
> <URL: https://qa.debian.org/developer.php?email=pkg-zfsonlinux-devel%40lists.alioth.debian.org >.
>
>> I have been subscribed to the mailing list for some time. I have joined
>> the IRC room now.
>
> Welcome. :)
>
>> Are you saying it was discussed in one of these places?
>
> At least I am happy with the amount of information flowing in these
> channels. All commits are automatically send to the list, for those
> with time and interest to review, and all bug reports and discussions
> are sent there too. But most important, everyone in the group seem to
> welcome contributions from interested and skilled people. The team is
> short on man-power, as is most teams of course, but everyone is doing as
> good as we can.
>
> My main problem is personal, I lack the required spare time to
> contribute technically, and so far has mostly helped by guiding the
> package through the NEW queue and commiting a few translations. :)
>
>> /usr/share/initramfs-tools/scripts/zfs was significantly changed in
>> Zesty from what it is in Xenial:
>> 1 file changed, 936 insertions(+), 89 deletions(-)
>
> Aha. I misunderstood what to look for in git. I do not know anything
> about that file, unfortunately. :)
>

I guess the change was led by the Debian sync, for zesty the packages are synced and derived from the Debian one, while previous packages aren't.

I don't think there's extensive use of those Debian packages for ZFS as rootfs at the moment, while other common cases were tested to be working. So if there's trouble led by those packages let's work on it...

Regards,
Aron

[Jens Elkner (jelmd)]

It should not restrict import to hardcoded /dev and /dev/disk/by-id - see https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1688890

[Sam Van den Eynde (samvde)]

Can't we upstream the Xenial zfs script instead of fixing the "new" one? It supports the same syntax, and does what it needs to do? And it does not try to mount datasets I explicitly configured not to be mountable ;)

Seems like the faster solution, no?

[Richard Laager (rlaager)]

I suspect that wouldn't work, for one reason or another. The upstream one has more features, probably.

I'd rather just keep this Debian-specific. The initramfs script is likely to have distro specific code. I don't see the idea of one unified script working out well.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christopher J. Ruwe (cruwe)]

Are there any recommendations how to proceed with the EOL for yakkety set? I use zfs clones as inpromptu boot environments when updating and use ZFS datasets for docker and lxd, so I have a fair amount of 'canmount=none's and 'mountpoint=legacy's. I am reluctant to just c&p the script from yakkety over to zesty, but seriously short of better ideas.

Cheers!

[Richard Laager (rlaager)]

Copying the script is probably fine for now. I still intend to look at this, hopefully in the next month or so. It's been relatively low on my list, since LTS releases are my main priority.

[Colin Ian King (colin-king)]

Hi Richard, I'll be happy to upload this fixes into the non-LTS releases where this script is missing.

[Richard Laager (rlaager)]

samvde, can you provide your `zfs list` output? The script seems designed to only import filesystems *below* the filesystem that is the root filesystem. In the typical case, the root filesystem is something like rpool/ROOT/ubuntu. There typically shouldn't be children of rpool/ROOT/ubuntu.

[Richard Laager (rlaager)]

I fixed this upstream, which was released in 0.7.4. Bionic has 0.7.5.

[Colin Ian King (colin-king)]

Fix released in Bionic.