zfs services fail on firstboot if zfs-utils is integrated into the deployment image

Bug #1607920 reported by Dave Chiluk on 2016-07-29
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
sysvinit (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
zfs-linux (Ubuntu)
Medium
Eric Desrochers
Xenial
Medium
Eric Desrochers

Bug Description

[Impact]

 * zfs services fail on firstboot if zfs-utils is integrated into the deployment image.

 * Output from systemd -
sudo systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● zfs-import-scan.service loaded failed failed Import ZFS pools by device scanning
● zfs-mount.service loaded failed failed Mount ZFS filesystems

 * This is particularly frustrating for users who use automated monitoring as it means virtual machines must always be restarted before showing as clean.

 * This failure is due to zfs services starting up before /etc/mtab has a chance to be symlinked to /proc/mounts.

[Test Case]

 1. Grab a stock xenial image, and unpack it and add zfs-utils to it. Repack it.
 2. Boot machine
 3. Check systemctl --failed.

[Regression Potential]

 * none expected, patch has been intensively tested by the upsteam zfs test script suite.

 * This is a upstream commit merge in 0.7.0.

 * A ubuntu package has been tested (including the upstream commit) by a user of the community facing this bug, and confirmed it addresses the problem (see comment #7).

[Other Info]

* The "reading" is redirected to /proc/self/mounts.
The writing to /etc/mtab. Some distros still need that. The current hope is to replace the writing (and maybe reading) with libmount, in a second phase.

 * Upstream commit : https://github.com/zfsonlinux/zfs/commit/792517389fad5c495a2738b61c2e9c65dedaaa9a

 * Upstream bug: https://github.com/zfsonlinux/zfs/issues/4680

 * Debian bug : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839071

Dave Chiluk (chiluk) wrote :

Personally I think we should modify MNTTAB in zfs-linux to point to /proc/self/mounts. This may not be completely straightforward *(there are a number of places that point to /etc/mnttab directly, or have logic around it and /proc/mounts), but it should be doable. I like this solution most because it doesn't require us to increase the number of dependencies in the zfs systemd scripts.

Steve Langasek (vorlon) wrote :

No reason to fix this via sysvinit, which should be considered deprecated at this point.

Changed in sysvinit (Ubuntu):
status: New → Won't Fix
Richard Laager (rlaager) wrote :

chiluk is looking to work on this for upstream. I might jump in too. https://github.com/zfsonlinux/zfs/issues/4680

Martin Pitt (pitti) wrote :

Confirmed. /etc/mtab is dead, dead, dead, and an ill-conceived and broken concept. Don't ever use it please ☺

Eric Desrochers (slashd) on 2016-09-13
Changed in zfs-linux (Ubuntu):
assignee: nobody → Eric Desrochers (slashd)
importance: Undecided → Medium
status: New → In Progress
tags: added: sts
Eric Desrochers (slashd) wrote :

I jointly worked with the zfsonlinux maintainer on a patch that will be merged in 0.7.0 milestone
https://github.com/zfsonlinux/zfs/milestone/11?closed=1

#4680, has been tackled in two parts. the first part (patch that will be merge in 0.7.0), change from /etc/mtab to /proc/self/mounts, and keep the existing mtab update functions alone in the mount helper.

Eventually, a second part would be for ZoL contributor/maintainers to do the integration with libmount. libmount didn't exist on Linux when the mount helper code was originally written. That could allow ZoL to drop their custom code.

Scott Emmons (lscotte) wrote :

I tested this using my reproduction method in AWS EC2 with a test package/PPA provided by Eric@Ubuntu (slashd) containing the commit referenced in comment #6.

All looks good now - systemd units for ZFS no longer fail. Thanks for moving this along!

Eric Desrochers (slashd) wrote :

Thanks scotte for the feedbacks, I will now start the SRU process.

Eric

description: updated
Eric Desrochers (slashd) on 2016-09-28
description: updated
tags: added: sts-sponsor sts-sru
description: updated
Eric Desrochers (slashd) wrote :

debdiff for Xenial (16.04 LTS)

description: updated
Eric Desrochers (slashd) wrote :

debdiff for devel release : Yakkety (16.10)

Changed in zfs-linux (Ubuntu Xenial):
assignee: nobody → Eric Desrochers (slashd)
status: New → In Progress
importance: Undecided → Medium
Changed in sysvinit (Ubuntu Xenial):
status: New → Won't Fix
Eric Desrochers (slashd) wrote :
description: updated
Eric Desrochers (slashd) wrote :

@scotte,

After some verification, I found out that cking uploaded a patch on Sept 26 on Xenial for another zfs-linux bug[1] using version "0.6.5.6-0ubuntu13", which interfere with my debdiff.

"pull-lp-source" and "rmadison" didn't shown the existence of "0.6.5.6-0ubuntu13".
I found it by looking in the upload queue : https://launchpad.net/ubuntu/xenial/+queue?queue_state=1

With that being said, I will need to wait for the current zfs-linux SRU on Xenial to be completed and then I'll re-do the debdiff in order to increment the version for this bug to accordingly "0.6.5.6-0ubuntu14".

The Xenial SRU might take a little bit more time than expected to land in stable (-update), but
the devel release debdiff (Yakkety) is still fine.

I'll keep you posted.

[1] - https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1627909

Eric
Eric

Eric Desrochers (slashd) wrote :

I removed the Xenial debdiff attached to this bug and I will re-do it when current zfs-linux SRU for Xenial by cking will be completed.

Until then, the Yakkety debdiff is still fine and can be sponsored.

Eric

Eric Desrochers (slashd) wrote :

I pulled the source code per version and rebase my debdiff on top of "0.6.5.6-0ubuntu13".

Will attach the new debdiff in a couple of minutes.

Eric

Eric Desrochers (slashd) wrote :

Rebased debdiff for Xenial (16.04 LTS)

Eric Desrochers (slashd) on 2016-09-29
description: updated
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.6.5.8-0ubuntu3

---------------
zfs-linux (0.6.5.8-0ubuntu3) yakkety; urgency=medium

  [ Eric Desrochers ]
  * Fix zfs services to fail on first boot due to zfs services starting up
    before /etc/mtab has a chance to be symlinked to /proc/mounts. (LP: #1607920)
    (upstream commit 792517389fad5c495a2738b61c2e9c65dedaaa9a)

 -- Eric Desrochers <email address hidden> Thu, 29 Sep 2016 16:13:38 +0200

Changed in zfs-linux (Ubuntu):
status: In Progress → Fix Released
Eric Desrochers (slashd) wrote :

As per irc conversation with apw, I'm merging both LPs #1607920 & #1628279 into one debdiff.

Andy Whitcroft (apw) wrote :

@Eric -- I am concerned about the effect of some of these changes on translations. For example the below will make the format string a new translatable and therefore will not be translated for non english locales. If we simply replaces MNTTAB with "/etc/mtab" the translatable string would be unchanged:

+ (void) fprintf(stderr, gettext(
+- "filesystem '%s' was mounted, but %s "
++ "filesystem '%s' was mounted, but /etc/mtab "
+ "could not be opened due to error %d\n"),
+- dataset, MNTTAB, errno);
++ dataset, errno);

There are a few like that.

Also as you are changing MNTTAB is there any reason not to be using that here?

+- (void) fprintf(stderr, gettext("warning: %s not in mtab\n"),
+- path);
++ (void) fprintf(stderr, gettext("warning: %s not in"
++ "/proc/self/mounts\n"), path);

Richard Laager (rlaager) wrote :

Does ZFS actually have any translations? There are no .po files in the source and no .mo files in the binary package.

Eric Desrochers (slashd) wrote :

@apw

To answer your question:
"Also as you are changing MNTTAB is there any reason not to be using that here?"

The "reading" is redirected to /proc/self/mounts and the "writing" to /etc/mtab, but if it doesn't exist, that's fine. ZFS just won't update /etc/mtab.

Ideally, ZFS should be integrated with libmount in a second phase (ZoL maintainers are aware that eventually a 2nd is needed to integrate libmount), until then /proc/self/mounts is a viable solution to address issues cause by mtab.

Eric

Eric Desrochers (slashd) wrote :

@apw

With regard to the translation ....

The commit has been intensively tested in different scenarios/env/... and of course with the "ZFS Test Suite"[1] which include functionality, perf, and stress tests.

All the tests were reported as successful.

I hope this answer your questions and addresses your concern.

Let rlaager and I know if you need more clarifications.

[1] - https://github.com/zfsonlinux/zfs/tree/master/tests

Eric

Hello Dave, or anyone else affected,

Accepted zfs-linux into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/0.6.5.6-0ubuntu14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in zfs-linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Scott Emmons (lscotte) wrote :

I tested again with xenial-proposed and confirmed that everything looks good as far as systemd no longer failing ZFS units due to no /etc/mtab.

ii libzfs2linux 0.6.5.6-0ubuntu14 amd64 Native OpenZFS filesystem library for Linux
ii zfs-dkms 0.6.5.6-0ubuntu14 amd64 Native OpenZFS filesystem kernel modules for Linux
ii zfs-doc 0.6.5.6-0ubuntu14 all Native OpenZFS filesystem documentation and examples.
ii zfs-initramfs 0.6.5.6-0ubuntu14 all Native OpenZFS root filesystem capabilities for Linux
ii zfs-zed 0.6.5.6-0ubuntu14 amd64 OpenZFS Event Daemon (zed)
ii zfsutils-linux 0.6.5.6-0ubuntu14 amd64 Native OpenZFS management utilities for Linux

# systemctl --failed
0 loaded units listed. [...]

Eric Desrochers (slashd) wrote :

Thanks scotte for the feedbacks.

tags: added: verification-done
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.6.5.6-0ubuntu14

---------------
zfs-linux (0.6.5.6-0ubuntu14) xenial; urgency=medium

  * Fix zfs services to fail on first boot due to zfs services starting up
    before /etc/mtab has a chance to be symlinked to /proc/mounts. (LP: #1607920)
    (upstream commit 792517389fad5c495a2738b61c2e9c65dedaaa9a)

  * Symlink utilities in /usr/sbin to remove suffix as per policy 10.4 Scripts (LP: #1628279)
    Users can rely on the new names, but relying on the old names won't break before
    release upgrade.

 -- Eric Desrochers <email address hidden> Tue, 04 Oct 2016 15:07:52 +0200

Changed in zfs-linux (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for zfs-linux has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Eric Desrochers (slashd) wrote :

Hi scotte,

The fix is now released as shown in comment#25 & #26

Eric

Scott Emmons (lscotte) wrote :

Thanks Eric, we will remove our local workaround.

Louis Bouchard (louis) on 2016-10-20
tags: removed: sts-sponsor
Louis Bouchard (louis) on 2016-11-09
tags: removed: sts-sru
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.