network file systems in FSTAB no longer mount at boot with NetworkManager

[lnxusr]

[Impact]

 * This breaks NFS mounts coming up reliably. In some cases this can be quite intermittent.

[Test Case]

 1. Add a mountpoint to an NFS share by name to fstab.
 2. See that it mounts with the mount command.
 3. Reboot 5+ times, note that it doesn't come up all (or the majority of the time).

[Regression Potential]

 * There could be a regression for the same reason it's blocked in Debian, because of dependency issues with packages not converted to systemd that want to run /etc/rcS.d/.

You can check for this by looking for breaking/disable services during bootup. Ubuntu is in a much better state than Debian, but there is a possibility we missed something.

Otherwise, this will increase boot time, but when the user logs the networking will be more likely to be ready.

-Previous description-
After a fresh install of 15.10, nfs shares no longer mount on boot. I'm using the same line to mount as I did in 14.04 prior:

<server>:/share /mnt/share ntfs4 _netdev, auto 0 0

This line worked just fine in 14.04, and 14.10 on my laptop, to mount the shares at boot.

Manual mounting after boot works fine. Systemctl shows a name resolution failure (see below)

lsb_release -rd
Description: Ubuntu 15.10
Release: 15.10

lsb_release -rd
Description: Ubuntu 15.10
Release: 15.10
bjwest@razorback:~$ apt-cache policy nfs-common
nfs-common:
  Installed: 1:1.2.8-9ubuntu10
  Candidate: 1:1.2.8-9ubuntu10
  Version table:

 *** 1:1.2.8-9ubuntu10 0
        500 http://us.archive.ubuntu.com/ubuntu/ wily/main amd64 Packages
        100 /var/lib/dpkg/status

systemctl status mnt-share.mount
● mnt-share.mount - /mnt/share
   Loaded: loaded (/etc/fstab)
   Active: failed (Result: exit-code) since Wed 2015-11-11 18:58:13 CST; 2min 15s ago
    Where: /mnt/share
     What: hog:/share
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)
  Process: 731 ExecMount=/bin/mount hog:/share /mnt/share -t nfs4 -o _netdev (code=exited, status=32)

Nov 11 18:58:13 razorback systemd[1]: Mounting /mnt/share...
Nov 11 18:58:13 razorback mount[731]: mount.nfs4: Failed to resolve server hog: Temporary failure in name resolution
Nov 11 18:58:13 razorback systemd[1]: mnt-share.mount: Mount process exited, code=exited status=32
Nov 11 18:58:13 razorback systemd[1]: Failed to mount /mnt/share.
Nov 11 18:58:13 razorback systemd[1]: mnt-share.mount: Unit entered failed state.

[Steve Langasek (vorlon)]

Opening a task on systemd as well for this.

When called early in boot, mount.nfs may fail because of any number of problems with the network. In this case, it appears it's being called by systemd before /etc/resolv.conf has been set up, resulting in a temporary DNS failure. There is no way for mount.nfs to work when invoked at this point in the boot.

So either systemd needs to more finely control the timing of the mount call so that it definitely happens only after the network is up (including /etc/resolv.conf configuration), or it needs to handle retrying the mount in the case of a temporary failure. (The latter is what mountall did in Ubuntu 14.10 and earlier.)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Doug Rohm (drohm)]

I'm seeing this behavior as well on a fresh install of Xubuntu 15.10.

http://askubuntu.com/questions/689515/mount-network-share-automatically-on-startup

[ganrald (ganrald)]

This bug has not been tagged before as 'wily' and unfortunately I have created a duplicate bug report of this one, not being able to find it earlier.

A few more observations from my original bug report:

It looks like the network is not up when the mounting is done however systemd dependencies as well as unit files generated from /etc/fstab seem to be OK

$ tail -2 /etc/fstab
# Media Library
disk.lan:/volume1/Library /media/Library nfs rw,async,auto,nfsvers=3 0 0

$ systemd-analyze dot media-Library.mount
digraph systemd {
        "remote-fs.target"->"media-Library.mount" [color="green"];
        "remote-fs.target"->"media-Library.mount" [color="black"];
        "media-Library.mount"->"network-online.target" [color="green"];
        "media-Library.mount"->"remote-fs-pre.target" [color="green"];
        "media-Library.mount"->"systemd-journald.socket" [color="green"];
        "media-Library.mount"->"system.slice" [color="green"];
        "media-Library.mount"->"network.target" [color="green"];
        "media-Library.mount"->"-.mount" [color="green"];
        "media-Library.mount"->"-.mount" [color="black"];
        "media-Library.mount"->"system.slice" [color="grey66"];
        "media-Library.mount"->"network-online.target" [color="grey66"];
        "media-Library.mount"->"umount.target" [color="red"];
        "umount.target"->"media-Library.mount" [color="green"];
        "umount.target"->"media-Library.mount" [color="red"];
}
   Color legend: black = Requires
                 dark blue = Requisite
                 dark grey = Wants
                 red = Conflicts
                 green = After

Despite the correct dependency between "media-Library.mount" and the "network-online.target" looking at the journal log reveals that the network is not yet started when the "network-online.target" is reached.

$ journalctl --boot _COMM=systemd _COMM=NetworkManager
Dez 04 15:16:55 localhost systemd[1]: Starting Hostname Service...
Dez 04 15:16:55 localhost systemd[1]: Started Hostname Service.
Dez 04 15:16:55 localhost NetworkManager[693]: <info> NetworkManager (version 1.0.4) is starting...
Dez 04 15:16:55 localhost NetworkManager[693]: <info> Read config: /etc/NetworkManager/NetworkManager.conf
Dez 04 15:16:55 localhost systemd[1]: Started Network Manager.
Dez 04 15:16:55 localhost NetworkManager[693]: <info> VPN: loaded org.freedesktop.NetworkManager.pptp
Dez 04 15:16:55 localhost systemd[1]: Reached target Network.
Dez 04 15:16:55 localhost NetworkManager[693]: <info> DNS: loaded plugin dnsmasq
Dez 04 15:16:55 localhost systemd[1]: Reached target Network is Online.
Dez 04 15:16:55 localhost systemd[1]: Mounting /media/Library...
Dez 04 15:16:55 localhost systemd[1]: Starting /etc/rc.local Compatibility...
Dez 04 15:16:55 localhost NetworkManager[693]: <info> init!
Dez 04 15:16:55 localhost NetworkManager[693]: <info> update_system_hostname
Dez 04 15:16:55 localhost systemd[1]: Started /etc/rc.local Compatibility.
Dez 04 15:16:55 localhost NetworkManager[693]: <info> interface-parser: parsing file /etc/network/interfaces
Dez 04 15:16:55 localhost systemd[1]: Started Authenticate and Authorize Users to Run Privileged Tasks.
Dez 04 15:16:55 localhos...

[Matt (mattdavisuk)]

I've been able to work around the problem by adding 'x-systemd.automount,' to the list of options on the fstab line.

[Doug Rohm (drohm)]

@mtd0, adding that to my fstab worked as well.

[lnxusr (bjwest)]

Workaround seems good. Adding 'x-systemd.automount' to fstab mounts NFS shares during boot.

[sanford rockowitz (rockowitz)]

The x-systemd.automout option works for me as well running 15.10.

Interestingly, it was commented out in my fstab. I believe I tried that option on 15.04, and it didn't work then.

[luca_ing (luca-ingianni)]

Confirming on Lubuntu 15.10

[Bryan Quigley (bryanquigley)]

I looked at what Fedora was doing. The only difference seems to be network-online waits for Networkmanager to say it's online.
sudo systemctl add-wants network-online.target NetworkManager-wait-online.service

This fixes the issue in my testing. Unfortunately it also makes NetworkManager-wait-online.service the biggest blame for bootup time. Any other ideas?

[Bryan Quigley (bryanquigley)]

To be clear, Fedora does not have this problem in 23 or Rawhide.

[Bryan Quigley (bryanquigley)]

This is not an issue in Debian stretch. Same systemd version, newer network manager version.
Debian does not have anything defined (on fs) in network-online.target. I tried removing all of the network-online.target.wants but that hasn't helped (the only meaningful difference I've found so far)

[Bryan Quigley (bryanquigley)]

That's not quite correct. The mount initially fails but then it appears to try again. In any case it does appear end up mounted in Debian. This also fails on an install without Network-manager on Ubuntu. Will retry over many restarts on Debian just to confirm.

[Bryan Quigley (bryanquigley)]

Currently, my testing has not found a specific package that is the cause (ignore network-manager comment above)
Start with Ubuntu-desktop - it won't mount reliably. Remove packages like unity, still fails...
Start with Ubuntu-server - it works every time. Install ubuntu-desktop meta package, it still works every time!

[Bryan Quigley (bryanquigley)]

Oops - It is related to network-manager.. or more precisely it works fine if you specify to use dhcp in /etc/network/interfaces. If you leave it blank (and network-manager actually does the work) then you have the failures.

[Martin Pitt (pitti)]

So is that a regression from bug 1430280, perhaps this got inadvertently dropped in a merge?

[Bryan Quigley (bryanquigley)]

It's disabled explicitly in the rules.
Also in network-manager.postinst it seems that causes it to be disabled as well in my test.

[Bryan Quigley (bryanquigley)]

Unfortunately my patch only makes it enabled on a fresh install of network-manager (or after a purge). It (the upgrade) tries to make have the services respect if the user changed them starting. Do we care for network-manager's services? If you don't want them you should have the package uninstalled?

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "network-manager_1.0.4-0ubuntu8.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Bryan Quigley (bryanquigley)]

This should do it properly now on upgrade.. Just readd the blurb that says it enables it in postinst that actually disabled it..

[Martin Pitt (pitti)]

Hey Bryan,

thanks for investigating and the patch! Indeed it seems the enabling in debian/rules got reverted in some merge. The patch is good, except that I changed postinst to

- if dpkg --compare-versions "$2" lt-nl "0.9.10.0-4ubuntu10"; then
+ if dpkg --compare-versions "$2" lt-nl "1.0.4-0ubuntu8"; then

i. e. I just bumped the number, but I kept the "purge". We want to reset deb-systemd-helper's brain about the automatic/manual enablement state of NetworkManager-wait-online.service, the actual "enable" call will be automatically generated by dh_installinit/dh_systemd_enable. Sorry, that's a bit complicated..

I committed this updated patch to bzr and uploaded.

[Launchpad Janitor (janitor)]

This bug was fixed in the package network-manager - 1.0.4-0ubuntu8

---------------
network-manager (1.0.4-0ubuntu8) xenial; urgency=medium

  * Fix NetworkManager-wait-online.service so it is enabled so NFS
    and other remote FS work with name resolution. (LP: #1515446)

 -- Bryan Quigley <email address hidden> Thu, 28 Jan 2016 10:45:31 +0100

[Bryan Quigley (bryanquigley)]

Thanks for uploading to dev!

Here is a debdiff for an SRU to wily

[Bryan Quigley (bryanquigley)]

Debdiff from comment #24 is for wily and should fix this.

[Brian Murray (brian-murray)]

Hello lnxusr, or anyone else affected,

Accepted network-manager into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/1.0.4-0ubuntu5.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[lnxusr (bjwest)]

Thanks Brian. I just installed from the proposed repository, removed the '.x-systemd.automount' from fstab and rebooted. All seems to be working well. I may compile and package it up for local install this weekend, I don't want the other proposed packages always in my upgradable listing.

Either way, seems to be working great. Thank you.

[Rolf Anders (rolf-anders)]

I can confirm that the bug is fixed in network-manager 1.0.4-0ubuntu5.3. Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package network-manager - 1.0.4-0ubuntu5.3

---------------
network-manager (1.0.4-0ubuntu5.3) wily; urgency=medium

  * Fix NetworkManager-wait-online.service so it is enabled so NFS
    and other remote FS work with name resolution. (LP: #1515446)

 -- Bryan Quigley <email address hidden> Thu, 03 Mar 2016 16:50:46 +0100

[Chris J Arges (arges)]

The verification of the Stable Release Update for network-manager has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[lnxusr (bjwest)]

This bug is back in 17.04. Network (NFS4) shares are no longer automatically mounted on bootup.

[Christian González (droetker)]

I can confirm that. I updated 16.04 (works) -> 16.10 (works) -> 17.04 (doesn't work any more).
calling "mount -a" mounts the directories without problems.

[Christian González (droetker)]

As workaround you can put "mount -a" into /etc/rc.local just before the "exit 0". This at least works here to get my shares. How can I help to solve this?

[Thomas Mack (mack3457)]

I see the problem here as well in 17.04.

The solution with x-systemd.automount mount option in fstab seems to help.

[Gerhard Beck (gerhardbeck-de)]

mount -a doesn't work in my case (17.04) but x-systemd.automount helps