NFS mounts in /etc/fstab fail

[Joe Warren-Meeks]

I have a bunch of NFS mounts in fstab on some of my webservers. These mount perfectly well if done manually after the boot, but if I do them automatically at boot time, they cause the server to fail to boot properly.

If I comment them out of fstab, then the server reboots successfully every time.

I've tried hostnames and IP addresses in /etc/fstab, but neither work.

Example fstab entry:
10.0.41.14:/mnt/beast/pgw /var/www/pgw/docs nfs ro,rsize=32768,hard,intr,nfsvers=3,udp,noatime,nodev,async,_netdev 0 0

I'm using static networking in /etc/network/interfaces:
auto eth0
iface eth0 inet static
        address 10.0.41.17
        netmask 255.255.255.0
        network 10.0.41.0
        broadcast 10.0.41.255
        gateway 10.0.41.1
        dns-nameservers 10.0.0.2
        dns-search int.xxxxxxx.co.uk

The dmesg log looks as follows: (I've pruned out most of the boot messages)
[ 1.257789] udevd[81]: starting version 175
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
[ 1.404157] Refined TSC clocksource calibration: 2593.390 MHz.
[ 1.424141] usb 1-1: new full-speed USB device number 2 using uhci_hcd
Begin: Running /scripts/local-premount ... done.
[ 1.653148] EXT4-fs (vda1): INFO: recovery required on readonly filesystem
[ 1.656874] EXT4-fs (vda1): write access will be enabled during recovery
[ 1.734688] FDC 0 is a S82078B
[ 1.817756] EXT4-fs (vda1): recovery complete
[ 1.822182] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
[ 2.898556] EXT4-fs (vda1): re-mounted. Opts: (null)
rpcbind: Cannot open '/run/rpcbind/rpcbind.xdr' file for reading, errno 2 (No such file or directory)
rpcbind: Cannot open '/run/rpcbind/portmap.xdr' file for reading, errno 2 (No such file or directory)
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: Network is unreachable
mountall: mount /var/www/tecom/docs [345] terminated with status 32
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: Network is unreachable
mountall: mount /var/www/theatrepeople/docs [350] terminated with status 32
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: Network is unreachable
mountall: mount /var/www/eolts/docs [337] terminated with status 32
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: Network is unreachable
mountall: mount /mnt [341] terminated with status 32
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: Network is unreachable
mountall: mount /var/www/cbolds/docs [400] terminated with status 32
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: Network is unreachable
mountall: mount /var/www/pgw/docs [339] terminated with status 32

[Steve Langasek (vorlon)]

This has nothing to do with rpcbind; reassigning.

The error in your log is:

mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: Network is unreachable
mountall: mount /var/www/theatrepeople/docs [350] terminated with status 32

What version of Ubuntu are you using, and what version of nfs-common is installed? Does /var/log/upstart/statd.log exist? If so, what are its contents (note: only readable as root)? Do you have a /var/log/upstart/statd-mounting-_var_www_theatrepeople_docs.log?

[Joe Warren-Meeks (joe-warren-meeks)]

Sorry for the delay in response, and for misassigning it.

joe@w1:~$ dpkg -l |grep nfs-common
ii nfs-common 1:1.2.5-3ubuntu3.1 NFS support files common to client and server

I'm running Ubuntu precise (12.04.2 LTS)

root@w1:/var/log/upstart# cat statd.log
UPSTART_EVENTS = local-filesystems started
portmap-wait stop/waiting

Seems normal

And no mounting logs.

It looks to be a race condition at startup. I think it's trying to mount the filesystems before the network and statd have started up and then hangs at the "Press blah to continue" dialogue. That would explain the statd not running and the mount.nfs: Network is unreachable entries.

If I remove the entries from fstab and add them into /etc/rc.local then everything works fine, which would lend some creedence to the theory.

Note that I have put _netdev in the fstab line, which should make mount wait until the network is available.

Let me know if there is anything I can do to help?

 -- joe.

[Steve Langasek (vorlon)]

> And no mounting logs.

Is the /etc/init/statd-mounting.conf file present on your system? This job should trigger whenever mountall asks to mount an nfs filesystem, and block the mounting of that filesystem until statd has started. If the job file is missing, that would explain the behavior seen here.

If the job file were present, I would expect to see at least a /var/log/upstart/statd-mounting.log containing the line 'Terminated'. Are there some nfs mounts that *do* mount correctly at boot? (i.e., is the list of mounts complaining about missing statd complete, or does one of the nfs mounts actually mount successfully?) If the latter, this may be related to the following change that was made to nfs-common 1:1.2.6-3ubuntu2 in Ubuntu 12.10:

  * Add an instance to statd-mounting, and change it to just wait for statd
    instead of trying to trigger it potentially out of order. This also means
    we don't need to try to force portmap to start from statd.

You could try installing the nfs-common package from 12.10 to see if it addresses the problem for you. If it does, we could look at updating nfs-common in 12.04 with these fixes.

> Note that I have put _netdev in the fstab line, which should
> make mount wait until the network is available.

_netdev is a no-op for network filesystems.

[Joe Warren-Meeks (joe-warren-meeks)]

Hey Steve,

Really sorry for the delay in replying.

statd-mounting.conf is there.
joe@w2:/var/log/upstart$ ls -las /etc/init/statd-mounting.conf
4 -rw-r--r-- 1 root root 738 Sep 28 2012 /etc/init/statd-mounting.conf

But there is no log in /var/log/upstart

joe@w2:/var/log/upstart$ ls -l *mounting*
ls: cannot access *mounting*: No such file or directory

No NFS shares mount. If I can get it to boot, which it does sometimes
(without any NFS mounts there), running a manual mount works fine.

I've put all the mount commands into /etc/rc.local and that works perfectly
fine.

I'll have a crack with the 12.10 nfs-common package and get back to you.

Kind regards

 -- joe.

On 24 March 2013 17:30, Steve Langasek <email address hidden> wrote:

> > And no mounting logs.
>
> Is the /etc/init/statd-mounting.conf file present on your system? This
> job should trigger whenever mountall asks to mount an nfs filesystem,
> and block the mounting of that filesystem until statd has started. If
> the job file is missing, that would explain the behavior seen here.
>
> If the job file were present, I would expect to see at least a
> /var/log/upstart/statd-mounting.log containing the line 'Terminated'.
> Are there some nfs mounts that *do* mount correctly at boot? (i.e., is
> the list of mounts complaining about missing statd complete, or does one
> of the nfs mounts actually mount successfully?) If the latter, this may
> be related to the following change that was made to nfs-common
> 1:1.2.6-3ubuntu2 in Ubuntu 12.10:
>
> * Add an instance to statd-mounting, and change it to just wait for statd
> instead of trying to trigger it potentially out of order. This also
> means
> we don't need to try to force portmap to start from statd.
>
> You could try installing the nfs-common package from 12.10 to see if it
> addresses the problem for you. If it does, we could look at updating
> nfs-common in 12.04 with these fixes.
>
> > Note that I have put _netdev in the fstab line, which should
> > make mount wait until the network is available.
>
> _netdev is a no-op for network filesystems.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1157171
>
> Title:
> NFS mounts in /etc/fstab fail
>
> Status in “nfs-utils” package in Ubuntu:
> Incomplete
>
> Bug description:
>
> I have a bunch of NFS mounts in fstab on some of my webservers. These
> mount perfectly well if done manually after the boot, but if I do them
> automatically at boot time, they cause the server to fail to boot properly.
>
> If I comment them out of fstab, then the server reboots successfully
> every time.
>
> I've tried hostnames and IP addresses in /etc/fstab, but neither work.
>
> Example fstab entry:
> 10.0.41.14:/mnt/beast/pgw /var/www/pgw/docs nfs
> ro,rsize=32768,hard,intr,nfsvers=3,udp,noatime,nodev,async,_netdev 0 0
>
>
> I'm using static networking in /etc/network/interfaces:
> auto eth0
> iface eth0 inet static
> address 10.0.41.17
> netmask 255.255.255.0
> network 10.0.41.0
> broadcast 10.0.41.255
> gateway 10.0.41.1
> ...

[Launchpad Janitor (janitor)]

[Expired for nfs-utils (Ubuntu) because there has been no activity for 60 days.]

[Jarod (jarod42)]

The problem of NFS shares not being mounted does still exist.
/var/log/upstart/mountall.log showed the following today:

mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified
mountall: mount /video0 [940] terminated with status 32
mountall: mount /mp3 [949] terminated with status 32
mountall: Disconnected from Plymouth

My /etc/fstab looks like this for the two mounts:
192.168.3.42:/mnt/grave/video0 /video0 nfs rsize=32768,wsize=32768,rw,hard,intr,tcp,nfsvers=3,addr=192.168.3.42 0
192.168.3.42:/mp3 /mp3 nfs nfsvers=3,rsize=2048,wsize=2048,tcp

I would also guess, that there is some kind of race condition where nfs filesystems are supposed to mounted before statd and/or network is running.
As long as eth0 got its IP via DHCP there were also lines like this in the log:

mount.nfs: Network is unreachable
mount.nfs: Network is unreachable
mountall: mount /video0 [812] terminated with status 32
mountall: mount /mp3 [815] terminated with status 32

BTW: My machine boots from a 250GB SSD.

When the mounting of the above filesystems fail, it does not start a self added service which should start on the following conditions:

start on (started dbus and started udev and started irserver and mounted MOUNTPOINT=/video0)

[Jarod (jarod42)]

I forgot...
Ubuntu 13.10 64bit server is running on my machine.

[Steve Langasek (vorlon)]

Jarod,

Why does /etc/init/statd-mounting.conf not handle this nfs startup condition for you? What shows up in /var/log/upstart/statd-mounting*.log?

[Jarod (jarod42)]

Both files show exactly one entry:
cat /var/log/upstart/statd-mounting-_video0.log /var/log/upstart/statd-mounting-_mp3.log
Terminated
Terminated

[Jarod (jarod42)]

I dug a little deeper and found that only 2 of my 4 NFS Shares were mentioned in the mountall.log.
The /var/log/upstart/statd-mounting*.log for the 2 other filesystems also contained "Terminated" as their only line.

The machine booted 4 times today with no error in the logs so far. So the error is not reproduce-able...

The following might not be directly related to this issue, but some logs under /var/log/upstart show errors multiple times about missing files.:

upstart-socket-bridge.log:
upstart-socket-bridge: Unable to write pid file: No such file or directory
upstart-udev-bridge.log:
upstart-udev-bridge: Unable to write pid file: No such file or directory
ureadahead-other.log:
ureadahead:/var/lib/ureadahead/var.pack: No such file or directory

I think that all of those errors are a result of /var not being mounted when the process is started. /var resides on a separate LV as do /boot, /home, /usr, /var and /tmp.

This is my /etc/fstab:
/dev/mapper/vdrstorage-root / xfs defaults 0 1
UUID=e5f29573-ce86-4f23-9b5f-68f6b1961943 /boot ext4 defaults 0 2
/dev/mapper/vdrstorage-home /home xfs defaults 0 2
/dev/mapper/vdrstorage-tmp /tmp xfs defaults 0 2
/dev/mapper/vdrstorage-usr /usr xfs defaults 0 2
/dev/mapper/vdrstorage-var /var xfs defaults 0 2
/dev/mapper/vdrstorage-swap none swap sw 0 0

192.168.3.42:/mnt/grave/video0 /video0 nfs rsize=32768,wsize=32768,rw,hard,intr,tcp,nfsvers=3,addr=192.168.3.42 0 0
192.168.3.42:/mnt/dvdisos /mnt/dvdisos nfs rsize=32768,wsize=32768,rw,hard,intr,tcp,nfsvers=3,addr=192.168.3.42 0 0
192.168.3.42:/mp3 /mp3 nfs nfsvers=3,rsize=2048,wsize=2048,tcp 0 0
192.168.3.42:/mnt/mplayer /mplayer nfs nfsvers=3,rsize=2048,wsize=2048,tcp

[Steve Langasek (vorlon)]

On Mon, Jan 13, 2014 at 04:38:45PM -0000, Jarod wrote:
> The /var/log/upstart/statd-mounting*.log for the 2 other filesystems also
> contained "Terminated" as their only line.

That much is expected.

> I think that all of those errors are a result of /var not being mounted
> when the process is started. /var resides on a separate LV as do /boot,
> /home, /usr, /var and /tmp.

Oh, thanks, that's helpful to know. It's possible that we've overlooked a
dependency that rpc.statd has on /var being mounted... I had believed that
rpc.statd didn't need /var given that it's located in /sbin, but strings
/sbin/rpc.statd does show a few references to /var/. I'll see what I
can find out here.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Launchpad Janitor (janitor)]

This bug was fixed in the package nfs-utils - 1:1.2.8-5ubuntu1

---------------
nfs-utils (1:1.2.8-5ubuntu1) trusty; urgency=medium

  * Merge from Debian unstable, remaining changes:
    - debian/nfs-common.{statd,statd-mounting,gssd,idmapd}.upstart,
      debian/control, debian/nfs-common.{preinst,postinst,prerm,postrm},
      debian/rules: drop nfs-common init script in favor of upstart jobs.
    - Depend on rpcbind (>= 0.2.0-6ubuntu1) for upstart support.
    - Depend on mountall (>= 2.41) to avoid deadlocks on boot.
    - debian/nfs-common.default: always start idmapd automatically; drop
      the configuration option.
    - nfs-kernel-server.init: Unmount nfsd fs when init script stops
    - Allow issuing options to rpc.nfsd
    - Add "-e" (ticket expiry is error) option to rpc.gssd to prevent hangs
      due to EKEYEXPIRED error from kernel on ticket expiry.
  * Dropped changes, included in Debian:
    - Move /var/lib/nfs/rpc_pipefs to /run/rpc_pipefs. This does not belong
      in /var/lib.
    - debian/nfs-kernel-server.postinst: don't call "invoke-rc.d nfs-common"
      in the postinst, this is redundant anyway and the nfs-common init script
      is gone now.

nfs-utils (1:1.2.8-5) unstable; urgency=medium

  [ Ben Hutchings ]
  * Remove Luk Claes from uploaders (Closes: #723602)

  [ Steve Langasek ]
  * Migrate the rpc_pipefs mount out of /var/lib to /run, to better
    support /var on NFS.
  * Don't start the nfs-common init script from the nfs-kernel-server
    maintainer script, this is entirely redundant.
  * debian/patches/21-no-more-var-run.patch: PID files should be in
    /run, not /var/run.
    Closes LP: #1157171.
 -- Steve Langasek <email address hidden> Mon, 13 Jan 2014 21:13:11 -0800

[henry (henry-torres)]

I know this has been close for over a year and it was fix on trusty but I wanted to point out how I work around it with out removing entries from fstab.

Description: Ubuntu 12.04.4 LTS

In the end removing apparmor did the trick. (boo I know)

I should mention before that I updated the kernel to the latest trusty LTS kernel & also updated nfs-common to the latest precise version avail:

$ dpkg -l | grep nfs-common
ii nfs-common 1:1.2.5-3ubuntu3.2 NFS support files common to client and server

Now every time I reboot even though I still get the:

mount.nfs: Network is unreachable

The machines boots up to prompt successfully every time now.

[Joe Warren-Meeks (joe-warren-meeks)]

Cool, thanks Henry! Much appreciated.

On 2 June 2015 at 23:43, henry <email address hidden> wrote:

> I know this has been close for over a year and it was fix on trusty but
> I wanted to point out how I work around it with out removing entries
> from fstab.
>
> Description: Ubuntu 12.04.4 LTS
>
> In the end removing apparmor did the trick. (boo I know)
>
> I should mention before that I updated the kernel to the latest trusty
> LTS kernel & also updated nfs-common to the latest precise version
> avail:
>
> $ dpkg -l | grep nfs-common
> ii nfs-common 1:1.2.5-3ubuntu3.2 NFS
> support files common to client and server
>
> Now every time I reboot even though I still get the:
>
> mount.nfs: Network is unreachable
>
> The machines boots up to prompt successfully every time now.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1157171
>
> Title:
> NFS mounts in /etc/fstab fail
>
> Status in nfs-utils package in Ubuntu:
> Fix Released
>
> Bug description:
>
> I have a bunch of NFS mounts in fstab on some of my webservers. These
> mount perfectly well if done manually after the boot, but if I do them
> automatically at boot time, they cause the server to fail to boot properly.
>
> If I comment them out of fstab, then the server reboots successfully
> every time.
>
> I've tried hostnames and IP addresses in /etc/fstab, but neither work.
>
> Example fstab entry:
> 10.0.41.14:/mnt/beast/pgw /var/www/pgw/docs nfs
> ro,rsize=32768,hard,intr,nfsvers=3,udp,noatime,nodev,async,_netdev 0 0
>
>
> I'm using static networking in /etc/network/interfaces:
> auto eth0
> iface eth0 inet static
> address 10.0.41.17
> netmask 255.255.255.0
> network 10.0.41.0
> broadcast 10.0.41.255
> gateway 10.0.41.1
> dns-nameservers 10.0.0.2
> dns-search int.xxxxxxx.co.uk
>
> The dmesg log looks as follows: (I've pruned out most of the boot
> messages)
> [ 1.257789] udevd[81]: starting version 175
> Begin: Loading essential drivers ... done.
> Begin: Running /scripts/init-premount ... done.
> Begin: Mounting root file system ... Begin: Running /scripts/local-top
> ... done.
> [ 1.404157] Refined TSC clocksource calibration: 2593.390 MHz.
> [ 1.424141] usb 1-1: new full-speed USB device number 2 using uhci_hcd
> Begin: Running /scripts/local-premount ... done.
> [ 1.653148] EXT4-fs (vda1): INFO: recovery required on readonly
> filesystem
> [ 1.656874] EXT4-fs (vda1): write access will be enabled during
> recovery
> [ 1.734688] FDC 0 is a S82078B
> [ 1.817756] EXT4-fs (vda1): recovery complete
> [ 1.822182] EXT4-fs (vda1): mounted filesystem with ordered data
> mode. Opts: (null)
> Begin: Running /scripts/local-bottom ... done.
> done.
> Begin: Running /scripts/init-bottom ... done.
> [ 2.898556] EXT4-fs (vda1): re-mounted. Opts: (null)
> rpcbind: Cannot open '/run/rpcbind/rpcbind.xdr' file for reading, errno
> 2 (No such file or directory)
> rpcbind: Cannot open '/run/rpcbind/portmap.xdr' file for read...