nfs mounts failing during reboot

[Chris Good]

12.04.1 LTS

I have 3 nfs mounts in /etc/fstab but only 2 succeed. After logging in, I can manually mount the 3rd nfs file system (/alansun_root) with no problems. This seems to be similar to https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/891825 but that seems to be closed do to lack of activity and was about an older version of Ubuntu.

If I change the order of the 3 nfs file systems in /etc/fstab, the file system not mounted changes, so it seems to be a timing issue. Sometimes, only 1 of the 3 file systems is mounted.

Here is /etc/fstab:

proc /proc proc nodev,noexec,nosuid 0 0
UUID=7ecd0957-9c46-4958-bf24-bb635271d4e5 / ext4 errors=remount-ro 0 1
UUID=7f698647-0a0c-48f1-994d-79d8ea3f4043 none swap sw 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
UUID=2405d070-aacd-48b7-9bc5-7be9d09c2f42 /usr/local ext4 errors=remount-ro 0 2
UUID=2613cf1b-dc94-447f-91fb-575b0112686d /usr/tims ext4 errors=remount-ro 0 2
UUID=84295b7a-961c-4357-9790-2ce0acb1ae05 /data1 ext4 errors=remount-ro 0 2
UUID=95ddb6ca-0322-4da6-872b-9cfacdb53c57 /u ext4 errors=remount-ro 0 2
UUID=e2b812d7-63d0-485f-b28d-acbeb76b70de /spare ext4 errors=remount-ro 0 2
LABEL=Passport1 /media/Passport1 ntfs defaults,windows_names,locale=en_US.utf8,nobootwait 0 0
LABEL=Passport2 /media/Passport2 ntfs defaults,windows_names,locale=en_US.utf8,nobootwait 0 0
alansun:/ /alansun_root nfs defaults,ro,bg,vers=3 0 0
alansun:/usr/tims /alansun_usr_tims nfs defaults,ro,bg,vers=3 0 0
chcaix01:/u/i22/FACT/clientsubs/clients /u/i22/FACT/clientsubs/clients nfs defaults,rw,bg 0 0

After boot, mountall is still running:

# ps -ef|grep mount
root 267 1 0 14:06 ? 00:00:00 mountall --verbose --daemon
root 829 1 0 14:06 ? 00:00:00 /sbin/mount.ntfs /dev/sdc1 /media/Passport2 -o rw,windows_names,locale=en_US.utf8

I added --verbose to mountall in /etc/init/mountall.conf and here is /var/log/boot.log:

Connected to Plymouth
/ is local
/proc is virtual
/sys is virtual
/sys/fs/fuse/connections is virtual
/sys/kernel/debug is virtual
/sys/kernel/security is virtual
/dev is virtual
/dev/pts is virtual
/tmp is local
/run is virtual
/run/lock is virtual
/run/shm is virtual
UUID=7f698647-0a0c-48f1-994d-79d8ea3f4043 is swap
/usr/local is local
/usr/tims is local
/data1 is local
/u is local
/spare is local
/media/Passport1 is nowait
/media/Passport2 is nowait
/alansun_root is nowait
/alansun_usr_tims is nowait
/u/i22/FACT/clientsubs/clients is nowait
local 0/7 remote 0/0 virtual 1/10 swap 0/1
local 0/7 remote 0/0 virtual 2/10 swap 0/1
local 0/7 remote 0/0 virtual 3/10 swap 0/1
local 0/7 remote 0/0 virtual 4/10 swap 0/1
local 0/7 remote 0/0 virtual 5/10 swap 0/1
mounting /sys/fs/fuse/connections
mount /sys/fs/fuse/connections [448] exited normally
local 0/7 remote 0/0 virtual 6/10 swap 0/1
mounting /sys/kernel/debug
mount /sys/kernel/debug [449] exited normally
local 0/7 remote 0/0 virtual 7/10 swap 0/1
mounting /sys/kernel/security
mount /sys/kernel/security [452] exited normally
local 0/7 remote 0/0 virtual 8/10 swap 0/1
mounting /run/lock
mount /run/lock [453] exited normally
local 0/7 remote 0/0 virtual 9/10 swap 0/1
mounting /run/shm
mount /run/shm [454] exited normally
virtual finished
remote finished
local 0/7 remote 0/0 virtual 10/10 swap 0/1
checking /
fsck from util-linux 2.20.1
activating /dev/disk/by-uuid/7f698647-0a0c-48f1-994d-79d8ea3f4043
swapon /dev/disk/by-uuid/7f698647-0a0c-48f1-994d-79d8ea3f4043 [466] exited normally
swap finished
local 0/7 remote 0/0 virtual 10/10 swap 1/1
checking /usr/local
checking /usr/tims
fsck from util-linux 2.20.1
checking /data1
fsck from util-linux 2.20.1
checking /u
checking /spare
fsck from util-linux 2.20.1
fsck from util-linux 2.20.1
fsck from util-linux 2.20.1
rpcbind: Cannot open '/run/rpcbind/rpcbind.xdr' file for reading, errno 2 (No such file or directory)
rpcbind: Cannot open '/run/rpcbind/portmap.xdr' file for reading, errno 2 (No such file or directory)
Received SIGUSR1 (network device up)
ata_id[571]: HDIO_GET_IDENTITY failed for '/dev/sdc': Invalid argument

/dev/sda1: clean, 176620/393216 files, 1081761/1572864 blocks
fsck / [464] exited normally
remounting /
mount / [789] exited normally
mount / [791] exited normally
mount /proc [792] exited normally
mount /sys [793] exited normally
mount /sys/fs/fuse/connections [794] exited normally
mount /sys/kernel/debug [795] exited normally
mount /sys/kernel/security [796] exited normally
mount /dev [797] exited normally
mount /dev/pts [798] exited normally
mount /run [799] exited normally
mount /run/lock [800] exited normally
mount /run/shm [801] exited normally
local 1/7 remote 0/0 virtual 10/10 swap 1/1
usr_local: clean, 47/122160 files, 16627/487966 blocks
local 2/7 remote 0/0 virtual 10/10 swap 1/1
mounting /media/Passport2
spare: clean, 11/1414448 files, 132798/5652480 blocks
u: clean, 984/610800 files, 108455/2441728 blocks
usr_tims: clean, 150509/1831424 files, 2057142/7323631 blocks
data1: clean, 15371/610800 files, 763789/2441872 blocks
mount /media/Passport2 [814] exited normally
local 2/7 remote 0/0 virtual 10/10 swap 1/1
mounting /alansun_root
mount.nfs: Network is unreachable
mounting /alansun_usr_tims
mount.nfs: Network is unreachable
fsck /usr/local [483] exited normally
mounting /usr/local
mount /usr/local [856] exited normally
local 3/7 remote 0/0 virtual 10/10 swap 1/1
fsck /usr/tims [485] exited normally
mounting /usr/tims
mount /usr/tims [860] exited normally
local 4/7 remote 0/0 virtual 10/10 swap 1/1
fsck /data1 [497] exited normally
mounting /data1
mount /data1 [864] exited normally
local 5/7 remote 0/0 virtual 10/10 swap 1/1
fsck /u [498] exited normally
mounting /u
mount /u [868] exited normally
local 6/7 remote 0/0 virtual 10/10 swap 1/1
fsck /spare [499] exited normally
mounting /spare
mount /spare [872] exited normally
local finished
All filesystems mounted
local 7/7 remote 0/0 virtual 10/10 swap 1/1
mountall: mount /alansun_root [843] terminated with status 32
Filesystem could not be mounted: /alansun_root
mountall: mount /alansun_usr_tims [848] terminated with status 32
Filesystem could not be mounted: /alansun_usr_tims
modem-manager[929]: <info> ModemManager (version 0.5.2.0) starting...

modem-manager[929]: <info> Loaded plugin Wavecom

modem-manager[929]: <info> Loaded plugin X22X

modem-manager[929]: <info> Loaded plugin Gobi

modem-manager[929]: <info> Loaded plugin Longcheer

modem-manager[929]: <info> Loaded plugin Novatel

modem-manager[929]: <info> Loaded plugin Samsung

modem-manager[929]: <info> Loaded plugin Nokia

modem-manager[929]: <info> Loaded plugin Option High-Speed

modem-manager[929]: <info> Loaded plugin AnyData

modem-manager[929]: <info> Loaded plugin Sierra

modem-manager[929]: <info> Loaded plugin MotoC

modem-manager[929]: <info> Loaded plugin SimTech

modem-manager[929]: <info> Loaded plugin Ericsson MBM

modem-manager[929]: <info> Loaded plugin Huawei

modem-manager[929]: <info> Loaded plugin ZTE

modem-manager[929]: <info> Loaded plugin Option

modem-manager[929]: <info> Loaded plugin Linktop

modem-manager[929]: <info> Loaded plugin Generic

mounting /alansun_root
mount.nfs: Network is unreachable
 * Starting mDNS/DNS-SD daemon [ OK ]
 * Starting bluetooth daemon [ OK ]
 * Starting Block the mounting event for NFS filesytems until statd is running [ OK ]
mounting /alansun_usr_tims
 * Stopping Block the mounting event for NFS filesytems until statd is running [ OK ]
mount.nfs: Network is unreachable
Skipping profile in /etc/apparmor.d/disable: usr.bin.firefox
 * Starting Block the mounting event for NFS filesytems until statd is running [ OK ]
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
 * Stopping Block the mounting event for NFS filesytems until statd is running [ OK ]
mounting /u/i22/FACT/clientsubs/clients
local 7/7 remote 0/0 virtual 10/10 swap 1/1
mountall: mount /alansun_root [973] terminated with status 32
Filesystem could not be mounted: /alansun_root
mountall: mount /alansun_usr_tims [995] terminated with status 32
Filesystem could not be mounted: /alansun_usr_tims
 * Starting AppArmor profiles [ OK ]
mounting /alansun_root
 * Starting Block the mounting event for NFS filesytems until statd is running [ OK ]
 * Stopping Block the mounting event for NFS filesytems until statd is running [ OK ]
 * Starting Block the mounting event for NFS filesytems until statd is running [ OK ]
 * Stopping Block the mounting event for NFS filesytems until statd is running [ OK ]
mounting /alansun_usr_tims
 * Stopping System V initialisation compatibility [ OK ]
 * Not starting internet superserver: no services enabled
 * Starting Block the mounting event for NFS filesytems until statd is running [ OK ]
 * Starting System V runlevel compatibility [ OK ]
 * Starting crash report submission daemon [ OK ]
 * Starting automatic crash report generation [ OK ]
 * Starting ACPI daemon [ OK ]
 * Starting anac(h)ronistic cron [ OK ]
 * Starting save kernel messages [ OK ]
 * Starting regular background program processing daemon [ OK ]
 * Starting deferred execution scheduler [ OK ]
 * Starting CPU interrupts balancing daemon [ OK ]
 * Starting LightDM Display Manager [ OK ]
 * Stopping Block the mounting event for NFS filesytems until statd is running [ OK ]
 * Starting CUPS printing spooler/server [ OK ]
 * Stopping anac(h)ronistic cron [ OK ]

# apt-cache policy nfs-common
nfs-common:
  Installed: 1:1.2.5-3ubuntu3.1
  Candidate: 1:1.2.5-3ubuntu3.1
  Version table:
 *** 1:1.2.5-3ubuntu3.1 0
        500 http://au.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1:1.2.5-3ubuntu3 0
        500 http://au.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

Please let me know if you need any more info.

[Chris Good (chris-good)]

I have worked around the problem by running a script during reboot which sleeps for 30 seconds, then mounts any unmounted nsf file systems.

Add to /etc/rc.local :

nohup /usr/local/bin/nfs_fix >/tmp/nfs_fix.out 2>&1 &

/usr/local/bin/nfs_fix :

#!/bin/ksh

# /usr/local/bin/nfs_fix

# Do to timing problem causing some nfs mounts to fail during reboot,
# this script is initiated from /etc/rc.local.

do_fix_nfs()
{
        echo Checking $1
        if df | grep $1 >/dev/null ; then
                echo $1 is already mounted
        else
                echo Mounting $1
                mount $1
        fi
}

sleep 30 # wait for networking to start

awk '($3=="nfs") {print $2;}' /etc/fstab |
grep -v '^#' > /tmp/nfs_fix.fs

while read fs ; do
        do_fix_nfs $fs
done < /tmp/nfs_fix.fs

[Steve Langasek (vorlon)]

This could be a race condition where the network interface reports itself to be up, but traffic is not actually being routed and the mount request fails, and mountall never gets told to try again. If you run 'killall -USR1 mountall' after boot (basically, the same thing /etc/init/mountall-net.conf does), does the share get mounted?

The mount.ntfs process still running after boot is also concerning. If you manually kill *that* process, does anything happen with the nfs mount?

Finally, please attach the output of 'sudo initctl list' run after boot.

[Chris Good (chris-good)]

Hi Steve,

1) The share was not mounted after I did 'killall -USR1 mountall'
2) After I killed mount.ntfs pid, the nfs file system was still not mounted, although /media/Passport2 (ntfs) was no longer mounted.

I have attached output of 'initctl list' & boot.log.

[Chris Good (chris-good)]

Output of initctl list

[Chris Good (chris-good)]

boot.log

[Launchpad Janitor (janitor)]

[Expired for nfs-utils (Ubuntu) because there has been no activity for 60 days.]

[Samuel Hassine (samuel-hassine-s)]

Confirmed under 12.04.2 LTS here.

If DHCP lease is acquired with delay, NFS partitions are unable to be mounted at boot time. Need sleep & remount.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Tome (pinevalleytome)]

I too just ran into this issue on my 12.04.2 server. It's been working for a while and then a couple updates and a reboot stopped all that. I can run a "sudo mount -a" after I log into my server to fix it but it will fail to mount the NFS share at boot.

FSTAB:
h1p-nas1:/volume1/backups /mnt/backups nfs rw,hard,tcp 0 0
h1p-nas1:/volume1/restores /mnt/restores nfs rw,hard,tcp 0 0

[Martin Fischer (mfischer)]

also affected on a fresh LTS 12.04.2 (server) installation:

while booting, i've got

fsck from util-linux 2.20.1
/dev/sda1: clean, 103901/14745600 files, 7194418/58958592 blocks
rpcbind: Cannot open '/run/rpcbind/rpcbind.xdr' file for reading, errno 2 (No such file or directory)
rpcbind: Cannot open '/run/rpcbind/portmap.xdr' file for reading, errno 2 (No such file or directory)
Skipping /srv/backup at user request

and had to skip this manually. my raid-device md0 change to md127 after this fault. if i'am commented out the entry of the raid-device everthings fine while booting.

my fstab:

proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda1 during installation
UUID=4f80a96c-ac33-4b2d-9548-3212fcb7c2b2 / ext4 errors=remount-ro 0 1
# swap was on /dev/sda5 during installation
UUID=5738cfa1-22ab-4a04-acfb-6830d728ddee none swap sw 0 0
# /srv/backup on /dev/md0
UUID=a4cd04b8:a59e2e9e:0e9cea39:78881cd5 /srv/backup ext4 errors=remount-ro 0 1
#
192.168.1.8:/volume1/share /srv/share nfs defaults,rsize=32768,wsize=32768,intr 0 0

regards...

[Bas van den Dikkenberg (bas-dikkenberg)]

The same thing om Ubuntu server 12.10 and 13.04

[Bas van den Dikkenberg (bas-dikkenberg)]

The workaround didn´t work for me :-(

The only way for me is reboot in recovery mode select network en continue normal boot

My /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda1 during installation
UUID=38804c97-5eb9-4bbb-afe9-eb51dd53c821 / ext4 errors=remount-ro 0 1
# swap was on /dev/sda5 during installation
UUID=fe0eb90b-3caf-4a6e-bd76-b734ed8bd952 none swap sw 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
172.31.1.210:/disk/usr/local/someting/bin /usr/local/someting/bin nfs ro,hard,intr,nobootwait 1 0
172.31.1.210:/disk/usr/local/someting/etc /usr/local/someting/etc nfs ro,hard,intr,nobootwait 1 0
172.31.1.210:/disk/usr/local/someting/lib /usr/local/someting/lib nfs ro,hard,intr,nobootwait 1 0
172.31.1.210:/disk/usr/local/someting/home /usr/local/someting/home nfs rw,hard,intr,nobootwait 1 0
172.31.1.210:/disk/usr/local/someting/ldap_auth /usr/local/someting/ldap_auth nfs ro,rsize=32768,wsize=32768,intr,noatime 1 0
172.31.1.210:/disk/usr/local/someting/firewall/general /usr/local/someting/firewall/general nfs rw,hard,intr,nobootwait 1 0
172.31.1.210:/disk/dir/dir/hccforum-beta /disk/dir/dir/hccforum-beta nfs rw,rsize=32768,wsize=32768,intr,noatime 1 0
172.31.1.210:/disk/dir/etc/ssl/some-site /disk/dir/etc/ssl/some-site nfs ro,rsize=32768,wsize=32768,intr,noatime 1 0

/var/log/boot.log

swapon: /dev/disk/by-uuid/fe0eb90b-3caf-4a6e-bd76-b734ed8bd952: swapon failed: Device or resource busy
mountall: swapon /dev/disk/by-uuid/fe0eb90b-3caf-4a6e-bd76-b734ed8bd952 [1222] terminated with status 255
mountall: Problem activating swap: /dev/disk/by-uuid/fe0eb90b-3caf-4a6e-bd76-b734ed8bd952
rpcbind: Cannot open '/run/rpcbind/rpcbind.xdr' file for reading, errno 2 (No such file or directory)
rpcbind: Cannot open '/run/rpcbind/portmap.xdr' file for reading, errno 2 (No such file or directory)
 * Starting NFSv4 id <-> name mapper [74G[ OK ]
 * Starting Block the mounting event for NFS4 filesytems until idmapd is running [74G[ OK ]
 * Starting Block the mounting event for NFS filesytems until statd is running [74G[ OK ]
 * Stopping Block the mounting event for NFS filesytems until statd is running [74G[ OK ]
 * Stopping Block the mounting event for NFS4 filesytems until idmapd is running [74G[ OK ]
 * Starting Block the mounting event for NFS4 filesytems until idmapd is running [74G[ OK ]
 * Starting Block the mounting event for NFS filesytems until statd is running [74G[ OK ]
 * Stopping Block the mounting event for NFS4 filesytems until idmapd is running [74G[ OK ]
 * Stopping Block the mounting event for NFS filesytems until statd is running [74G[ OK ]
 * S...

[dbrossard (dbrossard)]

I had the same problem using a solaris ZFS server as the NFS file server. I was able to fix this by adding vers=3 to the options. I you do not need version 4 of NFS try using this in your fstab.

[Chris Good (chris-good)]

Hi dbrossard,

I guess you meant this suggestion for other people with this problem as my fstab shows I already have vers=3 on 2 of my 3 nfs mounts, but just for completeness, I have already tried that but it did not help.

[Torsten Bronger (bronger)]

I run into this problem with Lubuntu 13.10. The problem occurs non-deterministically, so I cannot really tell whether the workaround works for me. Also for me, nfsvers=3 did not help.

[Torsten Bronger (bronger)]

It seems to happen at exactly every other reboot.

The workaround script from above does not work for me because rc.local is run too late in the boot process. The computer hangs and waits for the nfs mounts because it is execued.

If anyone is willing to investigate this, I'm ready to make tests and give feedback.

[Tom Hallman (hallmant)]

I'm also having this problem with a fresh 12.04 Ubuntu server install. I've added a couple NFS shares to /etc/fstab and now the server rarely if ever comes up properly. Like bas-dikkenberg, I can sometimes get around it if I do recovery console > network > continue normal boot. At other times recovery console > network just hangs as well.

[Scott Johnson (scottj75074)]

I'm having this problem with 14.04. If I run `killall -USR1 mountall`, then the mounts complete.