NFS mounts on clients don't appear in `mount` or `df` output

[David Goodwin]

Binary package hint: coreutils

Hi,

(Could be a bug with either mount, coreutils, the kernel or nfs-common??)

2 computer config, 1 server, 1 desktop.

Server exports /home (and others) via NFS to desktop.

Desktop has in /etc/fstab :
fitz:/home /home nfs defaults 0 0
fitz:/pub /pub nfs defaults 0 0

File systems are mounted (i.e. I can create files within the NFS filesystem) on the client, but when running `df` or `mount`, they do not appear as being mounted - i.e.
root@bonzo:~# mount
/dev/hda1 on / type reiserfs (rw,notail)
proc on /proc type proc (rw)
/sys on /sys type sysfs (rw)
varrun on /var/run type tmpfs (rw)
varlock on /var/lock type tmpfs (rw)
procbususb on /proc/bus/usb type usbfs (rw)
udev on /dev type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
devshm on /dev/shm type tmpfs (rw)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
root@bonzo:~# mount | grep -i nfs
root@bonzo:~#
root@bonzo:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 9775184 3600680 6174504 37% /
varrun 517336 104 517232 1% /var/run
varlock 517336 4 517332 1% /var/lock
udev 517336 96 517240 1% /dev
devshm 517336 0 517336 0% /dev/shm
root@bonzo:~# uname -a
Linux bonzo 2.6.15-22-k7 #1 SMP PREEMPT Sun May 7 17:27:47 UTC 2006 i686 GNU/Linux

/proc/mounts contains the correct information however :

root@bonzo:~# cat /proc/mounts
rootfs / rootfs rw 0 0
none /sys sysfs rw 0 0
none /proc proc rw,nodiratime 0 0
udev /dev tmpfs rw 0 0
/dev/hda1 / reiserfs rw 0 0
/dev/hda1 /dev/.static/dev reiserfs rw 0 0
tmpfs /var/run tmpfs rw 0 0
tmpfs /var/lock tmpfs rw 0 0
usbfs /proc/bus/usb usbfs rw 0 0
devpts /dev/pts devpts rw 0 0
tmpfs /dev/shm tmpfs rw 0 0
tmpfs /var/run tmpfs rw 0 0
tmpfs /var/lock tmpfs rw 0 0
fitz:/home /home nfs rw,v3,rsize=32768,wsize=32768,hard,lock,proto=tcp,addr=fitz 0 0
fitz:/pub /pub nfs rw,v3,rsize=32768,wsize=32768,hard,lock,proto=tcp,addr=fitz 0 0
tmpfs /var/run tmpfs rw 0 0
tmpfs /var/lock tmpfs rw 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0

I'll hope I'm not doing something really stupid, and it's a bug :)

root@bonzo:~# dpkg -l | grep coreutils
ii coreutils 5.93-5ubuntu4
root@bonzo:~# dpkg -l | grep mount
ii mount 2.12r-4ubuntu5

[Christian Eckerle (ce-publishing-etc)]

There seems to be a race condition between the initialization of the mtab file and mounting the nfs file systems by the if-up script.

In addition to that every NFS share mounted on other partitions than root is not mounted at all because these partitions get mounted later.

I worked around the issue by inserting a "sleep 15" in /etc/network/if-up.d/mountnfs

[Scott James Remnant (Canonical) (canonical-scott)]

Confirmed as an initscripts problem

[Scott James Remnant (Canonical) (canonical-scott)]

Confirmed, but relatively minor.

[David Goodwin (david-codepoets)]

Any chance of a fix before Dapper goes gold?

(problem still exists as of 23/05/2006)

[Pieter Nagel (pieter-equinox)]

In my case this bug is more than "minor": nfs mounts are not merely missing from mtab, they are often missing from the filesystem as well.

To summarise: after boot, if an NFS mountpoint is missing from mtab, then:
    1) In the case of an NFSv3 mount, the mounted files are sometimes visible, sometimes missing
    2) In the case of an NFSv4 mount, the mounted files are alwaus missing, too.

I marked Bug #46145 which I just reported as a duplicate of this one.

[Scott James Remnant (Canonical) (canonical-scott)]

If NFS mounts are not actually mounted (ie not in /proc/mounts) then you have a combination of two bugs, one new one and this one.

[Scott James Remnant (Canonical) (canonical-scott)]

David: absolutely no chance

[Pieter Nagel (pieter-equinox)]

NFS mounts *are* sometimes not actually mounted. NFSv4 mounts seem to be more often missing than not. After reboot:

pieter@danae:~$ grep : /etc/fstab
# /etc/fstab: static file system information.
repo:/aegis /aegis nfs4 rw,rsize=32768,wsize=32768,defaults 0 0
pixar:/net/pixar /net/pixar nfs rw,tcp,rsize=32768,wsize=32768,defaults 0 0

pieter@danae:~$ grep : /proc/mounts
pixar:/net/pixar /net/pixar nfs rw,v3,rsize=32768,wsize=32768,hard,lock,proto=tcp,addr=pixar 0 0

pieter@danae:~$ sudo mount -a
pieter@danae:~$ grep : /proc/mounts
pixar:/net/pixar /net/pixar nfs rw,v3,rsize=32768,wsize=32768,hard,lock,proto=tcp,addr=pixar 0 0
repo:/aegis /aegis nfs4 rw,v4,rsize=32768,wsize=32768,hard,intr,lock,proto=tcp,addr=repo 0 0

But if I re-order the NFSv4 mount after the NFSv3 mount, it *was* mounted 6 reboots in a row.

[Pieter Nagel (pieter-equinox)]

I submitted new Bug #46516 for the more serious "NFSv4 mounts not made at all" issue, so this bug number can continue to refer to the less serious "NFS mounts made but missing from mtab" problem.

[Scott James Remnant (Canonical) (canonical-scott)]

On Thu, 2006-05-25 at 07:47 +0000, Pieter Nagel wrote:

> I submitted new Bug #46516 for the more serious "NFSv4 mounts not made
> at all" issue, so this bug number can continue to refer to the less
> serious "NFS mounts made but missing from mtab" problem.
>
Many Thanks!

Keeping things separate makes our lives soooo much easier :)

Scott
--
Scott James Remnant
<email address hidden>

[Andreas Heinz (crash-blab)]

since this bug is open a long time now: is there a chance that this will be fixed?

i just tried kde with kde free disk applet which lets you only choose "partitions" mounted into the system. with this bug present, network shares can't be checked.

so after sysvinit will be replaced, will this bug be fixed?

[RParr (rparr)]

I am seeing this problem in recently installed Feisty (final) amd64.

NFS mounts made during boot do not appear in "mount" or "df" commands but the mounts have occurred and show up in /proc/mounts.

NFS mounts made from command-line after boot do appear in "mount" and "df" command.

[Andreas Heinz (crash-blab)]

@Scott:
since this bug is now attached to sysvinit and sysvinit has been replaced since edgy: will this bug be fixed anytime or shouldn't this happen, when upstart ist used instead of sysvinit?

thanks
Andreas

[Georg C.F. Wolff (boeser-wolff)]

An alternate solution - instead of waiting a fixed time - is to modify /etc/network/if-up.d/mountnfs to include a runlevel-check:

                RUNLEVEL=`runlevel`

                while [ "${RUNLEVEL}" = "unknown" ]; do
                  if ps aux | grep -q " sh /etc/rcS.d/S45waitnfs.sh start$"; then
# pidof doesnt work if waitnfs is started "sh /etc/.../S45waitnfs.sh"
                    break
                  fi

                  sleep 0.1
                  RUNLEVEL=`runlevel`
                done

This will prevent the mountnfs script from being run to early, but also abort if the waitnfs script is running. This prevents collisions between these two scripts.

[AlexLG (alex-alexlg)]

I have the same problem with The Gutsy Gibbon and the patch works wery well for me, thanks.

[David Vincent (dvincent)]

I have the same problem on Gutsy Gibbon and the patch worked for me as well.

[ski (skibrianski)]

This patch works for me on feisty. Haven't tested on gutsy yet.

[Jeffery Bond (jeff-clearspeed)]

The patch to mountnfs worked for me on ubuntu 7.10, but this bug has another nasty effect:

When updatedb/slocate is running, it will scan NFS mounts, even if the PRUNEFS string contains "nfs" in the /etc/updatedb.conf file. The PRUNEPATHS option does seem to work however. This is pretty bad since it results in lots of NFS traffic, and a general slowdown of the machine, and also updatedb tages AGES to complete!

[Moritz Bunkus (m-bunkus)]

7.10: I noticed the same issue Jeffery Bond noticed. This also happened with 6.06, probably because the NFS mounts did not show in /etc/mtab (even though they did in /proc/mounts). However, with the patch Georg Wulff supplied the NFS mounts show up in /etc/mtab again, and slocate's updatedb skips them again.

[Matthew L. Dailey (matthew-l-dailey)]

This bug is still present in hardy as of 20080124. I've got a kerberized nfs3 mount in fstab:
thayerfs.thayer.dartmouth.edu:/vol/apps /thayerfs/apps nfs vers=3,sec=krb5,nosuid

It is accessible after boot, but only shows up in /proc/mounts - not in /etc/mtab or via the mount command.

Adding Goerg's patch from comment 14 fixes the issue.

[Jorge Allyson Azevedo (allyson-land)]

I think I found another problem related to this bug.
If the missing nfs mount uses quota, the quota command will not know it at all. If I want to know my quota limits on that nfs mount I have no way unless I manually remount it.
I was wondering if there are other programs that would be affected by this.

[Dustin Kirkland  (kirkland)]

I could not reproduce this problem in Hardy as of 2008-02-20.

I did reproduce the problem on Gutsy on 2008-02-02, and solved it by the code suggested in Comment 14.

I'm attaching a unified diff patch.

[Dustin Kirkland  (kirkland)]

Better yet, a debdiff against gutsy's initscripts, with changelog entry and version incrementing.

[Dustin Kirkland  (kirkland)]

Typo in debdiff.... s/Launnchpad/Launchpad/

[Dustin Kirkland  (kirkland)]

I have updated this debdiff with one that solves this problem, as well as Bug #45842. In this version, however, I create a lock file, /var/run/waitnfs.sh.pid, and check for that, rather than using the ps aux | grep ... construct.

See:
https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/45842/comments/29

[Dustin Kirkland  (kirkland)]

Per request from Kees Cook, explicit steps to reproduce this problem....

* Fresh install of Hardy (VM is fine)
* apt-get install nfs-common
* Have a remote NFS server elsewhere running, and exporting a mount
* Add that mount to your fresh install's /etc/fstab, perhaps something like:
192.168.1.3:/nfs/foo /nfs/foo nfs defaults 0 0
* Test your fstab entry with "mount -a", make sure it mounts properly
* Reboot the test system
* After boot, you can see /nfs/foo in /proc/mounts. You can list files and everything in it. However, running the "mount" command or the "df" command will fail to show /nfs/foo in your mtab

[Launchpad Janitor (janitor)]

This bug was fixed in the package sysvinit - 2.86.ds1-14.1ubuntu37

---------------
sysvinit (2.86.ds1-14.1ubuntu37) hardy; urgency=low

  * debian/initscripts/etc/init.d/mtab.sh:
    Explicitly add mtab entries for networked file systems; race condition
    exists where mtab can get overwritten *after* networked file systems are
    mounted (LP: #46145, #44836).
  * debian/initscripts/etc/init.d/waitnfs.sh, debian/control:
    Update inline documentation, as waitnfs ONLY waits on /usr and /var mounted
    filesystems.
    Before waiting for net file systems, try mounting them (with watershed)
    just in case the mount has not been attempted elsewhere.
    Added udev as dependency due to use of watershed.
  * debian/initscripts/etc/network/if-up.d/mountnfs:
    Remove the bulk of this script to a common library script
    Call the common library script with watershed for locking purposes
    (LP: #45842, #46516).
  * debian/initscripts/lib/init/mountall-net-fs, debian/rules:
    Common library script for mounting all network file systems (nfs, cifs,
    samba, etc).
    Script can be called in various meaningful locations; should be wrapped
    with watershed to correctly handle race conditions.

 -- Dustin Kirkland <email address hidden> Tue, 26 Feb 2008 16:10:56 -0600

[Eric Bursley (eric-bursley)]

I still have this problem in hardy. I'm running the latest patches. mount -a will mount all missing NFS mounts after I'm logged in. I have a Precision Workstation 690, so the race condition spoken about before may still be happening with faster systems.

[Marcus Bointon (marcus-synchromedia)]

History repeating - 3 years later and I'm seeing exactly this problem in a new 12.04 install! I've got a glusterFS volume that mounts via NFS - it shows in /proc/mounts but not df or mount output.

[Sam Bromley (sam-sambromley)]

Hi All, It would seem that this bug is still popping up.
I noticed this today, running 13.04.
/proc/mounts is fine.
/etc/mtab is missing nfs mounts.

[Ian Campbell (d-ian-k)]

Note sure if it's the same bug, but running 14.04.

# cat /etc/os-release
NAME="Ubuntu"
VERSION="14.04, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/mapper/kubuntu--vg-root / ext4 errors=remount-ro 0 1
# /boot was on /dev/sda1 during installation
UUID=3c4f0566-cf7a-4d7a-b710-2da7f27d202e /boot ext2 defaults 0 2
/dev/mapper/kubuntu--vg-swap_1 none swap sw 0 0

# Synology NFS Mounts
w.x.y.z:/volume1/video /media/nfs/video nfs rsize=8192,wsize=8192,timeo=14,intr 0 0
w.x.y.z:/volume1/Backups /media/nfs/backups nfs rsize=8192,wsize=8192,timeo=14,intr 0 0
w.x.y.z:/volume1/Documents /media/nfs/documents nfs rsize=8192,wsize=8192,timeo=14,intr 0 0
w.x.y.z:/volume1/Music /media/nfs/music nfs rsize=8192,wsize=8192,timeo=14,intr 0 0
w.x.y.z:/volume1/Downloads /media/nfs/downloads nfs rsize=8192,wsize=8192,timeo=14,intr 0 0
w.x.y.z:/volume1/VMachines /media/nfs/VMachines nfs rsize=8192,wsize=8192,timeo=14,intr 0 0
//w.x.y.z/video /media/cifs/video cifs user,uid=1000,rw,suid,username=ian,password=Maxp0wer1 0 0
//v.w.x.y/pi /media/cifs/pi cifs user,uid=1000,rw,suid,username=pi,password=raspberry 0 0

# cat /etc/mtab
/dev/mapper/kubuntu--vg-root / ext4 rw,errors=remount-ro 0 0
proc /proc proc rw,noexec,nosuid,nodev 0 0
sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0
none /sys/fs/cgroup tmpfs rw 0 0
none /sys/fs/fuse/connections fusectl rw 0 0
none /sys/kernel/debug debugfs rw 0 0
none /sys/kernel/security securityfs rw 0 0
udev /dev devtmpfs rw,mode=0755 0 0
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=0620 0 0
tmpfs /run tmpfs rw,noexec,nosuid,size=10%,mode=0755 0 0
none /run/lock tmpfs rw,noexec,nosuid,nodev,size=5242880 0 0
none /run/shm tmpfs rw,nosuid,nodev 0 0
none /run/user tmpfs rw,noexec,nosuid,nodev,size=104857600,mode=0755 0 0
none /sys/fs/pstore pstore rw 0 0
/dev/sda1 /boot ext2 rw 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw 0 0
systemd /sys/fs/cgroup/systemd cgroup rw,noexec,nosuid,nodev,none,name=systemd 0 0
//v.w.x.y/pi /media/cifs/pi cifs rw,noexec,nodev,uid=1000,username=pi,password=raspberry 0 0
//w.x.y.z/video /media/cifs/video cifs rw,noexec,nodev,uid=1000,username=ian,password=Maxp0wer1 0 0
w.x.y.z:/volume1/Music /media/nfs/music nfs rw,rsize=8192,wsize=8192,timeo=14,intr,addr=w.x.y.z 0 0
w.x.y.z:/volume1/VMachines /media/nfs/VMachines nfs rw,rsize=8192,wsize=8192,timeo=14,intr,addr=w.x.y.z 0 0
w.x.y.z:/volume1/video /media/nfs/video nfs rw,rsize=8192,wsize=8192,timeo=14,intr,addr=w.x.y.z 0 0
w.x.y.z:/volume1/Downloads /media/nfs/downloads nfs rw,rsize=8192,wsize=8192,timeo=14,intr,addr=w.x.y.z 0 0
w.x.y.z:/volume1/Backups /media/nfs/backups nfs r...