NFS not being mounted at boot time - rpc.statd is not running but is required for remote locking

[borninbronx]

Version: Ubuntu SERVER 11.04

installed nfs-commons

added this to /etc/fstab:

192.168.88.252:/repo /mnt/repos nfs rw,hard,intr 0 0

but I also tried
192.168.88.252:/repo /mnt/repos nfs _netdev,rw,hard,intr
192.168.88.252:/repo /mnt/repos nfs defaults,_netdev,rw,hard,intr

with
mnt /mnt/repos

it works.

reboot.

I don't get the NFS up at boot time.

I access to the server with SSH, eth0 and network is fine
no NetworkManager
no gui

mount -a

it works...

so to debug the issue I added this to /etc/rc.local (before exit 0)

 mount /mnt/repos > /tmp/mounting.$$ 2>&1

rebooted and I got this in that file:

mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified

I can't get NFS being mounted at boot time.
---
Architecture: amd64
DistroRelease: Ubuntu 11.04
InstallationMedia: Ubuntu-Server 11.04 "Natty Narwhal" - Release amd64 (20110426)
Package: mountall 2.25ubuntu1
PackageArchitecture: amd64
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.38-13.55-server 2.6.38.8
Tags: natty
Uname: Linux 2.6.38-13-server x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

[borninbronx (daniele-segato)]

Accordingly to TeTeT on the freenode #ubuntu-server IRC channel this bug is probably related to mountall

[borninbronx (daniele-segato)]

apport information

[Steve Langasek (vorlon)]

> so to debug the issue I added this to /etc/rc.local (before exit 0)

> mount /mnt/repos > /tmp/mounting.$$ 2>&1

> rebooted and I got this in that file:

> mount.nfs: rpc.statd is not running but is required for remote locking.
> mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
> mount.nfs: an incorrect mount option was specified

This is not really a valid test. The event-driven startup doesn't guarantee that the rpc.statd job (/etc/init/statd.conf) has started before rc.local is run.

Could you please attach /var/log/boot.log from the affected system?

[borninbronx (daniele-segato)]

Don't doubt that...
but NFS is not mounted and the problem is real..

Here the /var/log/boot.log

fsck from util-linux-ng 2.17.2
fsck from util-linux-ng 2.17.2
/dev/sda1: clean, 235/124496 files, 65983/248832 blocks
/dev/mapper/git-root: clean, 57793/474208 files, 334796/1895424 blocks
mount.nfs: Failed to resolve server 192.168.88.252: Name or service not known
mountall: mount /mnt/repos [560] terminated with status 32
 * Starting AppArmor profiles ESC[80G ^MESC[74G[ OK ]
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified
mountall: mount /mnt/repos [690] terminated with status 32

Apparently id doesn't resolve 192.168.88.252 (?)

[Steve Langasek (vorlon)]

Not disputing that it's a real problem, just saying that rc.local doesn't get us closer to understanding it :)

> Apparently id doesn't resolve 192.168.88.252 (?)

This just means that mountall first tried to mount the share before the network was fully up. This is perfectly normal; and as you can see, it tries again later in the log and gets a different error... specifically the statd error.

Is there any chance you can test this with a daily build of the 12.04 server? 12.04 has support for better logging of upstart jobs, which would enable us to see whether something is causing statd to fail to start on boot.

The way this is supposed to work is that /etc/init/statd-mounting.conf should trigger as soon as mountall tries to mount the nfs share, and this will block the mount attempt while trying to start the statd job, which in turn launches portmap and waits for it to come up before running rpc.statd. This has all shown itself to be rather reliable since 11.04 when it landed, so I don't know what's causing the problem in your case... we're really going to need some logging to figure it out.

[borninbronx (daniele-segato)]

Hi Steve,

the server is now being used with a very dirty workaround I've written:

crontab -e

@reboot /usr/bin/my-mount-script.sh

the script is really stupid, it just keep trying mount -a every 5 seconds until it found a file .checked that is present in the root directory of the NFS sharing.

and the first time it try it always get the statd error.

This is to say that I can't test on that server anymore.
If you can wait I'll be able to do tests after 22/02/2012 when this server will be taken off-line.

but I doubt that second try is performed by the bootup.. it's more like it's the log from my cron script trying.

I don't know the bootup process in details and what does print that message, but the common meaning of "Name or service not known" is DNS failure, which doesn't make sense on an IP address.

anyway I'm not alone:
http://ubuntuforums.org/showthread.php?t=1699331

do you know of anyway of making the networks logs be more verbose without upgrading the server?
I'm fine modifying the bootup scripts if you give me precise instructions

thanks

[Steve Langasek (vorlon)]

On Fri, Feb 17, 2012 at 09:09:13AM -0000, mastro wrote:
> This is to say that I can't test on that server anymore. If you can wait
> I'll be able to do tests after 22/02/2012 when this server will be taken
> off-line.

Ok, that would be fine.

> but I doubt that second try is performed by the bootup.. it's more like
> it's the log from my cron script trying.

You can take my word for it. That's how mountall works, and no output from
your cron jobs ever lands in the boot.log.

> I don't know the bootup process in details and what does print that
> message, but the common meaning of "Name or service not known" is DNS
> failure, which doesn't make sense on an IP address.

It's arguably a bug that mount.nfs tries to pass an IP address to the
resolver, but that's not the bug that's causing your problem.

> do you know of anyway of making the networks logs be more verbose without
> upgrading the server? I'm fine modifying the bootup scripts if you give
> me precise instructions

If you don't have a spare machine you can run a test from, and you aren't
prepared to upgrade this server to a pre-release version of 12.04 (which is
completely understandable), the next best thing would be to install the
upstart package from precise directly; there's no really good way to get
better logging just by modifying the boot scripts. However, installing the
upstart package will also require versions of the ifupdown and initscripts
packages from Ubuntu 11.10 or later.

Is there any reason this server is at 11.04 still? If it could be upgraded
to 11.10 and the precise upstart package installed from there, that might be
the best option.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Steve Langasek (vorlon)]

mastro, were you ever able to run the requested tests after this server was taken offline?

[borninbronx (daniele-segato)]

Hi Steve,

I'm really sorry, that server has been took out from production a while ago, and I completely forgotten about this.

last week we had some disk space issue in our multi-porpouse virtual machine manager and I get rid of old unused VMs, the one with this issue was one of them: it's gone.

I kept the workaround I used there and I report it here, it may be helpful to someone...
It's a script that start at boot time (as cron) and retry every 5 seconds to mount fstab disks until it's successfull.

I created this script in /usr/sbin/nfs-boot-mount

#!/bin/sh

LOG_DIR="/var/log/nfs-boot-mount";
LOG_FILE="${LOG_DIR}/nfs-boot-mount_`date +'%Y_%m_%d'`.log";
RETRY_IN_SEC=5;

checkAndMount() {
 echo "`date` - Try mounting nfs..." >> "${LOG_FILE}"
 mount -a >> "${LOG_FILE}" 2>&1;
 result=$?;
 if [ $result -eq 0 ] ; then
  echo "SUCCESS!" >> "${LOG_FILE}"
 else
  echo "FAILED! Retrying in ${RETRY_IN_SEC} seconds." >> "${LOG_FILE}"
 fi
 return $result;
}

loopCheckMount() {
 fail=1;
 while [ $fail -ne 0 ] ; do
  sleep ${RETRY_IN_SEC};
  checkAndMount;
  fail=$?;
 done;
}

mkdir -p "${LOG_DIR}";

checkAndMount
if [ $? -ne 0 ] ; then
 loopCheckMount &
fi

# END

gave it +x (execution permission)

then:
crontab -e

and add:
@reboot /usr/sbin/nfs-boot-mount-check

this will start the script at boot time and retry mount -a until it's successful, everything will be logged in /var/log/nfs-boot-mount/ directory.

I apologize again, I really forgot it.

[Launchpad Janitor (janitor)]

[Expired for nfs-utils (Ubuntu) because there has been no activity for 60 days.]