NFS not mounted

[stefand]

Binary package hint: usplash

Hi,
I have my home directory is mounted via NFS from an network attached storage (NFS)

the line looks like this:
192.168.178.100:/mnt/md1/home /home nfs rw,hard,intr 0 0

Just until yesterday everything worked find. But the I updated to the latest Karmic Koala release.

No everything hangs with the following messages
inotify_add_watch(6, (null) 10) bad address // reported multiple times
/etc/udev/rules/z60_hdparam.rule cat not be read

DNS resolution faild 192.168.178.100

Good luck,
Stefan

Linux lianli 2.6.31-9-generic #29-Ubuntu SMP Sun Aug 30 17:39:26 UTC 2009 x86_64 GNU/Linux

# Usplash configuration file
# These parameters will only apply after running update-initramfs.

xres=1280
yres=1024

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by /usr/sbin/grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
load_env
set default=0
if [ ${prev_saved_entry} ]; then
  saved_entry=${prev_saved_entry}
  save_env saved_entry
  prev_saved_entry=
  save_env prev_saved_entry
fi
insmod ext2
set root=(hd0,1)
search --no-floppy --fs-uuid --set 051c6b44-34f7-441b-8ec9-2325dc765680
if loadfont /usr/share/grub/unicode.pf2 ; then
  set gfxmode=640x480
  insmod gfxterm
  insmod vbe
  if terminal_output gfxterm ; then true ; else
    # For backward compatibility with versions of terminal.mod that don't
    # understand terminal_output
    terminal gfxterm
  fi
fi
set timeout=10
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=white/black
set menu_color_highlight=black/white
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux ###
menuentry "Ubuntu, Linux 2.6.31-9-generic" {
 set quiet=1
 insmod ext2
 set root=(hd0,1)
 search --no-floppy --fs-uuid --set 051c6b44-34f7-441b-8ec9-2325dc765680
 linux /boot/vmlinuz-2.6.31-9-generic root=UUID=051c6b44-34f7-441b-8ec9-2325dc765680 ro quiet splash
 initrd /boot/initrd.img-2.6.31-9-generic
}
menuentry "Ubuntu, Linux 2.6.31-9-generic (recovery mode)" {
 insmod ext2
 set root=(hd0,1)
 search --no-floppy --fs-uuid --set 051c6b44-34f7-441b-8ec9-2325dc765680
 linux /boot/vmlinuz-2.6.31-9-generic root=UUID=051c6b44-34f7-441b-8ec9-2325dc765680 ro single
 initrd /boot/initrd.img-2.6.31-9-generic
}
### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_memtest86+ ###
menuentry "Memory test (memtest86+)" {
 linux16 /boot/memtest86+.bin
}
menuentry "Memory test (memtest86+, serial console 115200)" {
 linux16 /boot/memtest86+.bin console=ttyS0,115200n8
}
### END /etc/grub.d/20_memtest86+ ###

### BEGIN /etc/grub.d/30_os-prober ###
if [ ${timeout} != -1 ]; then
  if keystatus; then
    if keystatus --shift; then
      set timeout=-1
    else
      set timeout=0
    fi
  else
    if sleep --interruptible 3 ; then
      set timeout=0
    fi
  fi
fi
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

ProblemType: Bug
Architecture: amd64
Date: Thu Sep 17 10:20:42 2009
DistroRelease: Ubuntu 9.10
Package: usplash 0.5.37
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.31-9-generic root=UUID=051c6b44-34f7-441b-8ec9-2325dc765680 ro quiet splash
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-9.29-generic
SourcePackage: usplash
Uname: Linux 2.6.31-9-generic x86_64
UsplashConf:
 # Usplash configuration file
 # These parameters will only apply after running update-initramfs.

 xres=1280
 yres=1024
dmi.bios.date: 01/12/2009
dmi.bios.vendor: Intel Corp.
dmi.bios.version: MJG4110H.86A.0002.2009.0112.1041
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: DG41MJ
dmi.board.vendor: Intel Corporation
dmi.board.version: AAE54659-203
dmi.chassis.type: 3
dmi.modalias: dmi:bvnIntelCorp.:bvrMJG4110H.86A.0002.2009.0112.1041:bd01/12/2009:svn:pn:pvr:rvnIntelCorporation:rnDG41MJ:rvrAAE54659-203:cvn:ct3:cvr:

[Scott James Remnant (Canonical) (canonical-scott)]

The problem isn't so much that mountall doesn't mount them, but it seems to be related to portmap and nfs-common services. Since your NFS mount is one that mountall considers required for boot, it will wait for it before allowing the boot to proceed.

[meh (meh-deactivatedaccount)]

Same problem with NFS mountpoints from fstab here.

Upon booting the message "couldn't resolve <local hostname>" appears.
After that, nfs-common, and thus rpc.statd, is started, so it looks like mountall is
a) running too early in the boot process
or
b) requires a 2nd run when requirements are fulfilled

[stefand (stefandirnstorfer)]

Well, I don't really need this network drive at boot time. After all, it's the home directory.

I didn't change anything in the boot process. I largly lack the skill to do so. It's just a plain Karmic install and update.

My system might be a bit unusual though. I boot it from an SSD, which gives me a login screen within seconds. The home directory is mounted from a network drive via nfs, hence much slower than a standard setting.

[sancelot (sancelot)]

same problem after migration in a "standard" computer

it seems to be a bad order in the boot sequence

nfs is not mounted at boot time

I have to launch /etc/init.d/nfs-common restart
and
 mount manually

[Scott James Remnant (Canonical) (canonical-scott)]

Note that mountall is just here for tracking, the fix is needed for the nfs and portmap packages

[l8gravely (john-stoffel)]

I've run into this issue too, but I mount two paths from my home NFS server, which is a Debian Testing box. I mount /home and /local. If I comment out /home from my /etc/fstab, but leave in /local, the system will boot.

But if I put in just /home, or both /home and /local, then it won't boot properly. One thing I've noticed is that my system also seems to be screwed up network wise, so I wonder if this issue is really due to Network-Manager screwing up /etc/resolv.conf or other networking config files, which causes the NFS mounts to fail.

I've updated my system as of Sept 28, 2009 at 9pm EST and it's still not working properly. Though I'll do some more reboots and tests in a few to see what I can do. It would be nicer if mountall was *much* more explicit about what it depends on and what it thinks it needs to be mounted properly before the system can reboot.

I have mountal 0.1.8 installed. Here's my /etc/fstab file:

# /etc/fstab: static file system information.
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
# /dev/sda2
UUID=31e678a8-1ad7-4293-b3cf-c58ff4bc8c6f / ext3 defaults,errors=remount-ro 0 1
# /dev/sda1
UUID=026f5609-313d-4ec4-80a4-7d45f96b9f7a /boot ext3 defaults 0 2
/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec 0 0
#jfsnew:/home /home nfs soft,sec=sys,vers=3,tcp,rsize=32768,wsize=32768 0 0
jfsnew:/local /local nfs soft,sec=sys,vers=3,tcp,rsize=32768,wsize=32768 0 0
none /proc/bus/usb usbfs defaults 0 0
#none /sys/kernel/config configfs defaults 0 0

Scott, can you be more explicit about what changes you think are needed in the nfs and portmap packages?

Thanks,
John

[l8gravely (john-stoffel)]

Ooops, I'm a moron. When I have /local enabled in my /etc/fstab, it does NOT mount properly, but the mount process doesn't hang either.

It's complains about rpc.statd not being started properly, but I need to enable bootlogd and do some testing to get better logs of the boot process and what happens.

More in a few...
John

[Steve Langasek (vorlon)]

Here's a preliminary patch to upstartify portmap.

The 'stopped nfs-common' is incorrect, both because there are multiple daemons involved with nfs-common that will each need their own upstart job (AFAICS), and because it should be stopped in cases where nfs-common is not available.

Also, this upstart job loses the sendsigs handling found in the init script, so on shutdown I see the job being killed multiple times - which could also cause problems for NFS mounts. I don't know if there's a better way to implement this besides duplicating the same sendsigs logic found in the init script.

[Steve Langasek (vorlon)]

And here is a patch for the nfs-common side of things. This is dependent on debhelper and upstart changes so that we get sensible restart-after-upgrade behavior for these services.

The problem of when to stop the service is also not completely solved here and needs more attention.

[Steve Langasek (vorlon)]

Updated nfs-utils patch superseding the previous, now with bugs shaken out so upstart is able to actually manage the jobs. :)

[Launchpad Janitor (janitor)]

This bug was fixed in the package portmap - 6.0-10ubuntu1

---------------
portmap (6.0-10ubuntu1) karmic; urgency=low

  * Convert portmap to upstart. LP: #431248.

 -- Steve Langasek <email address hidden> Tue, 29 Sep 2009 09:27:48 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nfs-utils - 1:1.2.0-2ubuntu1

---------------
nfs-utils (1:1.2.0-2ubuntu1) karmic; urgency=low

  * Drop nfs-common init script in favor of new upstart jobs. LP: #431248.
  * Build-depend on debhelper (>= 7.3.15ubuntu3) for correct upstart init
    handling.
  * Depend the upstart-using version of portmap, 6.0-10ubuntu1; and drop the
    alternative depends on rpcbind, which hasn't been converted.

 -- Steve Langasek <email address hidden> Fri, 02 Oct 2009 19:23:19 +0000

[l8gravely (john-stoffel)]

I must be stupid, but I don't see the nfs-utils package anywhere. I did install nfs-kernel-server package, which is now at version: nfs-kernel-server/karmic uptodate 1:1.2.0-2

But it didn't make any difference. NFS directories don't mount properly on bootup still. And I've done a full update and upgrade of my system as of 10PM EST, 10/2/2009. Do I need to add in a new apt repository for testing?

Thanks,
John

[Steve Langasek (vorlon)]

On Sat, Oct 03, 2009 at 02:09:38AM -0000, l8gravely wrote:
> I must be stupid, but I don't see the nfs-utils package anywhere. I did
> install nfs-kernel-server package, which is now at version: nfs-kernel-
> server/karmic uptodate 1:1.2.0-2

> But it didn't make any difference. NFS directories don't mount properly
> on bootup still. And I've done a full update and upgrade of my system
> as of 10PM EST, 10/2/2009. Do I need to add in a new apt repository for
> testing?

nfs-utils is the source package. You need to install the nfs-common
package, at the version shown in the bug closure message 1:1.2.0-2ubuntu1.
And it may not be available on your mirror yet, publishing is slow today due
to load from the beta release.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Douglas Thrift (douglaswth)]

I'm seeing configuration errors after upgrading to this new version:

Setting up nfs-common (1:1.2.0-2ubuntu1) ...
 * Stopping NFS common utilities
   ...done.
 Removing any system startup links for /etc/init.d/nfs-common ...
   /etc/rc0.d/K20nfs-common
   /etc/rc1.d/K20nfs-common
   /etc/rc2.d/S20nfs-common
   /etc/rc3.d/S20nfs-common
   /etc/rc4.d/S20nfs-common
   /etc/rc5.d/S20nfs-common
   /etc/rc6.d/K20nfs-common
   /etc/rcS.d/S44nfs-common
statd start/running, process 14751
start: Job failed to start
invoke-rc.d: initscript gssd, action "restart" failed.
dpkg: error processing nfs-common (--configure):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Errors were encountered while processing:
 nfs-common
E: Sub-process /usr/bin/dpkg returned an error code (1)
A package failed to install. Trying to recover:
Setting up nfs-common (1:1.2.0-2ubuntu1) ...
 * Stopping NFS common utilities
   ...done.
 Removing any system startup links for /etc/init.d/nfs-common ...
statd start/running, process 14985
start: Job failed to start
invoke-rc.d: initscript gssd, action "restart" failed.
dpkg: error processing nfs-common (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 nfs-common

[Steve Langasek (vorlon)]

Yes, this upgrade issue has been reported as bug #441055, and a fix is in progress (will be uploaded as version -2ubuntu2).

[l8gravely (john-stoffel)]

Thanks for the fixes, reboots when /home or other filesystems are mounted via NFS now works properly. I still see a bunch of errors on startup, but they must be ignored or re-tried. Maybe I'll see how it works when the network is unplugged on bootup, whether it hangs the system still or not.

Thanks!
John

[Jonas Bonn (jonas.bonn)]

There's still a problem somewhere... seems to be a timing issue. Occasionally NFS mount still fails because rpc.statd is not running yet.... happens about half the time. I'll attach a screenshot.

[Jonas Bonn (jonas.bonn)]

I have a hunch that the resolution to bug #447654 will also resolve the issue that I reported above as a "timing issue". Can't test until Monday, though, so I'll get back with more info then.

[Steve Langasek (vorlon)]

Hi Jonas,

No, your issue is probably not related to bug #447654. I believe your problem is fixed with the latest upload of nfs-utils, which arranges for statd to be started (if not already running) whenever we are asked to mount an NFS filesystem.

 nfs-utils (1:1.2.0-2ubuntu5) karmic; urgency=low
 .
   * Set upstart jobs to also start on mount attempt, in the event that
     mountall gets to them before the daemons are done starting. Really-fixes
     LP: #431248.
   * Call 'stop' in the pre-start scripts for all jobs when we want to prevent
     the job from starting; this lets upstart know that it's a clean stop,
     and avoids boot-time messages about service start failures

In your case, since you're netbooting it's entirely possible that it's not a timing issue at all - depending on how your filesystems and network devices are set up, the signal might never have arrived that would retry the mount on your system after statd was started. With the above change, though, I think it should work reliably for you.

[l8gravely (john-stoffel)]

Quick update, I rebooted yesterday after doing a bunch of updates, into linux kernel 2.6.32-rc3 (my own compiled version) and it hung on bootup again, not being able to mount /home and /local via NFS. It sat there for *hours* since I did the reboot remotely.

It would be really much nicer if mountall would just timeout and continue the boot, even if it can't mount /home or other filesystems. Sure, if it can't mount / or /usr it should fail, but those are also easier to notice. Hanging on NFS mounts not working is painful.

Now, when I did reboot again, I used 2.6.31-11-generic and it booted, so there's something in my kernel config that needs tweaking as well. Any hints? I'm going to try 2.6.32-rc4 in a few, more details when I see what happens.

John

[Steve Langasek (vorlon)]

John,

It is a design decision that a missing /home blocks the boot; this is unlikely to be changed.

However, with the latest mountall, you can opt to mark your /home filesystem with the 'nobootwait' option, indicating to mountall not to wait for this before proceeding with the boot.

Problems with 2.6.32's nfs support are not relevant to karmic, so this is not the right place to discuss them.

[flowerdealer (pixelcowboy79)]

We also have this problem, it is really annoying, as boot takes quite a long time. Any fix for it coming soon?

[Steve Langasek (vorlon)]

You've followed up to a bug that has already been fixed. If you think there's still a problem, you should file a new bug report.

[Alvin (alvind)]

Yes, this one is fixed. I think the bug you're looking for is Bug #461133. It has similar symptoms. (crashing the boot process if you mount /home over NFS.)

[raliegh (steve-ubuntu-sr-tech)]

Steve Langasek wrote "It is a design decision that a missing /home blocks the boot; this is unlikely to be changed."

So what about folks who boot root over nfs? If the changes in Karmic break systems with core filesystems mounted via nfs shares, what is the logic of the "design decision"?

[Steve Langasek (vorlon)]

The changes in karmic /don't/ break the ability to mount core filesystems via nfs shares.

[Alvin (alvind)]

Maybe they don't, but there's definitely a difference. I quote a boot with NFS mounted /home

mount.nfs4: DNS resolution failed for leto: Name or service not known
mountall: mount /home [901] terminated with status 32
mountall: Filesystem could not be mounted: /home
One or more of the mounts listed in /etc/fstab cannot yet be mounted:
(ESC for recovery shell)
/home: waiting for leto:/home
 * Setting preliminary keymap
.... and so on

Please, don't let Ubuntu tell that /home is not mounted when that is not true. The same message appears, when the mounting does not succeed. Does that make sense?

[Steve Langasek (vorlon)]

Alvin,

Please file a separate bug report against mountall for this.

[raliegh (steve-ubuntu-sr-tech)]

I was having a difficult time getting a root over nfs system to boot after several attempts to distro upgrade to 9.10. Having it explicitly stated that nfs core mounts are not broken, I gave it a few more tries, but finally today got 9.10 booting by first doing a fresh install and full update onto the harddrive in a typical workstation, then tar'ed up the filesystem, extracted it onto the server, made the necessary fs tweaks and it booted! I still get a dozen "inotify_add_watch(6, (null) 10) bad address" lines during boot, but it appears harmless.

The downside is that I had to reinstall a bunch of apps, but it took less time overall compared to a distro upgrade, and I ended up with a nice clean system. I should have tried it sooner :)

[Alvin (alvind)]

ok, the error message issue is reported as bug 504224.