rpc.statd is killed and restarted during shutdown multiple times

[Ernst Kloppenburg]

the following message appears during system shutdown at least two times:
"statd main process ended" "respawning"

this looks like some inconsistency in the upstart configuration of nfs-common

ProblemType: Bug
Architecture: i386
Date: Mon Nov 9 13:56:30 2009
DistroRelease: Ubuntu 9.10
NonfreeKernelModules: fglrx
Package: nfs-common 1:1.2.0-2ubuntu8
ProcEnviron:
 LANGUAGE=
 PATH=(custom, user)
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-14.48-generic
SourcePackage: nfs-utils
Uname: Linux 2.6.31-14-generic i686

[Jonathan Michalon (johndescs)]

Same here… but only one time at shutdown (maybe it hasn't time to get up and down two times).
May this indicate an incorrect stop condition ?

[bamyasi (iadzhubey)]

Same here except when it happens upon shutdown there is a 50% chance of it causing a kernel NULL pointer trap in reiserfs module, forcing full filesystem scan on the next boot. This on average takes 4 hours on my 6 TB RAID volume, very annoying. I basically cannot shutdown my server other than by means of MagicKey+REISUB sequence. Is there any chance this will be ever fixed? Otherwise I will be forced to install some other distribution, I really can't afford 4 hours of server downtime each time I need to bring it down for a scheduled maintenance.

[Ernst Kloppenburg (ernst-kloppenburg)]

In my original bug report I wrote "this looks like some inconsistency in the upstart configuration of nfs-common"

Now I do not think this is true. It rather looks like something is killing statd during shutdown without using an upstart event. Therefore upstart tries to restart the job a number of times.

I would like to debug this. Are there any suggestions on how I could do this?

[Ernst Kloppenburg (ernst-kloppenburg)]

1) analysis: this is what is going wrong w.r.t. nfs-utils during shutdown

# the script S20sendsigs in /etc/rc0.d/ is run
-> the call to killall5 -15 ... kills rpc.statd

# at the same time upstart tries to keep rpc.statd alive due to "respawn" in /etc/init/statd.conf
-> rpc.statd is restarted

These two conflicting actions repeat at least two times due to S20sendsigs containing multiple killall5 invocations. Maybe more often when upstart does all this in parallel.

2) possible solutions (I know too less to get this right myself, but post three suggestions)

a) if rpc.statd is needed as long as S31umountnfs.sh has not yet been run
  then it should be protected from the killall5 like portmap is (/lib/init/rw/sendsigs.omit.d/portmap)

b) if rpc.statd is no longer needed at that time it should get a statement like
   "stop on (stopping portmap or runlevel [06])"
  in /etc/init.statd.conf
  so that it is no longer present when S20sendsigs runs

c) or, maybe, the respawn mechanism of upstart should be deactivated before running S20sendsigs

[Steve Langasek (vorlon)]

As it happens, the sendsigs script in sysvinit has just been corrected in lucid to not kill processes belonging to upstart jobs. So I believe this is now resolved for 10.04.

[Ernst Kloppenburg (ernst-kloppenburg)]

as a workaround for karmic, the
   /etc/init.d/sendsigs
from the lucid package can be used. It works fine here. See attachment.

[Zisu Andrei (matzipan)]

I have a similar problem which does not allow me to shutdown, but instead statd, it gives an error on rsyslog, upstart-udev-bridge, ssh (which also gives status 255), udev, rc, plymouth (which gives a status 1).

I created a dir and a file in /var/run/sendsigs.omit.d with the name rsyslog with no result.

Reboot works ok.

[Zisu Andrei (matzipan)]

I installed a version of 10.04 which O downloaded a couple of weeks ago. My computer is buggy so the install didn't succeed, instead I took the HDD and installed on another computer, and then changed the bootloader and network interfaces to fit the new computer.

[Zisu Andrei (matzipan)]

That's strange, sudo shutdown -P now works like a charm.