Rootfs fails to unmount cleanly after upstart upgrade

[Ed Swierk]

Binary package hint: upstart

I have an automatic VM provisioning script that clones a Maverick UEC VM image, boots it, runs apt-get upgrade, and shuts it down.

The root filesystem fails to unmount cleanly during this last step, apparently because pid 1 is holding references to files that were deleted during apt-get upgrade.

Inserting lsof / into /etc/init.d/umountroot shows:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
init 1 root cwd DIR 252,0 4096 2 /
init 1 root rtd DIR 252,0 4096 2 /
init 1 root txt REG 252,0 125640 4081 /sbin/init
init 1 root DEL REG 252,0 1997 /lib/libnss_files-2.12.1.so
init 1 root DEL REG 252,0 1999 /lib/libnss_nis-2.12.1.so
init 1 root DEL REG 252,0 1994 /lib/libnsl-2.12.1.so
init 1 root DEL REG 252,0 1995 /lib/libnss_compat-2.12.1.so.dpkg-new
init 1 root DEL REG 252,0 1962 /lib/libc-2.12.1.so
init 1 root DEL REG 252,0 2019 /lib/librt-2.12.1.so
init 1 root DEL REG 252,0 2016 /lib/libpthread-2.12.1.so
init 1 root mem REG 252,0 269104 1968 /lib/libdbus-1.so.3.5.2
init 1 root mem REG 252,0 34792 1992 /lib/libnih-dbus.so.1.0.0
init 1 root mem REG 252,0 92144 1993 /lib/libnih.so.1.0.0
init 1 root DEL REG 252,0 1952 /lib/ld-2.12.1.so

A comment in the umountroot script mentions the need for init to re-exec itself to release open files.

# Ask init to reexec itself before we go down if it has been
# upgraded this cycle. It'll lose all its state, but at least
# it won't hold open files on the root filesystem
[ -f /var/run/init.upgraded ] && telinit u || :

Two problems with this:

- /var/run/init.upgraded doesn't exist at this point; either upstart isn't creating it, or someone else is deleting it.

- telinit u seems to have no effect; likewise kill -TERM 1 (or -QUIT or -KILL)

[Ed Swierk (eswierk)]

$ lsb_release -rd
Description: Ubuntu 10.10
Release: 10.10
$ dpkg -l | grep -e upstart -e sysvinit
ii sysvinit-utils 2.87dsf-4ubuntu18 System-V-like utilities
ii upstart 0.6.6-3 event-based init daemon