mountall not actually mounting and blocking boot

[Brian J. Murrell]

Binary package hint: mountall

I've just debootstrapped a lucid rootfs but it hangs on boot. Adding "--debug --verbose" produces the following debug during boot:

Begin: Running /scripts/init-bottom ...
Done.
init: Handling startup event
init: console_getty goal changed from stop to start
init: console_getty state changed from waiting to starting
init: mountall goal changed from stop to start
init: mountall state changed from waiting to starting
init: hostname goal changed from stop to start
init: hostname state changed from waiting to starting
init: Handling starting event
init: console_getty state changed from starting to pre-start
init: console_getty state changed from pre-start to spawned
init: console_getty main process (143)
init: console_getty state changed from spawned to post-start
init: console_getty state changed from post-start to running
init: Handling starting event
init: plymouth goal changed from stop to start
init: plymouth state changed from waiting to starting
init: hwclock goal changed from stop to start
init: hwclock state changed from waiting to starting
init: ureadahead goal changed from stop to start
init: ureadahead state changed from waiting to starting
init: Handling starting event
init: hostname state changed from starting to pre-start
init: hostname state changed from pre-start to spawned
init: hostname main process (144)
init: hostname state changed from spawned to post-start
init: hostname state changed from post-start to running
init: Handling started event
init: Handling starting event
init: plymouth state changed from starting to pre-start
init: plymouth state changed from pre-start to spawned
init: plymouth main process (145)
init: Handling starting event
init: hwclock state changed from starting to pre-start
init: hwclock state changed from pre-start to spawned
init: hwclock main process (146)
init: hwclock state changed from spawned to post-start
init: hwclock state changed from post-start to running
init: Handling starting event
init: ureadahead state changed from starting to pre-start
init: ureadahead state changed from pre-start to spawned
init: ureadahead main process (147)
init: Handling started event
init: Handling started event
init: hostname main process (144) exited normally
init: hostname goal changed from start to stop
init: hostname state changed from running to stopping
init: Handling stopping event
init: hostname state changed from stopping to killed
init: hostname state changed from killed to post-stop
init: hostname state changed from post-stop to waiting
init: Handling stopped event
init: plymouth main process (145) executable changed
init: hwclock main process (146) exited normally
init: hwclock goal changed from start to stop
init: hwclock state changed from running to stopping
init: Handling stopping event
init: hwclock state changed from stopping to killed
init: hwclock state changed from killed to post-stop
init: hwclock state changed from post-stop to waiting
init: Handling stopped event
init: plymouth main process (145) became new process (148)
init: plymouth state changed from spawned to post-start
init: plymouth post-start process (149)
init: ureadahead main process (147) became new process (150)
init: ureadahead state changed from spawned to post-start
init: ureadahead state changed from post-start to running
init: Handling started event
init: plymouth post-start process (149) exited normally
init: plymouth state changed from post-start to running
init: mountall state changed from starting to pre-start
init: mountall state changed from pre-start to spawned
init: mountall main process (155)
init: Handling started event

And the boot hangs there and goes no further.

I added a job in /etc/init to put a tty on the console so that I can inspect the system at the point that it's hanging. Here's what I can report about it:

# cat /etc/fstab:
proc /proc proc defaults 0 0
#/dev/xvda / ext3 defaults,errors=remount-ro 0 1
/dev/xvda / ext3 defaults 0 1
#/dev/xvdb /mnt ext3 defaults 0 0
#/dev/xvdc swap swap defaults 0 0

# cat /proc/cmdline:
 root=/dev/xvda --debug --verbose

# ls -l /dev/root
ls: cannot access /dev/root: No such file or directory

# cat /proc/self/mount
mountinfo mounts mountstats
root@brian-laptop:~# cat /proc/self/mountinfo
15 19 0:0 / /sys rw,nosuid,nodev,noexec,relatime - sysfs none rw
16 19 0:3 / /proc rw,nosuid,nodev,noexec,relatime - proc none rw
17 19 0:5 / /dev rw,relatime - devtmpfs none rw,size=249440k,nr_inodes=62360,mode=755
18 17 0:12 / /dev/pts rw,nosuid,noexec,relatime - devpts none rw,gid=5,mode=620,ptmxmode=000
19 1 202:0 / / ro,relatime - ext3 /dev/xvda ro,errors=continue,data=ordered

# initctl list
mountall-net stop/waiting
rc stop/waiting
rsyslog start/running, process 195
tty4 stop/waiting
udev stop/waiting
upstart-udev-bridge stop/waiting
ureadahead-other stop/waiting
console-setup stop/waiting
hwclock-save stop/waiting
plymouth-log stop/waiting
tty5 stop/waiting
plymouth start/running, process 148
console_getty start/running, process 143
control-alt-delete stop/waiting
hwclock stop/waiting
module-init-tools stop/waiting
cron stop/waiting
mountall start/running, process 156
plymouth-stop stop/waiting
rcS stop/waiting
mounted-varrun stop/waiting
rc-sysinit stop/waiting
tty2 stop/waiting
udevtrigger stop/waiting
mounted-dev stop/waiting
tty3 stop/waiting
udev-finish stop/waiting
hostname stop/waiting
mountall-reboot stop/waiting
mountall-shell stop/waiting
mounted-tmp stop/waiting
network-interface stop/waiting
plymouth-splash stop/waiting
tty1 stop/waiting
udevmonitor stop/waiting
dmesg stop/waiting
network-interface-security stop/waiting
networking stop/waiting
procps stop/waiting
tty6 stop/waiting
ureadahead stop/waiting

# touch /foor
touch: cannot touch `/foor': Read-only file system

So mountall didn't in fact (re-)mount / (rw). If I run mountall by hand, it will:

# mountall --verbose
Connected to Plymouth
/ is local
/proc is virtual
/sys is virtual
/sys/fs/fuse/connections is virtual
/sys/kernel/debug is virtual
/sys/kernel/security is virtual
/dev is virtual
/dev/pts is virtual
/dev/shm is virtual
/tmp is local
/var/run is virtual
/var/lock is virtual
/lib/init/rw is virtual
swap finished
local 0/2 remote 0/0 virtual 1/11 swap 0/0
local 0/2 remote 0/0 virtual 2/11 swap 0/0
local 0/2 remote 0/0 virtual 3/11 swap 0/0
local 0/2 remote 0/0 virtual 4/11 swap 0/0
mounting /sys/fs/fuse/connections
mount /sys/fs/fuse/connections [201] exited normally
local 0/2 remote 0/0 virtual 5/11 swap 0/0
mounting /sys/kernel/debug
mount /sys/kernel/debug [202] exited normally
local 0/2 remote 0/0 virtual 6/11 swap 0/0
mounting /sys/kernel/security
mount /sys/kernel/security [203] exited normally
local 0/2 remote 0/0 virtual 7/11 swap 0/0
mounting /dev/shm
mount /dev/shm [204] exited normally
local 0/2 remote 0/0 virtual 8/11 swap 0/0
mounting /var/run
mount /var/run [205] exited normally
local 0/2 remote 0/0 virtual 9/11 swap 0/0
mounting /var/lock
mount /var/lock [210] exited normally
local 0/2 remote 0/0 virtual 10/11 swap 0/0
mounting /lib/init/rw
mount /lib/init/rw [211] exited normally
virtual finished
remote finished
local 0/2 remote 0/0 virtual 11/11 swap 0/0
checking /
fsck from util-linux-ng 2.17.2
ROOT: clean, 17194/262144 files, 179348/1048576 blocks
fsck / [218] exited normally
remounting /
mount / [225] exited normally
mount / [310] exited normally
mount /proc [321] exited normally
mount /sys [324] exited normally
mount /sys/fs/fuse/connections [325] exited normally
mount /sys/kernel/debug [327] exited normally
mount /sys/kernel/security [328] exited normally
mount /dev [329] exited normally
mount /dev/pts [331] exited normally
mount /dev/shm [332] exited normally
mount /var/run [334] exited normally
mount /var/lock [336] exited normally
mount /lib/init/rw [337] exited normally
local 1/2 remote 0/0 virtual 11/11 swap 0/0
local finished
All filesystems mounted
local 2/2 remote 0/0 virtual 11/11 swap 0/0
init: plymouth main process (148) exited normally
init: plymouth goal changed from start to stop
init: plymouth state changed from running to stopping
init: Handling stopping event
init: plymouth state changed from stopping to killed
init: plymouth state changed from killed to post-stop
init: plymouth state changed from post-stop to waiting
init: Handling stopped event
init: plymouth-stop goal changed from start to stop
init: plymouth-stop pre-start process (488) exited normally
init: plymouth-stop state changed from pre-start to stopping
init: Handling stopping event
init: plymouth-stop state changed from stopping to killed
init: plymouth-stop state changed from killed to post-stop
init: plymouth-stop state changed from post-stop to waiting
init: Handling stopped event

And now / is (re-)mounted (rw):

# touch /foor
#

And /dev/root is created:

# ls -l /dev/root
lrwxrwxrwx 1 root root 4 Sep 18 15:09 /dev/root -> xvda

So, what's going on here?

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: mountall 2.15.2
ProcVersionSignature: Ubuntu 2.6.32-24.42-generic 2.6.32.15+drm33.5
Uname: Linux 2.6.32-24-generic i686
NonfreeKernelModules: nvidia
Architecture: i386
Date: Sat Sep 18 10:57:17 2010
ProcEnviron:
 PATH=(custom, user)
 LANG=en_CA.utf8
 SHELL=/bin/bash
SourcePackage: mountall

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.