/etc/init/mountall.conf fails when root is mounted using NFS instead of NBD

[Dolganov Victor]

Like this:
When I switch the new chroot to nfs (still basically
according to https://help.ubuntu.com/community/UbuntuLTSP/LTSPWithoutNFS
), the thin client does boot up but then apparently cannot mount the
nfsroot, or so it seems - the last output message on the screen is from
scripts/nfs-bottom saying /root/etc/hostname can't be written because
it's a read-only filesystem, then there seems to be a call to mountall
that ends in some sort of timeout. According to the messages on the
screen from mtab, the root is already mounted on /, but it's still
trying to mount / and /tmp and fails.

[Lenin (gagarin)]

Me and other people are affected too:
http://<email address hidden>/msg37462.html

Please tell me if there's a fix/patch to try...

[Nikolaus Rath (nikratio)]

I can confirm this. The exact error message is:

One or more of the mounts listed in /etc/fstab cannot yet be mounted:
/: waiting for xxx.xxx.xxx.xxx:/opt/ltsp/i386
/tmp: waiting for (null)
Press ESC to enter a recovery shell

(xxx. etc is the correct server ip)

[Nikolaus Rath (nikratio)]

I tried to debug this a little bit more.

The initrd is executed without any problems (only some warnings because /root/etc/hostname cannot be written to) and control is passed to /sbin/init on the now NFS mounted real root fs.

The problem occurs when init executes /etc/init/mountall.conf. At this point, /etc/fstab is still empty, but mountall nevertheless tries to mount some file systems and then fails.

I changed /etc/init/mountall.conf to print out /etc/fstab before calling mountall:

script
    . /etc/default/rcS
    [ -f /forcefsck ] && force_fsck="--force-fsck"
    [ "$FSCKFIX" = "yes" ] && fsck_fix="--fsck-fix"
    [ -n "$TMPTIME" ] && tmptome="--tmptime=$TMPTIME"
    cat /etc/fstab
    exec mountall --daemon -v $force_fsck $fsck_fix $tmptime
end script

Attached is a screenshot of the output.

I guess the crucial question is now: why is mountall trying to mount something if /etc/fstab is empty? And where does it get the information about what it is supposed to mount?

[Nikolaus Rath (nikratio)]

Adding mountall package since this seems to be actually triggering the problem.

[Scott James Remnant (Canonical) (canonical-scott)]

Please run mountall --debug and attach the output

[Nikolaus Rath (nikratio)]

There is a little technical problem here. I added --debug, but there is so much output that it does not all fit into the buffer that I can reach with Shift+PgUp. I also can't redirect the output into a file,because root is mounted ro and everything else is a tmpfs (and the system is totally unresponsive after mountall, pressing Esc does not bring up a shell).

I recorded a video of the output instead, but it is possible that it scrolls so fast that some information is still missing. Please let me know if there is a better way to save the debug output.

[Nikolaus Rath (nikratio)]

Ok, I think I came up with a solution to capture the output. I'm writing it onto a local harddrive, but for the redirection to work I had to remove the --daemon option:

script
    . /etc/default/rcS
    [ -f /forcefsck ] && force_fsck="--force-fsck"
    [ "$FSCKFIX" = "yes" ] && fsck_fix="--fsck-fix"
    [ -n "$TMPTIME" ] && tmptome="--tmptime=$TMPTIME"
    mount -o sync /dev/sda1 /mnt
    cat /etc/fstab
    exec mountall --debug $force_fsck $fsck_fix $tmptime > /mnt/stdout 2> /mnt/stderr
end script

[Nikolaus Rath (nikratio)]

I have created two logfiles of mountall --debug for direct comparison. The first one is with root mounted as NFS (mountall hangs) and the second is with root mounted as NBD (mountall works).

[Nikolaus Rath (nikratio)]

I managed to come up with a workaround.

The following patch skips the entire ordinary NFS mounting procedure and only replaces NBD with NFS in the ltsp scripts. That means that the root folder will be a unionfs of /cow and /rofs, but /rofs is not mounted with NBD but NFS.

The patch has to be applied to /usr/share/initramfs-tools/scripts/ltsp_nbd inside the ltsp chroot, and in addition to that the PXE configuration should look exactly look like the one for NBD, i.e. the boot=nfs and nfsroot= options should *not* be given.

--- ltsp_nbd.bak 2010-01-05 15:32:43.232058101 -0500
+++ ltsp_nbd 2010-01-05 15:40:16.196557912 -0500
@@ -27,7 +27,7 @@
 # NBD root mounting
 mountroot()
 {
- [ "$quiet" != "y" ] && log_begin_msg "Running /scripts/nfs-top"
+ [ "$quiet" != "y" ] && log_begin_msg "Running /scripts/ltsp_nbd"
     run_scripts /scripts/nfs-top
     [ "$quiet" != "y" ] && log_end_msg

@@ -67,16 +67,7 @@
     mount -t tmpfs tmpfs /cow

     # mount the readonly root
- if nbd-client ${NBD_ROOT_SERVER} ${NBD_ROOT_PORT} /dev/nbd0 ; then
- eval $(fstype /dev/nbd0)
- if [ -z "$FSTYPE" ]; then
- echo "Warning: unable to detect filesystem, assuming squashfs."
- FSTYPE=squashfs
- fi
- mount -o ro -t $FSTYPE /dev/nbd0 /rofs
- else
- panic "Error: failed to connect to NBD server."
- fi
+ mount -o ro,nolock -t nfs ${NBD_ROOT_SERVER}:/opt/ltsp/i386 /rofs

     # merge ro and rw filesystems
     mount -t aufs -o dirs=/cow=rw:/rofs=ro aufs ${rootmnt}

[Nikolaus Rath (nikratio)]

Launchpad messed up the whitespace, here is the patch again.

[Bettina Lapp (bettina-lapp)]

Thank you very much for this workaround, I can confirm it works, my thinclients now boot successfully from a 9.10 chroot with NFS.
To clarify perhaps, if you use this workaround patch, do not change /opt/ltsp/i386/etc/default/ltsp-client-setup (i.e. leave root_write_method=" "), and do not create /opt/ltsp/i386/etc/initramfs-tools/conf.d/ltsp resp. delete it if it exists.

[Nikolaus Rath (nikratio)]

Indeed, I forgot to mention that. Thanks for pointing that out.

But to be even more precise: initramfs-tools/conf.d/ltsp does not need to be deleted, it just should not set BOOT=nfs.

[Eelco Nieuwstad (eelcon)]

I also seem to have this problem but do not have installed the ltsp packages. I just did everything from scratch.

[Nikolaus Rath (nikratio)]

Note that it is also required to prevent /etc/init.d/umountfs and /etc/init.d/umountnfs.sh from unmounting /rofs and /cow, otherwise the client will freeze when trying to shutdown or restart.

[Nikolaus Rath (nikratio)]

Bug #507613 is most likely about the same underlying issue.

[Aron Griffis (agriffis)]

I don't use LTSP but I've found two major problems using nfsroots on Lucid, and probably Karmic as well:

1. By default NFS wants locking, which creates a circular dependency: rootfs -> statd -> portmap -> local-filesystems event -> rootfs -> statd -> etc.

2. To work around that problem, you can put "nolock" in /etc/fstab for the rootfs. However this runs into a bug in mountall where it doesn't notice the rootfs has been successfully mounted, so it will loop forever trying to finish mounting the local filesystems.

These two problems are described in more detail in bug #537133 with a patch that fixes the second issue.

[Scott James Remnant (Canonical) (canonical-scott)]

The mountall portion of this bug is better described by bug #537133 - however there's no way to only mark that task a dup

[Alkis Georgopoulos (alkisg)]

After the mountall bug #537133 was fixed, the method that was described previously in the https://help.ubuntu.com/community/UbuntuLTSP/LTSPWithoutNFS wiki page again works for me. Does anyone still experience this? Marking as incomplete...

The patch cannot be accepted in the LTSP code as it breaks nbd booting. Instead, maybe /opt/ltsp/i386/usr/share/initramfs-tools/conf.d/ltsp can be modified so that it automatically detects the BOOT variable:
MODULES="netboot"
BOOT="ltsp_nbd"
for p in $(cat /proc/cmdline 2>/dev/null); do
    case $p in
        boot=*)
            unset BOOT
            break
        ;;
    esac
done
COMPCACHE_SIZE="25%"

When unset, init will set BOOT=nfs if root=/dev/nfs is passed in the kernel command line. So this way that file doesn't need to be modified when one wants to boot an LTSP client with NFS.

[Alkis Georgopoulos (alkisg)]

The nfs-bottom/ltsp script was dropped in LTSP 5.3 in favor of the new custom init=/sbin/init-ltsp approach. NFS bind-mounting is no longer supported, but NFS+tmpfs+overlayfs mounting will probably be available in the future.