overlayfs: on-boot, mount device issue.

[Mike Mestnik]

Hello example fstab entries I've tested,

overlayfs /usr overlayfs defaults,lowerdir=/.usr-ro/local/..,upperdir=/.usr-rw/lost+found/.. 0 2
none /usr overlayfs defaults,lowerdir=/.usr-ro/local/..,upperdir=/.usr-rw/lost+found/.. 0 2

On boot I get an error about the device overlayfs/none, so I'm prompted to enter maintenance mode. I issue a "mount -a; exit;" and all is well the boot success.

I'm sick of doing this.

Here is a small script I wrote to resquash /usr, it's incomplete as I'm not sure how to size the lv.

#!/bin/sh
mksquashfs /usr /root/usr.sqfs -comp lzo -no-exports -always-use-fragments -wildcards -e local/*&
kpid=$!

(
#unsquashfs -s usr.sqfs
#lvcreate -L 5G Arcadia -n nuusr$$.ro
cryptsetup -q luksFormat /dev/Arcadia/nuusr$$.ro /etc/keys/key.txt
newo=$(cryptsetup luksUUID /dev/Arcadia/nuusr$$.ro)
cryptsetup luksOpen /dev/Arcadia/nuusr$$.ro $newo --key-file /etc/keys/key.txt
wait $kpid
{ dd if=/root/usr.sqfs bs=4k; cat /dev/urandom; } | dd bs=4M of=/dev/mapper/$newo
cryptsetup luksClose $newo
) &
opid=$!

lvcreate -L 8G Arcadia -n nuusr$$.rw
cryptsetup -q luksFormat /dev/Arcadia/nuusr$$.rw /etc/keys/key.txt
neww=$(cryptsetup luksUUID /dev/Arcadia/nuusr$$.rw)
cryptsetup luksOpen /dev/Arcadia/nuusr$$.rw $neww --key-file /etc/keys/key.txt
mkfs.ext4 /dev/mapper/$neww
tune2fs -e remount-ro -o journal_data_writeback,discard /dev/mapper/$neww
cryptsetup luksClose $neww

wait $opid
newo=$(cryptsetup luksUUID /dev/Arcadia/nuusr$$.ro)
oldo=$(grep fsUsr.ro /etc/crypttab | grep -o ........-....-....-....-............)
oldw=$(grep fsUsr.rw /etc/crypttab | grep -o ........-....-....-....-............)
sed -i -e "s/$oldo/$newo/" -e "s/$oldw/$neww/" /etc/crypttab

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: mount 2.20.1-1ubuntu3 [modified: bin/findmnt bin/mount bin/umount sbin/losetup sbin/swapon]
ProcVersionSignature: Ubuntu 3.2.0-23.31-lowlatency-pae 3.2.14
Uname: Linux 3.2.0-23-lowlatency-pae i686
ApportVersion: 2.0.1-0ubuntu5
Architecture: i386
Date: Fri Jun 1 19:28:18 2012
SourcePackage: util-linux
UpgradeStatus: Upgraded to precise on 2012-01-03 (150 days ago)

[Mike Mestnik (cheako)]

Here is a screen capture containing the error.

[ 14.085449] device-mapper: dm-raid45: initialized v0.2594b
done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ...
Unlocking the disk /dev/disk/by-uuid/772549dd-497a-40ee-95bf-c6e4158dd78f (fsRoot)
Enter passphrase: <entered>
cryptsetup: fsRoot set up successfully
done.
Begin: Running /scripts/local-premount ... done.
[ 873.380203] EXT4-fs (dm-9): mounted filesystem with ordered data mode. Opts: (null)
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
[ 874.350629] init: ureadahead main process (1170) terminated with status 5
fsck from util-linux 2.20.1
fsck from util-linux 2.20.1
udevd[1334]: failed to execute '/lib/udev/mtp-probe' 'mtp-probe /sys/devices/pci0000:00/0000:00:13.2/usb2/2-4 2 5': No such file or directory

udevd[1337]: failed to execute '/lib/udev/mtp-probe' 'mtp-probe /sys/devices/pci0000:00/0000:00:13.0/usb5/5-2 5 3': No such file or directory

udevd[1344]: failed to execute '/lib/udev/mtp-probe' 'mtp-probe /sys/devices/pci0000:00/0000:00:13.2/usb2/2-3/2-3.3 2 6': No such file or directory

udevd[1341]: failed to execute '/lib/udev/mtp-probe' 'mtp-probe /sys/devices/pci0000:00/0000:00:13.0/usb5/5-1 5 2': No such file or directory

udevd[1275]: kernel-provided name 'uinput' and NAME= 'input/uinput' disagree, please use SYMLINK+= or change the kernel to provide the proper name

[ 876.637986] SP5100 TCO timer: mmio address 0xb8fe00 already in use
rpcbind: Cannot open '/run/rpcbind/rpcbind.xdr' file for reading, errno 2 (No such file or directory)
rpcbind: Cannot open '/run/rpcbind/portmap.xdr' file for reading, errno 2 (No such file or directory)
fsck from util-linux 2.20.1
fsck from util-linux 2.20.1
fsck from util-linux 2.20.1
fsck from util-linux 2.20.1
usrlocal: clean, 901/655360 files, 89008/2621440 blocks
wine: clean, 17264/13107200 files, 12192890/52428800 blocks
var: clean, 30450/983040 files, 1701432/3932160 blocks
home: clean, 499953/16384000 files, 26611698/65536000 blocks
root: clean, 226080/3932160 files, 8578558/15728640 blocks (check in 4 mounts)
boot: clean, 235/25584 files, 60829/103424 blocks
fsck from util-linux 2.20.1
mount: special device overlayfs does not exist
mountall: mount /usr [3381] terminated with status 32
mountall: Filesystem could not be mounted: /usr
An error occurred while mounting /usr.
Press S to skip mounting or M for manual recovery
fsck from util-linux 2.20.1
fsck from util-linux 2.20.1
/dev/mapper/fsUsr.rw: clean, 64/524288 files, 70359/2096640 blocks
m
Spawning maintenance shell

[Mike Mestnik (cheako)]

I have some more information here. It seams that every time I boot the lowerdir=/.usr-ro/local/.. and upperdir=/.usr-rw/lost+found/.. are not mounted. Correct ordering of mount would be needed to make this work.

I've changed these to be just lowerdir=/.usr-ro and upperdir=/.usr-rw. I hope I won't find any bugs with this, for it to be OK that something is mounted AFTER the overlayfs is setup.

[Mike Mestnik (cheako)]

This was a complete train wreak. I added the 'folder' checks in to avoid booting with an empty /usr. Thus I believe that overlayfs grabs the folder as-is and doesn't detect future mounted filesystems, always using the device and inode detected at mount time... That's an(two in this case) empty folder.

Doing the mount this way prevents the system from booting into an unrecoverable state. Booting into rescue/single-user mode I was able to correct the system.

I believe this is an issue with the boot scripts, as "mount -a" always seams to do the correct thing.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.