remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Critical
|
Colin Ian King | ||
Disco |
Fix Released
|
Critical
|
Unassigned | ||
Eoan |
Fix Released
|
Critical
|
Unassigned | ||
Focal |
Fix Released
|
Critical
|
Colin Ian King | ||
linux-hwe (Ubuntu) |
Invalid
|
Undecided
|
Unassigned | ||
Bionic |
Fix Released
|
Critical
|
Colin Ian King |
Bug Description
== SRU Justification Disco, Eoan, Focal ==
Multiple squashfs filesystems with overlayfs cause file corruption issues
when modifying zero sized files
== Fix ==
The current fix is pending in https:/
== Test case ==
With an Ubuntu ISO on the cdrom drive, use:
#!/bin/bash -x
mkdir -p /cdrom
mount -t iso9660 -o ro,noatime /dev/sr0 /cdrom
sleep 1
mkdir -p /cow
mount -t tmpfs -o 'rw,noatime,
sleep 1
mkdir -p /cow/upper
mkdir -p /cow/work
modprobe -q -b overlay
sleep 1
modprobe -q -b loop
sleep 1
dev=$(losetup -f)
mkdir -p /filesystem.
losetup $dev /cdrom/
mount -t squashfs -o ro,noatime $dev /filesystem.
sleep 1
dev=$(losetup -f)
mkdir -p /installer.squashfs
losetup $dev /cdrom/
mount -t squashfs -o ro,noatime $dev /installer.squashfs
sleep 1
mkdir -p /root-tmp
mount -t overlay -o 'upperdir=
FILE=/root-
echo foo > $FILE
cat $FILE
sync
#
# dropping caches or remounting causes the bug
#
echo 3 > /proc/sys/
cat $FILE
Without the fix the cat of the file will produce an error. With the the cat will work correctly.
== Regression Potential ==
There is an unhandled corner case:
- two filesystems, A and B, both have null uuid
- upper layer is on A
- lower layer 1 is also on A
- lower layer 2 is on B
However, since this is an issue without the fix and will be addressed later with subsequent fixes once they are OK with upstream I think the risk is minimal considering nobody is complaining about these corner cases with the current broken overlayfs squashfs layering.
-------
1) Download focal subiquity pending image, or eoan release image
2) boot, and press ESC and edit boot command line (F6 in bios, e in UEFI)
3) After --- insert the following options
break=top debug init=/bin/bash
4) Continue boot (Enter in BIOS, ctrl+x in UEFI)
5) in the initramfs execute:
rm /scripts/
exit
6) you will be dropped into pivoted root filesystem, before systemd is execed as pid one
7) /run/initramfs/ will contain a debug log, showing how everything was mounted. Ie. cdrom mounted, squashfs losetup from there, then multilower overlay setup from them, moved to /root, and then pivot-root to /root done to finally end up as /. Underlying layers are moved into /cow for your convenience.
8) At this point modifying zero-byte length files, that exist in the lowest layer, but not the middle one, in certain ways, will results in them to be corrupted, after / is remounted.
9) Corruption examples
(On both focal & eoan)
cat /etc/.pwd.lock
systemd-sysusers
cat /etc/.pwd.lock
mount -o remount /
cat /etc/.pwd.lock
overlayfs: invalid origin (etc/.pwd.lock, ftype=8000, origin ftype=4000)
cat: /etc/.pwd.lock: Input/output error
(Only on eoan)
cat /etc/machine-id
systemd-
cat /etc/machine-id
mount -o remount /
cat /etc/machine-id
overlayfs: invalid origin (etc/machine-id, ftype=8000, origin ftype=4000)
cat: /etc/machine-id: Input/output error
Lots of things break once machine-id and .pwd.lock are corrupted. I.e. unable to dhcp, connect to dbus, add/remove/change users or groups, etc.
We were unable to recreate the issue outside of booting things with casper. Ie. statically on a regular host machine without pivot-root. But hopefully booting to a quite state with nothing running is sufficient to reproduce this.
Instead of booting with `bebroken init=/bin/bash` you can boot with `bebroken systemd.
Currently, we are shipping two hacks in casper's 25adduser script to "rm" the offending files, and create them again on the upper rw layer. They then survive remount without i/o errors. However, we'd rather not ship those hacks, and have kernel overlay fixed to work correctly with multi-lower-dir and not corrupt files upon remounting /.
description: | updated |
Changed in linux (Ubuntu): | |
status: | Incomplete → New |
Changed in linux-hwe (Ubuntu Bionic): | |
milestone: | none → ubuntu-18.04.4 |
importance: | Undecided → Critical |
no longer affects: | linux (Ubuntu Bionic) |
Changed in linux (Ubuntu): | |
milestone: | none → ubuntu-20.01 |
importance: | Undecided → Critical |
Changed in linux-hwe (Ubuntu): | |
status: | New → Invalid |
Changed in linux (Ubuntu): | |
status: | New → Confirmed |
Changed in linux-hwe (Ubuntu Bionic): | |
status: | New → Confirmed |
summary: |
- why does booting any livefs squashfs has kernel complaining about - unable to read metadata something rather + why does booting any livefs squashfs cause the kernel to complain about + being unable to read metadata |
summary: |
why does booting any livefs squashfs cause the kernel to complain about - being unable to read metadata + being unable to read metadata‽ |
tags: | added: rls-ff-incoming |
description: | updated |
description: | updated |
summary: |
- why does booting any livefs squashfs cause the kernel to complain about - being unable to read metadata‽ + remount of multilower moved pivoted-root overlayfs root, results in I/O + errors on some modified files |
Changed in linux (Ubuntu): | |
assignee: | nobody → Colin Ian King (colin-king) |
Changed in linux-hwe (Ubuntu Bionic): | |
assignee: | nobody → Colin Ian King (colin-king) |
description: | updated |
description: | updated |
description: | updated |
no longer affects: | linux-hwe (Ubuntu Focal) |
no longer affects: | linux-hwe (Ubuntu Eoan) |
no longer affects: | linux-hwe (Ubuntu Disco) |
Changed in linux (Ubuntu Focal): | |
status: | Confirmed → In Progress |
Changed in linux-hwe (Ubuntu Bionic): | |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu Eoan): | |
importance: | Undecided → Critical |
Changed in linux (Ubuntu Disco): | |
importance: | Undecided → Critical |
Changed in linux (Ubuntu Eoan): | |
status: | New → Fix Committed |
Changed in linux (Ubuntu Disco): | |
status: | New → Fix Committed |
Changed in linux (Ubuntu Focal): | |
status: | In Progress → Fix Committed |
This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:
apport-collect 1824407
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.