transient systemd ordering issue when using overlayroot
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
cloud-initramfs-tools |
Fix Released
|
Medium
|
Unassigned | ||
cloud-initramfs-tools (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
Xenial |
Fix Released
|
Medium
|
Unassigned | ||
Zesty |
Won't Fix
|
Medium
|
Unassigned | ||
Artful |
Fix Released
|
Medium
|
Unassigned |
Bug Description
==== Begin SRU Template ====
[Impact]
Overlayfs allows a user to mount the root filesystem read-only
and put an overlay over it.
It handles mounting of the root filesystem in the initramfs, and updates
/etc/fstab to account for non-root filesystems and generally make
/etc/fstab "correct".
systemd can get confused by a /etc/fstab that looks like this:
LABEL=
/media/root-ro/ / overlay lowerdir=...
It would not recognize that the LABEL=cloudimg-
This would lead to boot ordering cycles. The fix that was ultimately
applied was just to comment out the line for /media/root-ro.
That way systemd wouldnt try to ensure that /media/root-ro got mounted.
The fix that was put in place was just to comment out the '/' (/media/root-ro)
line in /etc/fstab. So essentially:
- LABEL=cloudimg-
+ #LABEL=
Since that happens in the initramfs before systemd ever generates mount
options, it has the desired affect. And /media/root-ro is already mounted
before systemd starts.
[Test Case]
## setup system to build maas images
$ sudo apt-get install -qy --no-install-
$ bzr branch lp:maas-images
$ cd maas-images
$ ./system-setup
## download the reference squashfs images.
$ arch=amd64
$ burl=http://
# download reference squashfs images
$ for rel in xenial zesty artful; do
file=
[ -f "$file" ] && continue
wget "$burl/
done
## build maas image output. This creates a squashfs image and kernel/initramfs
## files that have the updated packages inside.
## It does take a while to run, and lots of cpu and IO.
## You can reduce the kernels built by commenting out 'lowlatency' and 'edge'
## kernels:
## sed -i -e '/^[ ].*\(lowlatency
# either set 'proposed' or add the repo via M2E_ADD_REPOS
$ proposed=
$ export M2E_ADD_
$ for rel in xenial zesty artful; do
img=.
PATH=
$arch $rel $(date +"%Y%m%d") $img ../$rel.d/ 2>&1 |
tee ../$rel.log
done
## After running, you will have files like:
$ find xenial.d/xenial -type f | sort
xenial.
xenial.
xenial.
xenial.
xenial.
xenial.
## You can verify in the $rel.log files you have 'Setting up cloud-initramfs-'
$ grep '^Setting up overlayroot' xenial.log
Setting up overlayroot (0.27ubuntu1.
## Now download the test-boot script, provide it a directory and
## the squashfs, kernel, initramfs
$ ./test-boot test-x xenial.
xenial.
xenial.
$ r=zesty; ./test-boot work.d $r.d/*/
Now, do:
cd work.d
./boot
## Running ./boot will serve the squashfs image over http, and then
## boot the kernel and initrd with a kernel command line of
## root=http://<url>/squashfs
## and
## ds=nocloud-
## which will the read the 'user-data' and 'meta-data' files in that dir.
##
## The result is that you can log in with 'ubuntu' and 'passw0rd'
## Then collect some information
# you'll see the 'LABEL=' entry is commented out.
$ cat /etc/fstab
# local-fs.target should be fine
$ systemctl status local-fs.target --no-pager
# just for record
$ cat /proc/cmdline
[Regression Potential]
Not to omany things would depend on the /etc/fstab that exists in
an system booted with overlayroot. Something that was reading /etc/fstab
could be confused by this change.
[Other Info]
This change is believed to fix bug 1680197 as well, but due to the
sporadic nature, we are not certain.
The fix applied here is a backport what is currently in bionic to
each release. in xenial, that means a new binary package
(cloud-
==== End SRU Template ====
I had reported this once before in bug 1666573.
I thought it was fixed there, but apparently not.
The failure shows itself as (controls removed):
[ 80.089654] systemd[1]: media-root\
[ 80.099474] systemd[1]: media-root\
[ 80.108348] systemd[1]: media-root\
[ 80.117231] systemd[1]: Unable to break cycle
[ 80.132286] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
[ 80.180707] systemd[1]: Unnecessary job for VIRTUAL-DISK cloudimg-rootfs was removed.
...
[ TIME ] Timed out waiting for device dev-disk-
[DEPEND] Dependency failed for /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
The /etc/fstab as read from / once we've left the initramfs
looks like:
#
# This fstab is in an overlay. The real one can be found at
# /media/
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
LABEL=
/media/root-ro/ / overlay lowerdir=
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:
Notes:
* I do not believe this is strictly tied to open-iscsi, but the
test that we see it failing in is the open-iscsi test.
Related bugs:
* bug 1666573: transient systemd ordering cycle in boot with overlayroot ver read-only open-iscsi root
* bug 1680197: Zesty deployments failing sporadically
* bug 1723183: [systemd] transient systemd ordering cycle in boot with overlayroot
Changed in cloud-initramfs-tools: | |
importance: | Undecided → Medium |
status: | New → Confirmed |
Changed in cloud-initramfs-tools (Ubuntu): | |
status: | New → Fix Released |
status: | Fix Released → Confirmed |
importance: | Undecided → Medium |
Changed in cloud-initramfs-tools: | |
status: | Confirmed → Fix Released |
description: | updated |
Changed in cloud-initramfs-tools (Ubuntu Xenial): | |
status: | New → Confirmed |
Changed in cloud-initramfs-tools (Ubuntu Zesty): | |
status: | New → Confirmed |
Changed in cloud-initramfs-tools (Ubuntu Artful): | |
status: | New → Confirmed |
Changed in cloud-initramfs-tools (Ubuntu Xenial): | |
importance: | Undecided → Medium |
Changed in cloud-initramfs-tools (Ubuntu Zesty): | |
importance: | Undecided → Medium |
Changed in cloud-initramfs-tools (Ubuntu Artful): | |
importance: | Undecided → Medium |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
Attaching a tarball of all the currently available result logs from autopkgtest. ubuntu. com/packages/ o/open- iscsi/artful/ amd64
http://
easier grepping this way.