transient systemd ordering cycle in boot with overlayroot
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
systemd (Ubuntu) |
Invalid
|
High
|
Unassigned |
Bug Description
open-iscsi test utilizes overlayroot to boot a cloud-image with root
filesystem on a read-only iscsi server.
The /etc/fstab file in the image looks like this:
LABEL=
#LABEL=UEFI /boot/efi vfat defaults 0 0
when init takes over from the initramfs, we have
/proc/cmdline:
nomodeset iscsi_initiator
iscsi_
iscsi_
ip=
root=
overlayroot
ds=
/etc/fstab:
# cat /etc/fstab
#
# This fstab is in an overlay. The real one can be found at
# /media/
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
#LABEL=
/media/root-ro/ / overlay lowerdir=
#LABEL=UEFI /boot/efi vfat defaults 0 0
/root/
LABEL=
#LABEL=UEFI /boot/efi vfat defaults 0 0
/proc/mounts:
sysfs /sys sysfs rw,nosuid,
proc /proc proc rw,nosuid,
udev /dev devtmpfs rw,nosuid,
devpts /dev/pts devpts rw,nosuid,
tmpfs /run tmpfs rw,nosuid,
/dev/
tmpfs-root /media/root-rw tmpfs rw,relatime 0 0
overlayroot / overlay ro,relatime,
/proc/1/mountinfo:
21 28 0:20 / /sys rw,nosuid,
22 28 0:4 / /proc rw,nosuid,
23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=
24 23 0:21 / /dev/pts rw,nosuid,
25 28 0:22 / /run rw,nosuid,
26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/
27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw
28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=
overlayroot's scripts/
mkdir /media/root-ro /media/root-rw # in the initramfs
mount -t tmpfs tmpfs-root /media/root-rw
mkdir /media/
mount --move $ROOTMNT /media/root-ro
mount -t overlay -o lowerdir=
mkdir $ROOTMNT/
mkdir $ROOTMNT/
mount --move /media/root-ro "${ROOTMNT}
mount --move /media/root-rw "${ROOTMNT}
# then, if 'ro' on the command line, it mounts /root read-only.
mount -o remount,ro $ROOTMNT
The script then exits, as ROOTMNT is now set up with a read-only mount
of the overlayroot. All the mounts it has done have been moved
under ROOTMNT.
On failure systemd reports:
[ 104.098833] systemd[1]: media-root\
[ 104.109897] systemd[1]: media-root\
[ 104.121386] systemd[1]: media-root\
[ 104.137591] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
On successful boot, we can login and see:
$ find /run/systemd/ -name "*.mount" | xargs ls -l
-rw-r--r-- 1 root root 322 Aug 21 13:55 /run/systemd/
lrwxrwxrwx 1 root root 10 Aug 21 13:55 /run/systemd/
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/
$ for f in $(find /run/systemd/ -name '*.mount' -type f); do echo == $f ==; cat $f; done
== /run/systemd/
# Automatically generated by systemd-
[Unit]
SourcePath=
Documentation=
Before=
[Mount]
Where=/
What=/media/
Type=overlay
Options=
--
[1] https:/
Related bugs:
* bug 1723183: [overlayroot] transient systemd ordering issue when using overlayroot
ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: systemd 237-3ubuntu10
ProcVersionSign
Uname: Linux 4.17.0-6-generic x86_64
ApportVersion: 2.20.10-0ubuntu7
Architecture: amd64
Date: Tue Aug 21 14:06:24 2018
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
ProcEnviron:
TERM=vt220
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash
ProcKernelCmdLine: nomodeset iscsi_initiator
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.11.1-1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.
dmi.modalias: dmi:bvnSeaBIOS:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.
dmi.sys.vendor: QEMU
While it might seem unimportant "only" hitting some open-iscsi tests we can't be sure how many cases and how many people are hit by this every now and then.
So I think this really is more important than it might seem at first.