transient systemd ordering cycle in boot with overlayroot

Bug #1788188 reported by Scott Moser on 2018-08-21
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
High
Unassigned

Bug Description

open-iscsi test utilizes overlayroot to boot a cloud-image with root
filesystem on a read-only iscsi server.

The /etc/fstab file in the image looks like this:
  LABEL=cloudimg-rootfs / ext4 defaults 0 0
  #LABEL=UEFI /boot/efi vfat defaults 0 0

when init takes over from the initramfs, we have

  /proc/cmdline:
    nomodeset iscsi_initiator=maas-enlist
    iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2
    iscsi_target_port=3260 iscsi_initiator=maas-enlist
    ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0
    root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1
    overlayroot=tmpfs console=ttyS0
    ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash

  /etc/fstab:
    # cat /etc/fstab
    #
    # This fstab is in an overlay. The real one can be found at
    # /media/root-ro/etc/fstab
    # The original entry for '/' and other mounts have been updated to be placed
    # under /media/root-ro.
    # To permanently modify this (or any other file), you should change-root into
    # a writable view of the underlying filesystem using:
    # sudo overlayroot-chroot
    #
    #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
    /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0
    #LABEL=UEFI /boot/efi vfat defaults 0 0

 /root/root-ro/etc/fstab:
    LABEL=cloudimg-rootfs / ext4 defaults 0 0
    #LABEL=UEFI /boot/efi vfat defaults 0 0

 /proc/mounts:
   sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
   proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
   udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0
   devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
   tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0
   /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0
   tmpfs-root /media/root-rw tmpfs rw,relatime 0 0
   overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0

 /proc/1/mountinfo:
   21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
   22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
   23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755
   24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
   25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755
   26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro
   27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw
   28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_

overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following:

 mkdir /media/root-ro /media/root-rw # in the initramfs
 mount -t tmpfs tmpfs-root /media/root-rw
 mkdir /media/root-rw/overlay-workdir/_
 mount --move $ROOTMNT /media/root-ro
 mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root
 mkdir $ROOTMNT/media/root-ro
 mkdir $ROOTMNT/media/root-rw
 mount --move /media/root-ro "${ROOTMNT}/media/root-ro"
 mount --move /media/root-rw "${ROOTMNT}/media/root-rw"

 # then, if 'ro' on the command line, it mounts /root read-only.
 mount -o remount,ro $ROOTMNT

The script then exits, as ROOTMNT is now set up with a read-only mount
of the overlayroot. All the mounts it has done have been moved
under ROOTMNT.

On failure systemd reports:
   [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start
   [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
   [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start
   [ 104.137591] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided

On successful boot, we can login and see:

$ find /run/systemd/ -name "*.mount" | xargs ls -l
-rw-r--r-- 1 root root 322 Aug 21 13:55 /run/systemd/generator/-.mount
lrwxrwxrwx 1 root root 10 Aug 21 13:55 /run/systemd/generator/local-fs.target.requires/-.mount -> ../-.mount
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:dev-hugepages.mount -> b847fc2d06b54b1ab86a6f526d149522
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:dev-mqueue.mount -> f991d3bfe35d4e2bb9aa86677ff31d70
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-fs-fuse-connections.mount -> a561df591a28437f990f94269fa08be9
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-config.mount -> 141e9bbaca9041e48458749e29bc58fb
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-debug.mount -> a406163f265c4b02b37e6f92ddbfc029

$ for f in $(find /run/systemd/ -name '*.mount' -type f); do echo == $f ==; cat $f; done
== /run/systemd/generator/-.mount ==
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
Where=/
What=/media/root-ro/
Type=overlay
Options=lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_

--
[1] https://git.launchpad.net/cloud-initramfs-tools/tree/overlayroot/scripts/init-bottom/overlayroot

Related bugs:
 * bug 1723183: [overlayroot] transient systemd ordering issue when using overlayroot

ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: systemd 237-3ubuntu10
ProcVersionSignature: Ubuntu 4.17.0-6.7-generic 4.17.9
Uname: Linux 4.17.0-6-generic x86_64
ApportVersion: 2.20.10-0ubuntu7
Architecture: amd64
Date: Tue Aug 21 14:06:24 2018
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
ProcEnviron:
 TERM=vt220
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.11.1-1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-cosmic
dmi.modalias: dmi:bvnSeaBIOS:bvr1.11.1-1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-cosmic:cvnQEMU:ct1:cvrpc-i440fx-cosmic:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-cosmic
dmi.sys.vendor: QEMU

Scott Moser (smoser) wrote :
description: updated
Changed in systemd (Ubuntu):
status: New → Confirmed
importance: Undecided → High

While it might seem unimportant "only" hitting some open-iscsi tests we can't be sure how many cases and how many people are hit by this every now and then.

So I think this really is more important than it might seem at first.

Scott Moser (smoser) wrote :

I pushed a branch up to
 https://code.launchpad.net/~smoser/ubuntu/+source/open-iscsi/+git/open-iscsi/+ref/bug/1788188-debug
that Christian and I are trying to get some additional debug information out of a failure.

Scott Moser (smoser) wrote :

here is a pastebin of boot with systemd debugging.
 http://paste.ubuntu.com/p/vGTRh6WB3B/

Scott Moser (smoser) wrote :

I've hit that 'retrigger' button 2 more times and have not seen failures.
I just did again. Don't know what to do. :-(

Scott Moser (smoser) wrote :

I uploaded open-iscsi_2.0.874-5ubuntu8 which has the debug code
that we were toying with included. It also includes a fix for
FTBFS bug 1791154.

Dimitri John Ledkov (xnox) wrote :

Note to self.

       umount.target
           A special target unit that unmounts all mount and automount points on system shutdown.

           Mounts that shall be unmounted on system shutdown shall add Conflicts dependencies to this unit for their mount unit, which is implicitly done
           when DefaultDependencies=yes is set (the default).

Given that /media/root-ro is a pre-requisite for /, imho it should be declared with DefaultDependencies=no.

@xnox,
Are you suggesting the /media/root-ro entry in /etc/fstab line for
/media/root-ro should have x-systemd.DefaultDependencies=no ?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers