transient systemd ordering cycle in boot with overlayroot ver read-only open-iscsi root

[Scott Moser]

In the open-iscsi test we boot a read-only root iscsi target, using
overlayroot.

Transiently this operation fails, ultimately leading to emergency shell.
Since this occurs in automated test, there is no access to that console
(it is being logged to a file).

The open iscsi test that triggers this is described at [1], including how
to run it.

Most recently, the open-iscsi version 2.0.873+git0.3b4b4500-14ubuntu17 has
run both successfully and failed when running for trigger cloud-utils and
systemd [2]

I am attaching a tarball with logs and artifacts from the autopackage
test runs here for posterity.

In the failure case we see evidence of failure at
  [ 52.729092] systemd[1]: media-root\x2dro.mount: Found ordering cycle on media-root\x2dro.mount/start
  [ 52.730938] systemd[1]: media-root\x2dro.mount: Found dependency on -.mount/start
  [ 52.735138] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
  [ 52.739910] systemd[1]: Unable to break cycle
  [ 52.744992] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided

And then it all goes sour later with:
  [ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
  [DEPEND] Dependency failed for /boot/efi.
  [DEPEND] Dependency failed for Local File Systems.
  [ TIME ] Timed out waiting for device VIRTUAL-DISK cloudimg-rootfs.

for reference, the image's fstab is:
  LABEL=cloudimg-rootfs / ext4 defaults 0 0
  LABEL=UEFI /boot/efi vfat defaults 0 0

but /etc/fstab is updated by overlayroot in the initramfs, and as such
contains the following:
  #
  # This fstab is in an overlay. The real one can be found at
  # /media/root-ro/etc/fstab
  # The original entry for '/' and other mounts have been updated to be placed
  # under /media/root-ro.
  # To permanently modify this (or any other file), you should change-root into
  # a writable view of the underlying filesystem using:
  # sudo overlayroot-chroot
  #
  LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults 0 0
  /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
  LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported

--
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md?h=ubuntu/devel
[2] http://autopkgtest.ubuntu.com/packages/o/open-iscsi/zesty/amd64

ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: open-iscsi 2.0.873+git0.3b4b4500-14ubuntu17
ProcVersionSignature: User Name 4.9.0-15.16-generic 4.9.5
Uname: Linux 4.9.0-15-generic x86_64
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
Date: Tue Feb 21 15:50:58 2017
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.iscsi.iscsid.conf: [inaccessible: [Errno 13] Permission denied: '/etc/iscsi/iscsid.conf']

Related bugs:
 * bug 1680197: Zesty deployments failing sporadically
 * bug 1723183: transient systemd ordering issue when using overlayroot
 * bug 1732028: [systemd] transient boot fail with overlayroot

[Scott Moser (smoser)]

I have reproduced this when running in a kvm guest without kvm enabled (my user did not have write access to /dev/kvm). I suspect that just slows things down and triggers it.

I've never been able to reproduce it outside of nested.

[Scott Moser (smoser)]

attaching a tarball of some files and status that i collected from inside a failed instance.
see 'log' for output of some commands, and also collected /run/systemd and /run/cloud-init and some other things.

[Dimitri John Ledkov (xnox)]

Does https://github.com/systemd/systemd/issues/3847 look familiar?

[Dimitri John Ledkov (xnox)]

Usually .mount units do not allow to replace the mount with new options, or have a duplicate mount.
Systemd brain already has -.mount unit, generated from /proc/self/mountinfo thus one cannot have a second -.mount unit with new/different options.

Thus e.g. instead of re-writing /etc/fstab like shown, it's probably best to have a .service file that executes mount to mount the overlay on top.

Alternatively all the mounts should be executed in the initramfs and not rely on systemd to mount these things.

[Dimitri John Ledkov (xnox)]

Also see http://unix.stackexchange.com/questions/296148/how-to-remount-a-systemd-mount-unit-with-different-options

I do not know how one can resolve the '/' mount which hosts '/media/root-ro', with remount of '/media/root-ro' as '/'. As both mounts are '-.mount' in the systemd brain.

[Scott Moser (smoser)]

Fix just uploaded. commit at
  http://bazaar.launchpad.net/~cloud-initramfs-tools/cloud-initramfs-tools/trunk/revision/145

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-initramfs-tools - 0.38ubuntu1

---------------
cloud-initramfs-tools (0.38ubuntu1) artful; urgency=medium

  * overlayroot: add 'noauto' to the options for the /media/root-ro
    fstab entry (LP: #1666573).

 -- Scott Moser <email address hidden> Tue, 13 Jun 2017 16:52:16 -0400

[Scott Moser (smoser)]

Fix here was specifically:
  http://bazaar.launchpad.net/~cloud-initramfs-tools/cloud-initramfs-tools/trunk/revision/145

[Meconiotronic (meconiotech)]

Hi, i have the same problem on raspberry pi3 and ubuntu server overlayroot says in fstab:
overlayroot:fs-unsupported

I've update my system now but the problem still here.

This is my real fstab in overlaychroot:

# Fissi
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=system-boot /boot/firmware vfat defaults,ro,noauto 0 1
LABEL=RW /mnt/RW ext4 defaults,rw,users 0 1

# Bind
/mnt/RW/var /var none bind
/mnt/RW/home /home none bind

#
# This fstab is in an overlay. The real one can be found at
# /media/root-ro/etc/fstab
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
# Fissi
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=system-boot /boot/firmware vfat defaults,ro,noauto 0 1 # overlayroot:fs-unsupported
LABEL=RW /mnt/RW ext4 defaults,rw,users 0 1
      # overlayroot:malformed-line
# Bind
/mnt/RW/var /var none bind # overlayroot:malformed-line
/mnt/RW/home /home none bind # overlayroot:malformed-line

I have patched by hand the file http://bazaar.launchpad.net/~cloud-initramfs-tools/cloud-initramfs-tools/trunk/revision/145

But can't work

Thanks

[Scott Moser (smoser)]

 * bug 1723183: transient systemd ordering issue when using overlayroot
 * bug 1732028: [systemd] transient boot fail with overlayroot

Hi, there is also this commit [1] that fixed bug 1723183. We just kind of gave up and commented out the root entry.

I've opened up a bug against systemd in bug 1732028 as I am still seeing failures in the automated open-iscsi tests.

--
[1] https://git.launchpad.net/~cloud-initramfs-tools/cloud-initramfs-tools/commit/?id=ce092cb57ffc15f139c08113d1e3b593023aad82

[Scott Moser (smoser)]

I'm going to un-tag this bug for sru, as I'm planning on sru-ing a the second fix.
The change in this bug resulted in the following change in the rendered /etc/fstab:
 - LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults 0 0
 + LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0

The change in bug 1723183 was:

 - LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
 + #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0

[Scott Moser (smoser)]

I'm going to un-tag this bug for sru, as I'm planning on sru-ing a the second fix.
The change in this bug resulted in the following change in the rendered /etc/fstab:
 - LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults 0 0
 + LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0

The change in bug 1723183 was:

 - LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
 + #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0

That change supersedes this one.