transient systemd ordering issue when using overlayroot

Bug #1723183 reported by Scott Moser on 2017-10-12
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-initramfs-tools
Medium
Unassigned
cloud-initramfs-tools (Ubuntu)
Medium
Unassigned
Xenial
Medium
Unassigned
Zesty
Medium
Unassigned
Artful
Medium
Unassigned

Bug Description

==== Begin SRU Template ====
[Impact]
Overlayfs allows a user to mount the root filesystem read-only
and put an overlay over it.
It handles mounting of the root filesystem in the initramfs, and updates
/etc/fstab to account for non-root filesystems and generally make
/etc/fstab "correct".

systemd can get confused by a /etc/fstab that looks like this:

   LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
   /media/root-ro/ / overlay lowerdir=...

It would not recognize that the LABEL=cloudimg-rootfs image *was* mounted.
This would lead to boot ordering cycles. The fix that was ultimately
applied was just to comment out the line for /media/root-ro.

That way systemd wouldnt try to ensure that /media/root-ro got mounted.

The fix that was put in place was just to comment out the '/' (/media/root-ro)
line in /etc/fstab. So essentially:
 - LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
 + #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0

Since that happens in the initramfs before systemd ever generates mount
options, it has the desired affect. And /media/root-ro is already mounted
before systemd starts.

[Test Case]
## setup system to build maas images
$ sudo apt-get install -qy --no-install-recommends bzr
$ bzr branch lp:maas-images
$ cd maas-images
$ ./system-setup

## download the reference squashfs images.
$ arch=amd64
$ burl=http://cloud-images.ubuntu.com/
# download reference squashfs images
$ for rel in xenial zesty artful; do
     file=../$rel-server-cloudimg-$arch.squashfs
     [ -f "$file" ] && continue
     wget "$burl/$rel/current/$file" -O "$file.tmp" && mv "$file.tmp" "$file";
  done

## build maas image output. This creates a squashfs image and kernel/initramfs
## files that have the updated packages inside.
## It does take a while to run, and lots of cpu and IO.
## You can reduce the kernels built by commenting out 'lowlatency' and 'edge'
## kernels:
## sed -i -e '/^[ ].*\(lowlatency\|edge\)/s/^/#/' conf/meph-v3.yaml
# either set 'proposed' or add the repo via M2E_ADD_REPOS
$ proposed="--proposed"
$ export M2E_ADD_REPOS=ppa:smoser/cloud-initramfs-tools
$ for rel in xenial zesty artful; do
  img=../$rel-server-cloudimg-$arch.squashfs
  PATH=$PWD/bin:$PATH ./bin/meph2-build $proposed --config=conf/meph-v3.yaml \
        $arch $rel $(date +"%Y%m%d") $img ../$rel.d/ 2>&1 |
        tee ../$rel.log
  done

## After running, you will have files like:
$ find xenial.d/xenial -type f | sort
xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-initrd
xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-kernel
xenial.d/xenial/amd64/20171221/hwe-16.04/generic/boot-initrd
xenial.d/xenial/amd64/20171221/hwe-16.04/generic/boot-kernel
xenial.d/xenial/amd64/20171221/squashfs
xenial.d/xenial/amd64/20171221/squashfs.manifest

## You can verify in the $rel.log files you have 'Setting up cloud-initramfs-'
$ grep '^Setting up overlayroot' xenial.log
Setting up overlayroot (0.27ubuntu1.5~ppa16.04.1) ...

## Now download the test-boot script, provide it a directory and
## the squashfs, kernel, initramfs

$ ./test-boot test-x xenial.d/xenial/amd64/20171221/squashfs \
    xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-kernel \
    xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-initrd

$ r=zesty; ./test-boot work.d $r.d/*/amd64/*/squashfs $r.d/*/amd64/*/ga-*/generic/boot-kernel $r.d/*/amd64/*/ga-*/generic/boot-initrd
Now, do:
  cd work.d
  ./boot

## Running ./boot will serve the squashfs image over http, and then
## boot the kernel and initrd with a kernel command line of
## root=http://<url>/squashfs
## and
## ds=nocloud-net;seedfrom=http://10.0.2.2:9999/
## which will the read the 'user-data' and 'meta-data' files in that dir.
##
## The result is that you can log in with 'ubuntu' and 'passw0rd'

## Then collect some information
# you'll see the 'LABEL=' entry is commented out.
$ cat /etc/fstab
# local-fs.target should be fine
$ systemctl status local-fs.target --no-pager
# just for record
$ cat /proc/cmdline

[Regression Potential]
Not to omany things would depend on the /etc/fstab that exists in
an system booted with overlayroot. Something that was reading /etc/fstab
could be confused by this change.

[Other Info]
This change is believed to fix bug 1680197 as well, but due to the
sporadic nature, we are not certain.

The fix applied here is a backport what is currently in bionic to
each release. in xenial, that means a new binary package
(cloud-initramfs-updateroot).

==== End SRU Template ====

I had reported this once before in bug 1666573.
I thought it was fixed there, but apparently not.

The failure shows itself as (controls removed):
  [ 80.089654] systemd[1]: media-root\x2dro.mount: Found ordering cycle on media-root\x2dro.mount/start
  [ 80.099474] systemd[1]: media-root\x2dro.mount: Found dependency on -.mount/start
  [ 80.108348] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
  [ 80.117231] systemd[1]: Unable to break cycle
  [ 80.132286] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
  [ 80.180707] systemd[1]: Unnecessary job for VIRTUAL-DISK cloudimg-rootfs was removed.
...
 [ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
 [DEPEND] Dependency failed for /boot/efi.
 [DEPEND] Dependency failed for Local File Systems.

The /etc/fstab as read from / once we've left the initramfs
looks like:
  #
  # This fstab is in an overlay. The real one can be found at
  # /media/root-ro/etc/fstab
  # The original entry for '/' and other mounts have been updated to be placed
  # under /media/root-ro.
  # To permanently modify this (or any other file), you should change-root into
  # a writable view of the underlying filesystem using:
  # sudo overlayroot-chroot
  #
  LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
  LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported

Notes:
 * I do not believe this is strictly tied to open-iscsi, but the
   test that we see it failing in is the open-iscsi test.

Related bugs:
 * bug 1666573: transient systemd ordering cycle in boot with overlayroot ver read-only open-iscsi root
 * bug 1680197: Zesty deployments failing sporadically

Scott Moser (smoser) wrote :

Attaching a tarball of all the currently available result logs from
 http://autopkgtest.ubuntu.com/packages/o/open-iscsi/artful/amd64

easier grepping this way.

Scott Moser (smoser) on 2017-10-12
Changed in cloud-initramfs-tools:
importance: Undecided → Medium
status: New → Confirmed
Scott Moser (smoser) wrote :

improved this time with consoles/ directory of extracted serial consoel log for runs that had it.

Dimitri John Ledkov (xnox) wrote :

I've seen this before, I can't remember how we fixed it there. I believe it should be fixable by using .mount units directly (instead of relying on systemd generator to convert fstab to .mount units) or by adding more ordering/deps between these by using x-systemd* options on the ftab lines.

it somehow does handle nested root mounts badly by default. Because it assumes that to mount /media/root-ro/ one needs /, and it cannot grasp the concept of nested / (as in subsequent mount is mounted on top of /)

in general dependency cycles are bad, and should be fixed. I'll play around with this.

tags: added: rls-bb-incoming
Scott Moser (smoser) wrote :

Perhaps if we just commented out the /media/root-ro line?
since we are already writing that file from the initramfs this is possible...

seems like that could have issues on shutdown, but honestly when you're unmounting everything
you should not be caring what is in /etc/fstab and rather just reading /proc/1/mountinfo backwards, so it seems like that could work.

Scott Moser (smoser) on 2017-10-27
Changed in cloud-initramfs-tools (Ubuntu):
status: New → Fix Released
status: Fix Released → Confirmed
importance: Undecided → Medium
Changed in cloud-initramfs-tools:
status: Confirmed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-initramfs-tools - 0.40ubuntu1

---------------
cloud-initramfs-tools (0.40ubuntu1) bionic; urgency=medium

  * rooturl: adjust squashfs url matching to not require '.'.
    Previously it would match <url>*.squashfs or <url>*.squash
    Now it will match <url>*squashfs or <url>*.squash.
    (LP: #1728048)
  * overlayroot: Comment out the read-only line for /. This hopes to
    avoid systemd confusion and transient ordering cycle. (LP: #1723183)

 -- Scott Moser <email address hidden> Fri, 27 Oct 2017 09:44:49 -0400

Changed in cloud-initramfs-tools (Ubuntu):
status: Confirmed → Fix Released
Scott Moser (smoser) on 2017-12-08
description: updated
Scott Moser (smoser) on 2017-12-15
Changed in cloud-initramfs-tools (Ubuntu Xenial):
status: New → Confirmed
Changed in cloud-initramfs-tools (Ubuntu Zesty):
status: New → Confirmed
Changed in cloud-initramfs-tools (Ubuntu Artful):
status: New → Confirmed
Changed in cloud-initramfs-tools (Ubuntu Xenial):
importance: Undecided → Medium
Changed in cloud-initramfs-tools (Ubuntu Zesty):
importance: Undecided → Medium
Changed in cloud-initramfs-tools (Ubuntu Artful):
importance: Undecided → Medium
Scott Moser (smoser) on 2017-12-21
description: updated
description: updated
Scott Moser (smoser) wrote :
description: updated
Changed in cloud-initramfs-tools (Ubuntu Xenial):
status: Confirmed → In Progress
Changed in cloud-initramfs-tools (Ubuntu Zesty):
status: Confirmed → In Progress
Changed in cloud-initramfs-tools (Ubuntu Artful):
status: Confirmed → In Progress
Scott Moser (smoser) on 2017-12-21
description: updated

Hello Scott, or anyone else affected,

Accepted cloud-initramfs-tools into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-initramfs-tools/0.39ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-initramfs-tools (Ubuntu Artful):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-artful
Brian Murray (brian-murray) wrote :

Hello Scott, or anyone else affected,

Accepted cloud-initramfs-tools into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-initramfs-tools/0.35ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-initramfs-tools (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: added: verification-needed-zesty
Brian Murray (brian-murray) wrote :

Hello Scott, or anyone else affected,

Accepted cloud-initramfs-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-initramfs-tools/0.27ubuntu1.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-initramfs-tools (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Scott Moser (smoser) wrote :
tags: added: verification-done-artful verification-done-xenial verification-done-zesty
removed: verification-needed verification-needed-artful verification-needed-xenial verification-needed-zesty
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-initramfs-tools - 0.39ubuntu1.1

---------------
cloud-initramfs-tools (0.39ubuntu1.1) artful; urgency=medium

  * sync with upstream at 0.40ubuntu1
    - rooturl: adjust squashfs url matching to not require '.'.
      Previously it would match <url>*.squashfs or <url>*.squash
      Now it will match <url>*squashfs or <url>*.squash. (LP: #1728048)
    - overlayroot: avoid confusing systemd by commenting out read-only
      line for /. (LP: #1723183)
    - overlayroot: remove unused variable 'rel_root'.

 -- Scott Moser <email address hidden> Fri, 15 Dec 2017 15:54:28 -0500

Changed in cloud-initramfs-tools (Ubuntu Artful):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for cloud-initramfs-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Changed in cloud-initramfs-tools (Ubuntu Zesty):
status: Fix Committed → Won't Fix
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-initramfs-tools - 0.27ubuntu1.5

---------------
cloud-initramfs-tools (0.27ubuntu1.5) xenial; urgency=medium

  * sync with upstream at 0.40ubuntu1
    - whitespace changes.
    - rooturl: adjust squashfs url matching to not require '.'.
      Previously it would match <url>*.squashfs or <url>*.squash
      Now it will match <url>*squashfs or <url>*squash. (LP: #1728048)
    - updateroot/*, debian/control: new package 'cloud-initramfs-updateroot'
    - debian/README.source: update upstream information for move to git.
    - overlayroot: change from storing password in
      /run/initramfs/overlayroot.passwd rather than randomly generated
      /run/initramfs/overlayroot.XXXXXX
    - overlayroot: avoid confusing systemd (LP: #1723183)
       + Mark / as 'noauto' in the newly written /etc/fstab
       + Comment out the read-only line for /.

 -- Scott Moser <email address hidden> Fri, 15 Dec 2017 15:57:35 -0500

Changed in cloud-initramfs-tools (Ubuntu Xenial):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers