transient systemd ordering cycle in boot with overlayroot ver read-only open-iscsi root
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
cloud-initramfs-tools (Ubuntu) |
Fix Released
|
Medium
|
Scott Moser |
Bug Description
In the open-iscsi test we boot a read-only root iscsi target, using
overlayroot.
Transiently this operation fails, ultimately leading to emergency shell.
Since this occurs in automated test, there is no access to that console
(it is being logged to a file).
The open iscsi test that triggers this is described at [1], including how
to run it.
Most recently, the open-iscsi version 2.0.873+
run both successfully and failed when running for trigger cloud-utils and
systemd [2]
I am attaching a tarball with logs and artifacts from the autopackage
test runs here for posterity.
In the failure case we see evidence of failure at
[ 52.729092] systemd[1]: media-root\
[ 52.730938] systemd[1]: media-root\
[ 52.735138] systemd[1]: media-root\
[ 52.739910] systemd[1]: Unable to break cycle
[ 52.744992] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
And then it all goes sour later with:
[ TIME ] Timed out waiting for device dev-disk-
[DEPEND] Dependency failed for /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
[ TIME ] Timed out waiting for device VIRTUAL-DISK cloudimg-rootfs.
for reference, the image's fstab is:
LABEL=
LABEL=UEFI /boot/efi vfat defaults 0 0
but /etc/fstab is updated by overlayroot in the initramfs, and as such
contains the following:
#
# This fstab is in an overlay. The real one can be found at
# /media/
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
LABEL=
/media/root-ro/ / overlay lowerdir=
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:
--
[1] https:/
[2] http://
ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: open-iscsi 2.0.873+
ProcVersionSign
Uname: Linux 4.9.0-15-generic x86_64
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
Date: Tue Feb 21 15:50:58 2017
ProcEnviron:
TERM=xterm-
PATH=(custom, no user)
XDG_RUNTIME_
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)
modified.
Related bugs:
* bug 1680197: Zesty deployments failing sporadically
* bug 1723183: transient systemd ordering issue when using overlayroot
* bug 1732028: [systemd] transient boot fail with overlayroot
Related branches
description: | updated |
description: | updated |
description: | updated |
description: | updated |
I have reproduced this when running in a kvm guest without kvm enabled (my user did not have write access to /dev/kvm). I suspect that just slows things down and triggers it.
I've never been able to reproduce it outside of nested.