| 2017-02-21 16:24:47 |
Scott Moser |
bug |
|
|
added bug |
| 2017-02-21 16:24:47 |
Scott Moser |
attachment added |
|
tarball of autopkg test results pass and fail https://bugs.launchpad.net/bugs/1666573/+attachment/4823500/+files/debug.tar.xz |
|
| 2017-02-21 20:25:12 |
Scott Moser |
attachment added |
|
failed-debug-info.tar.gz see 'log' inside and files https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1666573/+attachment/4823788/+files/failed-debug-info.tar.xz |
|
| 2017-03-13 15:50:15 |
Dimitri John Ledkov |
bug watch added |
|
https://github.com/systemd/systemd/issues/3847 |
|
| 2017-05-11 14:57:42 |
Scott Moser |
bug task added |
|
cloud-initramfs-tools (Ubuntu) |
|
| 2017-05-11 15:02:27 |
Scott Moser |
description |
In the open-iscsi test we boot a read-only root iscsi target, using
overlayroot.
Transiently this operation fails, ultimately leading to emergency shell.
Since this occurs in automated test, there is no access to that console
(it is being logged to a file).
The open iscsi test that triggers this is described at [1], including how
to run it.
Most recently, the open-iscsi version 2.0.873+git0.3b4b4500-14ubuntu17 has
run both successfully and failed when running for trigger cloud-utils and
systemd [2]
I am attaching a tarball with logs and artifacts from the autopackage
test runs here for posterity.
In the failure case we see evidence of failure at
[ 52.729092] systemd[1]: media-root\x2dro.mount: Found ordering cycle on media-root\x2dro.mount/start
[ 52.730938] systemd[1]: media-root\x2dro.mount: Found dependency on -.mount/start
[ 52.735138] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
[ 52.739910] systemd[1]: Unable to break cycle
[ 52.744992] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
And then it all goes sour later with:
[ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
[DEPEND] Dependency failed for /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
[ TIME ] Timed out waiting for device VIRTUAL-DISK cloudimg-rootfs.
for reference, the image's fstab is:
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
but /etc/fstab is updated by overlayroot in the initramfs, and as such
contains the following:
#
# This fstab is in an overlay. The real one can be found at
# /media/root-ro/etc/fstab
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults 0 0
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
--
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md?h=ubuntu/devel
[2] http://autopkgtest.ubuntu.com/packages/o/open-iscsi/zesty/amd64
ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: open-iscsi 2.0.873+git0.3b4b4500-14ubuntu17
ProcVersionSignature: User Name 4.9.0-15.16-generic 4.9.5
Uname: Linux 4.9.0-15-generic x86_64
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
Date: Tue Feb 21 15:50:58 2017
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.iscsi.iscsid.conf: [inaccessible: [Errno 13] Permission denied: '/etc/iscsi/iscsid.conf'] |
In the open-iscsi test we boot a read-only root iscsi target, using
overlayroot.
Transiently this operation fails, ultimately leading to emergency shell.
Since this occurs in automated test, there is no access to that console
(it is being logged to a file).
The open iscsi test that triggers this is described at [1], including how
to run it.
Most recently, the open-iscsi version 2.0.873+git0.3b4b4500-14ubuntu17 has
run both successfully and failed when running for trigger cloud-utils and
systemd [2]
I am attaching a tarball with logs and artifacts from the autopackage
test runs here for posterity.
In the failure case we see evidence of failure at
[ 52.729092] systemd[1]: media-root\x2dro.mount: Found ordering cycle on media-root\x2dro.mount/start
[ 52.730938] systemd[1]: media-root\x2dro.mount: Found dependency on -.mount/start
[ 52.735138] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
[ 52.739910] systemd[1]: Unable to break cycle
[ 52.744992] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
And then it all goes sour later with:
[ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
[DEPEND] Dependency failed for /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
[ TIME ] Timed out waiting for device VIRTUAL-DISK cloudimg-rootfs.
for reference, the image's fstab is:
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
but /etc/fstab is updated by overlayroot in the initramfs, and as such
contains the following:
#
# This fstab is in an overlay. The real one can be found at
# /media/root-ro/etc/fstab
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults 0 0
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
--
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md?h=ubuntu/devel
[2] http://autopkgtest.ubuntu.com/packages/o/open-iscsi/zesty/amd64
ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: open-iscsi 2.0.873+git0.3b4b4500-14ubuntu17
ProcVersionSignature: User Name 4.9.0-15.16-generic 4.9.5
Uname: Linux 4.9.0-15-generic x86_64
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
Date: Tue Feb 21 15:50:58 2017
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.iscsi.iscsid.conf: [inaccessible: [Errno 13] Permission denied: '/etc/iscsi/iscsid.conf']
Related bugs:
* bug 1680197: Zesty deployments failing sporadically |
|
| 2017-06-13 20:45:55 |
Scott Moser |
attachment added |
|
info collected from a system showing failure https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1666573/+attachment/4895436/+files/failed.txt |
|
| 2017-06-13 20:46:29 |
Scott Moser |
attachment added |
|
fixed.txt: info collected from system with fix applied to overlayroot (noauto) https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1666573/+attachment/4895437/+files/fixed.txt |
|
| 2017-06-13 20:46:42 |
Scott Moser |
cloud-initramfs-tools (Ubuntu): status |
New |
Confirmed |
|
| 2017-06-13 20:46:45 |
Scott Moser |
cloud-initramfs-tools (Ubuntu): importance |
Undecided |
Medium |
|
| 2017-06-13 20:46:48 |
Scott Moser |
bug task deleted |
open-iscsi (Ubuntu) |
|
|
| 2017-06-13 20:52:34 |
Launchpad Janitor |
branch linked |
|
lp:cloud-initramfs-tools |
|
| 2017-06-13 20:54:27 |
Scott Moser |
cloud-initramfs-tools (Ubuntu): status |
Confirmed |
In Progress |
|
| 2017-06-13 20:54:30 |
Scott Moser |
cloud-initramfs-tools (Ubuntu): assignee |
|
Scott Moser (smoser) |
|
| 2017-06-13 21:30:14 |
Launchpad Janitor |
cloud-initramfs-tools (Ubuntu): status |
In Progress |
Fix Released |
|
| 2017-10-12 17:55:08 |
Scott Moser |
description |
In the open-iscsi test we boot a read-only root iscsi target, using
overlayroot.
Transiently this operation fails, ultimately leading to emergency shell.
Since this occurs in automated test, there is no access to that console
(it is being logged to a file).
The open iscsi test that triggers this is described at [1], including how
to run it.
Most recently, the open-iscsi version 2.0.873+git0.3b4b4500-14ubuntu17 has
run both successfully and failed when running for trigger cloud-utils and
systemd [2]
I am attaching a tarball with logs and artifacts from the autopackage
test runs here for posterity.
In the failure case we see evidence of failure at
[ 52.729092] systemd[1]: media-root\x2dro.mount: Found ordering cycle on media-root\x2dro.mount/start
[ 52.730938] systemd[1]: media-root\x2dro.mount: Found dependency on -.mount/start
[ 52.735138] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
[ 52.739910] systemd[1]: Unable to break cycle
[ 52.744992] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
And then it all goes sour later with:
[ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
[DEPEND] Dependency failed for /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
[ TIME ] Timed out waiting for device VIRTUAL-DISK cloudimg-rootfs.
for reference, the image's fstab is:
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
but /etc/fstab is updated by overlayroot in the initramfs, and as such
contains the following:
#
# This fstab is in an overlay. The real one can be found at
# /media/root-ro/etc/fstab
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults 0 0
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
--
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md?h=ubuntu/devel
[2] http://autopkgtest.ubuntu.com/packages/o/open-iscsi/zesty/amd64
ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: open-iscsi 2.0.873+git0.3b4b4500-14ubuntu17
ProcVersionSignature: User Name 4.9.0-15.16-generic 4.9.5
Uname: Linux 4.9.0-15-generic x86_64
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
Date: Tue Feb 21 15:50:58 2017
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.iscsi.iscsid.conf: [inaccessible: [Errno 13] Permission denied: '/etc/iscsi/iscsid.conf']
Related bugs:
* bug 1680197: Zesty deployments failing sporadically |
In the open-iscsi test we boot a read-only root iscsi target, using
overlayroot.
Transiently this operation fails, ultimately leading to emergency shell.
Since this occurs in automated test, there is no access to that console
(it is being logged to a file).
The open iscsi test that triggers this is described at [1], including how
to run it.
Most recently, the open-iscsi version 2.0.873+git0.3b4b4500-14ubuntu17 has
run both successfully and failed when running for trigger cloud-utils and
systemd [2]
I am attaching a tarball with logs and artifacts from the autopackage
test runs here for posterity.
In the failure case we see evidence of failure at
[ 52.729092] systemd[1]: media-root\x2dro.mount: Found ordering cycle on media-root\x2dro.mount/start
[ 52.730938] systemd[1]: media-root\x2dro.mount: Found dependency on -.mount/start
[ 52.735138] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
[ 52.739910] systemd[1]: Unable to break cycle
[ 52.744992] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
And then it all goes sour later with:
[ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
[DEPEND] Dependency failed for /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
[ TIME ] Timed out waiting for device VIRTUAL-DISK cloudimg-rootfs.
for reference, the image's fstab is:
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
but /etc/fstab is updated by overlayroot in the initramfs, and as such
contains the following:
#
# This fstab is in an overlay. The real one can be found at
# /media/root-ro/etc/fstab
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults 0 0
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
--
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md?h=ubuntu/devel
[2] http://autopkgtest.ubuntu.com/packages/o/open-iscsi/zesty/amd64
ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: open-iscsi 2.0.873+git0.3b4b4500-14ubuntu17
ProcVersionSignature: User Name 4.9.0-15.16-generic 4.9.5
Uname: Linux 4.9.0-15-generic x86_64
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
Date: Tue Feb 21 15:50:58 2017
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.iscsi.iscsid.conf: [inaccessible: [Errno 13] Permission denied: '/etc/iscsi/iscsid.conf']
Related bugs:
* bug 1680197: Zesty deployments failing sporadically
* bug 1723183: transient systemd ordering issue when using overlayroot |
|
| 2017-10-25 15:39:58 |
Ryan Harper |
nominated for series |
|
Ubuntu Zesty |
|
| 2017-10-25 15:40:33 |
Scott Moser |
bug task added |
|
cloud-initramfs-tools (Ubuntu Zesty) |
|
| 2017-10-25 15:40:44 |
Scott Moser |
nominated for series |
|
Ubuntu Xenial |
|
| 2017-10-25 15:40:44 |
Scott Moser |
bug task added |
|
cloud-initramfs-tools (Ubuntu Xenial) |
|
| 2017-10-25 15:40:59 |
Scott Moser |
cloud-initramfs-tools (Ubuntu Xenial): status |
New |
Confirmed |
|
| 2017-10-25 15:41:01 |
Scott Moser |
cloud-initramfs-tools (Ubuntu Zesty): status |
New |
Confirmed |
|
| 2017-10-25 15:41:05 |
Scott Moser |
cloud-initramfs-tools (Ubuntu Xenial): importance |
Undecided |
Medium |
|
| 2017-10-25 15:41:06 |
Scott Moser |
cloud-initramfs-tools (Ubuntu Zesty): importance |
Undecided |
Medium |
|
| 2017-11-22 16:46:37 |
Scott Moser |
description |
In the open-iscsi test we boot a read-only root iscsi target, using
overlayroot.
Transiently this operation fails, ultimately leading to emergency shell.
Since this occurs in automated test, there is no access to that console
(it is being logged to a file).
The open iscsi test that triggers this is described at [1], including how
to run it.
Most recently, the open-iscsi version 2.0.873+git0.3b4b4500-14ubuntu17 has
run both successfully and failed when running for trigger cloud-utils and
systemd [2]
I am attaching a tarball with logs and artifacts from the autopackage
test runs here for posterity.
In the failure case we see evidence of failure at
[ 52.729092] systemd[1]: media-root\x2dro.mount: Found ordering cycle on media-root\x2dro.mount/start
[ 52.730938] systemd[1]: media-root\x2dro.mount: Found dependency on -.mount/start
[ 52.735138] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
[ 52.739910] systemd[1]: Unable to break cycle
[ 52.744992] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
And then it all goes sour later with:
[ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
[DEPEND] Dependency failed for /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
[ TIME ] Timed out waiting for device VIRTUAL-DISK cloudimg-rootfs.
for reference, the image's fstab is:
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
but /etc/fstab is updated by overlayroot in the initramfs, and as such
contains the following:
#
# This fstab is in an overlay. The real one can be found at
# /media/root-ro/etc/fstab
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults 0 0
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
--
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md?h=ubuntu/devel
[2] http://autopkgtest.ubuntu.com/packages/o/open-iscsi/zesty/amd64
ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: open-iscsi 2.0.873+git0.3b4b4500-14ubuntu17
ProcVersionSignature: User Name 4.9.0-15.16-generic 4.9.5
Uname: Linux 4.9.0-15-generic x86_64
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
Date: Tue Feb 21 15:50:58 2017
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.iscsi.iscsid.conf: [inaccessible: [Errno 13] Permission denied: '/etc/iscsi/iscsid.conf']
Related bugs:
* bug 1680197: Zesty deployments failing sporadically
* bug 1723183: transient systemd ordering issue when using overlayroot |
In the open-iscsi test we boot a read-only root iscsi target, using
overlayroot.
Transiently this operation fails, ultimately leading to emergency shell.
Since this occurs in automated test, there is no access to that console
(it is being logged to a file).
The open iscsi test that triggers this is described at [1], including how
to run it.
Most recently, the open-iscsi version 2.0.873+git0.3b4b4500-14ubuntu17 has
run both successfully and failed when running for trigger cloud-utils and
systemd [2]
I am attaching a tarball with logs and artifacts from the autopackage
test runs here for posterity.
In the failure case we see evidence of failure at
[ 52.729092] systemd[1]: media-root\x2dro.mount: Found ordering cycle on media-root\x2dro.mount/start
[ 52.730938] systemd[1]: media-root\x2dro.mount: Found dependency on -.mount/start
[ 52.735138] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
[ 52.739910] systemd[1]: Unable to break cycle
[ 52.744992] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
And then it all goes sour later with:
[ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
[DEPEND] Dependency failed for /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
[ TIME ] Timed out waiting for device VIRTUAL-DISK cloudimg-rootfs.
for reference, the image's fstab is:
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
but /etc/fstab is updated by overlayroot in the initramfs, and as such
contains the following:
#
# This fstab is in an overlay. The real one can be found at
# /media/root-ro/etc/fstab
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults 0 0
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
--
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md?h=ubuntu/devel
[2] http://autopkgtest.ubuntu.com/packages/o/open-iscsi/zesty/amd64
ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: open-iscsi 2.0.873+git0.3b4b4500-14ubuntu17
ProcVersionSignature: User Name 4.9.0-15.16-generic 4.9.5
Uname: Linux 4.9.0-15-generic x86_64
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
Date: Tue Feb 21 15:50:58 2017
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.iscsi.iscsid.conf: [inaccessible: [Errno 13] Permission denied: '/etc/iscsi/iscsid.conf']
Related bugs:
* bug 1680197: Zesty deployments failing sporadically
* bug 1723183: transient systemd ordering issue when using overlayroot
* bug 1732028: [systemd] transient boot fail with overlayroot |
|
| 2017-12-15 21:16:30 |
Scott Moser |
description |
In the open-iscsi test we boot a read-only root iscsi target, using
overlayroot.
Transiently this operation fails, ultimately leading to emergency shell.
Since this occurs in automated test, there is no access to that console
(it is being logged to a file).
The open iscsi test that triggers this is described at [1], including how
to run it.
Most recently, the open-iscsi version 2.0.873+git0.3b4b4500-14ubuntu17 has
run both successfully and failed when running for trigger cloud-utils and
systemd [2]
I am attaching a tarball with logs and artifacts from the autopackage
test runs here for posterity.
In the failure case we see evidence of failure at
[ 52.729092] systemd[1]: media-root\x2dro.mount: Found ordering cycle on media-root\x2dro.mount/start
[ 52.730938] systemd[1]: media-root\x2dro.mount: Found dependency on -.mount/start
[ 52.735138] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
[ 52.739910] systemd[1]: Unable to break cycle
[ 52.744992] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
And then it all goes sour later with:
[ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
[DEPEND] Dependency failed for /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
[ TIME ] Timed out waiting for device VIRTUAL-DISK cloudimg-rootfs.
for reference, the image's fstab is:
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
but /etc/fstab is updated by overlayroot in the initramfs, and as such
contains the following:
#
# This fstab is in an overlay. The real one can be found at
# /media/root-ro/etc/fstab
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults 0 0
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
--
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md?h=ubuntu/devel
[2] http://autopkgtest.ubuntu.com/packages/o/open-iscsi/zesty/amd64
ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: open-iscsi 2.0.873+git0.3b4b4500-14ubuntu17
ProcVersionSignature: User Name 4.9.0-15.16-generic 4.9.5
Uname: Linux 4.9.0-15-generic x86_64
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
Date: Tue Feb 21 15:50:58 2017
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.iscsi.iscsid.conf: [inaccessible: [Errno 13] Permission denied: '/etc/iscsi/iscsid.conf']
Related bugs:
* bug 1680197: Zesty deployments failing sporadically
* bug 1723183: transient systemd ordering issue when using overlayroot
* bug 1732028: [systemd] transient boot fail with overlayroot |
==== Begin SRU Template ====
[Impact]
Overlayfs allows a user to mount the root filesystem read-only
and put an overlay over it.
It handles mounting of the root filesystem in the initramfs, and updates
/etc/fstab to account for non-root filesystems and generally make
/etc/fstab "correct".
systemd can get confused by a /etc/fstab that looks like this:
LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
/media/root-ro/ / overlay lowerdir=...
It would not recognize that the LABEL=cloudimg-rootfs image *was* mounted.
This would lead to boot ordering cycles. The fix that was ultimately
applied was just to comment out the line for /media/root-ro.
That way systemd wouldnt try to ensure that /media/root-ro got mounted.
[Test Case]
TBD
[Regression Potential]
Not to omany things would depend on the /etc/fstab that exists in
an system booted with overlayroot. Something that was reading /etc/fstab
could be confused by this change.
[Other Info]
This change is believed to fix bug 1680197 as well, but due to the
sporadic nature, we are not certain.
==== End SRU Template ====
In the open-iscsi test we boot a read-only root iscsi target, using
overlayroot.
Transiently this operation fails, ultimately leading to emergency shell.
Since this occurs in automated test, there is no access to that console
(it is being logged to a file).
The open iscsi test that triggers this is described at [1], including how
to run it.
Most recently, the open-iscsi version 2.0.873+git0.3b4b4500-14ubuntu17 has
run both successfully and failed when running for trigger cloud-utils and
systemd [2]
I am attaching a tarball with logs and artifacts from the autopackage
test runs here for posterity.
In the failure case we see evidence of failure at
[ 52.729092] systemd[1]: media-root\x2dro.mount: Found ordering cycle on media-root\x2dro.mount/start
[ 52.730938] systemd[1]: media-root\x2dro.mount: Found dependency on -.mount/start
[ 52.735138] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
[ 52.739910] systemd[1]: Unable to break cycle
[ 52.744992] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
And then it all goes sour later with:
[ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
[DEPEND] Dependency failed for /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
[ TIME ] Timed out waiting for device VIRTUAL-DISK cloudimg-rootfs.
for reference, the image's fstab is:
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
but /etc/fstab is updated by overlayroot in the initramfs, and as such
contains the following:
#
# This fstab is in an overlay. The real one can be found at
# /media/root-ro/etc/fstab
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults 0 0
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
--
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md?h=ubuntu/devel
[2] http://autopkgtest.ubuntu.com/packages/o/open-iscsi/zesty/amd64
ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: open-iscsi 2.0.873+git0.3b4b4500-14ubuntu17
ProcVersionSignature: User Name 4.9.0-15.16-generic 4.9.5
Uname: Linux 4.9.0-15-generic x86_64
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
Date: Tue Feb 21 15:50:58 2017
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.iscsi.iscsid.conf: [inaccessible: [Errno 13] Permission denied: '/etc/iscsi/iscsid.conf']
Related bugs:
* bug 1680197: Zesty deployments failing sporadically
* bug 1723183: transient systemd ordering issue when using overlayroot
* bug 1732028: [systemd] transient boot fail with overlayroot |
|
| 2017-12-21 16:48:17 |
Scott Moser |
description |
==== Begin SRU Template ====
[Impact]
Overlayfs allows a user to mount the root filesystem read-only
and put an overlay over it.
It handles mounting of the root filesystem in the initramfs, and updates
/etc/fstab to account for non-root filesystems and generally make
/etc/fstab "correct".
systemd can get confused by a /etc/fstab that looks like this:
LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
/media/root-ro/ / overlay lowerdir=...
It would not recognize that the LABEL=cloudimg-rootfs image *was* mounted.
This would lead to boot ordering cycles. The fix that was ultimately
applied was just to comment out the line for /media/root-ro.
That way systemd wouldnt try to ensure that /media/root-ro got mounted.
[Test Case]
TBD
[Regression Potential]
Not to omany things would depend on the /etc/fstab that exists in
an system booted with overlayroot. Something that was reading /etc/fstab
could be confused by this change.
[Other Info]
This change is believed to fix bug 1680197 as well, but due to the
sporadic nature, we are not certain.
==== End SRU Template ====
In the open-iscsi test we boot a read-only root iscsi target, using
overlayroot.
Transiently this operation fails, ultimately leading to emergency shell.
Since this occurs in automated test, there is no access to that console
(it is being logged to a file).
The open iscsi test that triggers this is described at [1], including how
to run it.
Most recently, the open-iscsi version 2.0.873+git0.3b4b4500-14ubuntu17 has
run both successfully and failed when running for trigger cloud-utils and
systemd [2]
I am attaching a tarball with logs and artifacts from the autopackage
test runs here for posterity.
In the failure case we see evidence of failure at
[ 52.729092] systemd[1]: media-root\x2dro.mount: Found ordering cycle on media-root\x2dro.mount/start
[ 52.730938] systemd[1]: media-root\x2dro.mount: Found dependency on -.mount/start
[ 52.735138] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
[ 52.739910] systemd[1]: Unable to break cycle
[ 52.744992] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
And then it all goes sour later with:
[ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
[DEPEND] Dependency failed for /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
[ TIME ] Timed out waiting for device VIRTUAL-DISK cloudimg-rootfs.
for reference, the image's fstab is:
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
but /etc/fstab is updated by overlayroot in the initramfs, and as such
contains the following:
#
# This fstab is in an overlay. The real one can be found at
# /media/root-ro/etc/fstab
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults 0 0
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
--
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md?h=ubuntu/devel
[2] http://autopkgtest.ubuntu.com/packages/o/open-iscsi/zesty/amd64
ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: open-iscsi 2.0.873+git0.3b4b4500-14ubuntu17
ProcVersionSignature: User Name 4.9.0-15.16-generic 4.9.5
Uname: Linux 4.9.0-15-generic x86_64
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
Date: Tue Feb 21 15:50:58 2017
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.iscsi.iscsid.conf: [inaccessible: [Errno 13] Permission denied: '/etc/iscsi/iscsid.conf']
Related bugs:
* bug 1680197: Zesty deployments failing sporadically
* bug 1723183: transient systemd ordering issue when using overlayroot
* bug 1732028: [systemd] transient boot fail with overlayroot |
In the open-iscsi test we boot a read-only root iscsi target, using
overlayroot.
Transiently this operation fails, ultimately leading to emergency shell.
Since this occurs in automated test, there is no access to that console
(it is being logged to a file).
The open iscsi test that triggers this is described at [1], including how
to run it.
Most recently, the open-iscsi version 2.0.873+git0.3b4b4500-14ubuntu17 has
run both successfully and failed when running for trigger cloud-utils and
systemd [2]
I am attaching a tarball with logs and artifacts from the autopackage
test runs here for posterity.
In the failure case we see evidence of failure at
[ 52.729092] systemd[1]: media-root\x2dro.mount: Found ordering cycle on media-root\x2dro.mount/start
[ 52.730938] systemd[1]: media-root\x2dro.mount: Found dependency on -.mount/start
[ 52.735138] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
[ 52.739910] systemd[1]: Unable to break cycle
[ 52.744992] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
And then it all goes sour later with:
[ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
[DEPEND] Dependency failed for /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
[ TIME ] Timed out waiting for device VIRTUAL-DISK cloudimg-rootfs.
for reference, the image's fstab is:
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
but /etc/fstab is updated by overlayroot in the initramfs, and as such
contains the following:
#
# This fstab is in an overlay. The real one can be found at
# /media/root-ro/etc/fstab
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults 0 0
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
--
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md?h=ubuntu/devel
[2] http://autopkgtest.ubuntu.com/packages/o/open-iscsi/zesty/amd64
ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: open-iscsi 2.0.873+git0.3b4b4500-14ubuntu17
ProcVersionSignature: User Name 4.9.0-15.16-generic 4.9.5
Uname: Linux 4.9.0-15-generic x86_64
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
Date: Tue Feb 21 15:50:58 2017
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.iscsi.iscsid.conf: [inaccessible: [Errno 13] Permission denied: '/etc/iscsi/iscsid.conf']
Related bugs:
* bug 1680197: Zesty deployments failing sporadically
* bug 1723183: transient systemd ordering issue when using overlayroot
* bug 1732028: [systemd] transient boot fail with overlayroot |
|
| 2017-12-21 16:48:25 |
Scott Moser |
bug task deleted |
cloud-initramfs-tools (Ubuntu Xenial) |
|
|
| 2017-12-21 16:48:29 |
Scott Moser |
bug task deleted |
cloud-initramfs-tools (Ubuntu Zesty) |
|
|
