| 2017-11-13 22:11:26 |
Scott Moser |
bug |
|
|
added bug |
| 2017-11-13 22:11:26 |
Scott Moser |
attachment added |
|
log from open-iscsi failure with debconf-1.5.64ubuntu2 https://bugs.launchpad.net/bugs/1732028/+attachment/5008460/+files/open-iscsi-2.0.874-4ubuntu3-with-debconf-1.5.64ubuntu2.log.gz |
|
| 2017-11-17 15:35:30 |
Scott Moser |
systemd (Ubuntu): status |
New |
Confirmed |
|
| 2017-11-17 15:35:34 |
Scott Moser |
systemd (Ubuntu): importance |
Undecided |
Medium |
|
| 2018-02-26 15:56:41 |
Scott Moser |
summary |
transient boot fail with overlayroot |
transient boot fail with overlayroot [open-iscsi dep8 tests] |
|
| 2018-02-26 15:58:59 |
Scott Moser |
attachment added |
|
bionic failure log 2.0.874-5ubuntu2 qemu/1:2.11+dfsg-1ubuntu2 https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1732028/+attachment/5063566/+files/log.gz |
|
| 2018-02-26 15:59:53 |
Scott Moser |
summary |
transient boot fail with overlayroot [open-iscsi dep8 tests] |
transient boot fail with overlayroot [open-iscsi autopkg tests] |
|
| 2018-02-26 16:00:11 |
Scott Moser |
bug task added |
|
open-iscsi (Ubuntu) |
|
| 2018-07-19 11:02:55 |
Christian Ehrhardt |
summary |
transient boot fail with overlayroot [open-iscsi autopkg tests] |
timeout in iscsi boot fail with overlayroot [open-iscsi autopkg tests] |
|
| 2018-07-19 11:03:05 |
Christian Ehrhardt |
summary |
timeout in iscsi boot fail with overlayroot [open-iscsi autopkg tests] |
timeout in iscsi boot fail with overlayroot [open-iscsi autopkg tests on LP Infra] |
|
| 2018-07-19 12:11:19 |
Christian Ehrhardt |
bug |
|
|
added subscriber Christian Ehrhardt |
| 2018-07-19 13:02:21 |
Scott Moser |
description |
This issue keeps cropping up. It shows itself in open-iscsi autopkg tests.
I think it might just be "really slow system". It seems the timeout is only
1 minute 30 seconds for the disk to appear, and in a happy run you
might see something very close:
[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for dev-disk…-UEFI.device (1min 29s / 1min 32s)
[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for dev-disk…-UEFI.device (1min 30s / 1min 32s)
[K[[0;32m OK [0m] Found device VIRTUAL-DISK UEFI.
Mounting /boot/efi...
---
There is information on the open-iscsi tests at [1].
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md
The tests set up an iscsi target and boot a kvm guest off that read-only root with overlayroot.
# get open-iscsi source
$ apt-get source open-iscsi
$ cd open-iscsi-2.0.874/
$ sudo apt-get install -qy simplestreams tgt qemu-system-x86 \
cloud-image-utils distro-info
$ cd open-iscsi-2.0.874/
## we're now mostly following debian/tests/README-boot-test.md
# download the image and get kernel/initrd
$ PATH=$PWD/debian/tests:$PATH
$ get-image bionic.d bionic
$ sudo `which patch-image` \
--kernel=bionic.d/kernel --initrd=bionic.d/initrd bionic.d/disk.img
$ tgt-boot-test -v bionic.d/disk.img bionic.d/kernel bionic.d/initrd
....
Success is being able to log in with 'ubuntu' and 'passw0rd'.
Failure as seen in the log is dropping into an emergency shell.
Once inside (this was successful) you'll see a mostly sane system.
Some things to note:
a.) tgt-boot-test boots without kvm enabled. This is because using
kvm with qemu in nested virt would cause system lockups. Its slower
but more reliable to go wtihout.
b.) under bug 1723183 I made overlayroot comment out the root filesystem
from the rendered /etc/fstab. That was because systemd got confused and
assumed that /media/root-ro had to be on top of /.
c.) you can enable or disable kvm by setting _USE_KVM=0 or _USE_KVM=1
in your environment.
$ grep -v "^# " /etc/fstab
#
#
#LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overl
ay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
$ sudo blkid
/dev/sda1: LABEL="cloudimg-rootfs" UUID="7b1980bd-9102-4356-8df0-ec7a0c062411" TYPE="ext4" PARTUUID="c0b5ace0-4703-4667-babb-3d38137cab88"
/dev/sda15: LABEL="UEFI" UUID="B177-3CC9" TYPE="vfat" PARTUUID="0ab0b9fd-2c28-4724-857a-1559f0cf76ea"
/dev/sda14: PARTUUID="221662d6-cab0-4290-ba1c-e72acf2bf193"
$ cat /run/systemd/generator/local-fs.target.requires/boot-efi.mount
# Automatically generated by systemd-fstab-generator
[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target
[Mount]
Where=/boot/efi
What=/dev/disk/by-label/UEFI
Type=vfat
Related bugs:
* bug 1680197: Zesty deployments failing sporadically
* bug 1723183: transient systemd ordering issue when using overlayroot
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: systemd 234-2ubuntu12
ProcVersionSignature: User Name 4.13.0-16.19-generic 4.13.4
Uname: Linux 4.13.0-16-generic x86_64
ApportVersion: 2.20.7-0ubuntu4
Architecture: amd64
Date: Mon Nov 13 21:06:36 2017
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
ProcEnviron:
TERM=vt220
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=C.UTF-8
SHELL=/bin/bash
ProcKernelCmdLine: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-7xuhwl iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-7xuhwl-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-artful
dmi.modalias: dmi:bvnSeaBIOS:bvr1.10.2-1ubuntu1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-artful:cvnQEMU:ct1:cvrpc-i440fx-artful:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-artful
dmi.sys.vendor: QEMU |
This issue keeps cropping up. It shows itself in open-iscsi autopkg tests.
I think it might just be "really slow system". It seems the timeout is only
1 minute 30 seconds for the disk to appear, and in a happy run you
might see something very close:
[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for dev-disk…-UEFI.device (1min 29s / 1min 32s)
[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for dev-disk…-UEFI.device (1min 30s / 1min 32s)
[K[[0;32m OK [0m] Found device VIRTUAL-DISK UEFI.
Mounting /boot/efi...
---
There is information on the open-iscsi tests at [1].
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md
The tests set up an iscsi target and boot a kvm guest off that read-only root with overlayroot.
# get open-iscsi source
$ apt-get source open-iscsi
$ cd open-iscsi-2.0.874/
$ sudo apt-get install -qy simplestreams tgt qemu-system-x86 \
cloud-image-utils distro-info
$ cd open-iscsi-2.0.874/
## we're now mostly following debian/tests/README-boot-test.md
# download the image and get kernel/initrd
$ PATH=$PWD/debian/tests:$PATH
$ get-image bionic.d bionic
$ sudo `which patch-image` \
--kernel=bionic.d/kernel --initrd=bionic.d/initrd bionic.d/disk.img
$ tgt-boot-test -v bionic.d/disk.img bionic.d/kernel bionic.d/initrd
....
Success is being able to log in with 'ubuntu' and 'passw0rd'.
Failure as seen in the log is dropping into an emergency shell.
Once inside (this was successful) you'll see a mostly sane system.
Some things to note:
a.) tgt-boot-test boots without kvm enabled. This is because using
kvm with qemu in nested virt would cause system lockups. Its slower
but more reliable to go wtihout.
b.) under bug 1723183 I made overlayroot comment out the root filesystem
from the rendered /etc/fstab. That was because systemd got confused and
assumed that /media/root-ro had to be on top of /.
c.) you can enable or disable kvm by setting _USE_KVM=0 or _USE_KVM=1
in your environment.
$ grep -v "^# " /etc/fstab
#
#
#LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overl
ay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
$ sudo blkid
/dev/sda1: LABEL="cloudimg-rootfs" UUID="7b1980bd-9102-4356-8df0-ec7a0c062411" TYPE="ext4" PARTUUID="c0b5ace0-4703-4667-babb-3d38137cab88"
/dev/sda15: LABEL="UEFI" UUID="B177-3CC9" TYPE="vfat" PARTUUID="0ab0b9fd-2c28-4724-857a-1559f0cf76ea"
/dev/sda14: PARTUUID="221662d6-cab0-4290-ba1c-e72acf2bf193"
$ cat /run/systemd/generator/local-fs.target.requires/boot-efi.mount
# Automatically generated by systemd-fstab-generator
[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target
[Mount]
Where=/boot/efi
What=/dev/disk/by-label/UEFI
Type=vfat
Related bugs:
* bug 1680197: Zesty deployments failing sporadically
* bug 1723183: transient systemd ordering issue when using overlayroot
* bug 1666573: transient systemd ordering cycle in boot with overlayroot ver read-only open-iscsi root
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: systemd 234-2ubuntu12
ProcVersionSignature: User Name 4.13.0-16.19-generic 4.13.4
Uname: Linux 4.13.0-16-generic x86_64
ApportVersion: 2.20.7-0ubuntu4
Architecture: amd64
Date: Mon Nov 13 21:06:36 2017
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
ProcEnviron:
TERM=vt220
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=C.UTF-8
SHELL=/bin/bash
ProcKernelCmdLine: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-7xuhwl iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-7xuhwl-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-artful
dmi.modalias: dmi:bvnSeaBIOS:bvr1.10.2-1ubuntu1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-artful:cvnQEMU:ct1:cvrpc-i440fx-artful:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-artful
dmi.sys.vendor: QEMU |
|
| 2018-07-19 13:04:18 |
Scott Moser |
attachment added |
|
notes on running open-iscsi tests more interactively https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1732028/+attachment/5165257/+files/hackpad-test-notes.md |
|
| 2018-07-19 20:42:45 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~paelzer/ubuntu/+source/open-iscsi/+git/open-iscsi/+merge/350076 |
|
| 2018-07-19 23:46:42 |
Launchpad Janitor |
open-iscsi (Ubuntu): status |
New |
Fix Released |
|
| 2021-06-30 21:12:47 |
Dan Streetman |
systemd (Ubuntu): status |
Confirmed |
Invalid |
|
