vmtest: open-iscsi iscsi/LVM tests failing on cosmic/disco
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| curtin (Ubuntu) |
Undecided
|
Unassigned | ||
| Bionic |
Undecided
|
Unassigned | ||
| Cosmic |
Undecided
|
Unassigned | ||
| open-iscsi (Ubuntu) |
High
|
Scott Moser | ||
| Bionic |
High
|
Scott Moser | ||
| Cosmic |
High
|
Scott Moser |
Bug Description
=== Begin SRU Template ===
[Impact]
This bug results in lvm devices on top of iscsi block devices not being
available to the system. The journal will show messages like:
[ 17.666655] systemd[1]: dev-disk-
And ultimately
TIME [0m] Timed out waiting for device dev-di...device.
[Test Case]
This can be tested by running curtin's vmtest test harness.
A run like this should suffice:
env CURTIN_
CURTIN_
./tools/
That is all the iscsi based tests and a 'basic' install for good measure.
[Regression Potential]
Chance of regression is hopefully low, and hopefully lower now than before.
Regression is almost certainly limited to iscsi devices.
[Other Info]
An attempt to fix bug 1802354 added a udev rule for open-iscsi that looked
like:
SUBSYSTEM=
The problem was ENV{SYSTEMD_
The '=' operator would truncate the udev-added "wants". lvm also utilized
SYSTEMD_WANTS, and the two would conflict.
This is the second regression fix to the open-iscsi package as a result
of the fix for bug 1755858.
The regression was caught by curtin's test harness (vmtest). Specifically
the following tests would fail:
tests/
tests/
tests/
Related Bugs:
* bug 1755858: iscsid autostarts on all servers when it has nothing to do
* bug 1802354: iscsid does not run if there are only initramfs initiated targets
=== End SRU Template ===
TestIscsiBasic and other Iscsi tests on Cosmic/Disco are failing in vmtest on jenkins.
- looked some at curtin at iscsi failures cosmic, disco
- 604 was first fail like this.
cosmic 20181204
disco 20181129
https:/
602 PASSED and did run disco
cosmic 20181114
disco 20181109
- seems likely tied to open-iscsi upload https:/
I want to revert iscsi and see if it fixes.
Related branches
- Steve Langasek (community): Approve on 2018-12-12
- Ubuntu Server Dev import team: Pending requested 2018-12-12
-
Diff: 24 lines (+8/-1)2 files modifieddebian/changelog (+7/-0)
debian/iscsi-disk.rules (+1/-1)
- Steve Langasek (community): Approve on 2018-12-12
- Ubuntu Server Dev import team: Pending requested 2018-12-12
-
Diff: 24 lines (+8/-1)2 files modifieddebian/changelog (+7/-0)
debian/iscsi-disk.rules (+1/-1)
- Steve Langasek (community): Approve on 2018-12-11
-
Diff: 25 lines (+9/-1)2 files modifieddebian/changelog (+8/-0)
debian/iscsi-disk.rules (+1/-1)
Changed in open-iscsi (Ubuntu): | |
status: | New → Confirmed |
importance: | Undecided → High |
Changed in curtin (Ubuntu): | |
status: | New → Invalid |
Scott Moser (smoser) wrote : | #1 |
Scott Moser (smoser) wrote : | #2 |
Here is a full journal of a failing system with a bunch of systemd debug added.
http://
Steve Langasek (vorlon) wrote : | #3 |
This is caused by a bug in the added udev rule.
SUBSYSTEM=="block", ACTION=="add", ENV{ID_
This should be ENV{SYSTEMD_
Changed in open-iscsi (Ubuntu): | |
status: | Confirmed → Triaged |
tags: | added: regression-update |
summary: |
- vmtest: iscsi tests failing on cosmic/disco + vmtest: iscsi/LVM tests failing on cosmic/disco |
Changed in curtin (Ubuntu Bionic): | |
status: | New → Invalid |
Changed in curtin (Ubuntu Cosmic): | |
status: | New → Invalid |
Changed in open-iscsi (Ubuntu Bionic): | |
status: | New → In Progress |
Changed in open-iscsi (Ubuntu Cosmic): | |
status: | New → In Progress |
Changed in open-iscsi (Ubuntu Bionic): | |
importance: | Undecided → High |
Changed in open-iscsi (Ubuntu Cosmic): | |
importance: | Undecided → High |
Changed in open-iscsi (Ubuntu): | |
assignee: | nobody → Scott Moser (smoser) |
Changed in open-iscsi (Ubuntu Bionic): | |
assignee: | nobody → Scott Moser (smoser) |
Changed in open-iscsi (Ubuntu Cosmic): | |
assignee: | nobody → Scott Moser (smoser) |
description: | updated |
Hello Ryan, or anyone else affected,
Accepted open-iscsi into cosmic-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
Changed in open-iscsi (Ubuntu Cosmic): | |
status: | In Progress → Fix Committed |
tags: | added: verification-needed verification-needed-cosmic |
Changed in open-iscsi (Ubuntu Bionic): | |
status: | In Progress → Fix Committed |
Brian Murray (brian-murray) wrote : | #5 |
Hello Ryan, or anyone else affected,
Accepted open-iscsi into bionic-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
tags: | added: verification-needed-bionic |
This bug was fixed in the package open-iscsi - 2.0.874-5ubuntu14
---------------
open-iscsi (2.0.874-5ubuntu14) disco; urgency=medium
* debian/
(LP: #1807978)
-- Scott Moser <email address hidden> Tue, 11 Dec 2018 19:01:14 -0500
Changed in open-iscsi (Ubuntu): | |
status: | Triaged → Fix Released |
Scott Moser (smoser) wrote : | #7 |
Scott Moser (smoser) wrote : | #8 |
tags: |
added: verification-done-cosmic removed: verification-needed-cosmic |
Scott Moser (smoser) wrote : | #9 |
Scott Moser (smoser) wrote : | #10 |
tags: |
added: verification-done verification-done-bionic removed: verification-needed verification-needed-bionic |
Scott Moser (smoser) wrote : | #11 |
With respect to the resource-agents autopkgtest failure in bionic,
I had commented on that in a previous SRU:
https:/
copied here for eaiser view:
It seems this test fails transiently. It was quite reliable from
2017-11-27 to 2018-08-09
with only a single failure in 44 runs.
But 2018-08-22 and after there have been only 3 successful runs in 18.
The point of this is to show that there may well have been a regression
that caused resource-agents to fail, but it is not likely this change.
description: | updated |
Launchpad Janitor (janitor) wrote : | #12 |
This bug was fixed in the package open-iscsi - 2.0.874-5ubuntu2.6
---------------
open-iscsi (2.0.874-
* debian/
(LP: #1807978)
open-iscsi (2.0.874-
[Scott Moser]
* debian/
run ipconfig to gather all DHCP config info, including DNS search
domain, which iBFT can't provide. (LP: #1806777)
-- Scott Moser <email address hidden> Tue, 11 Dec 2018 19:14:17 -0500
Changed in open-iscsi (Ubuntu Bionic): | |
status: | Fix Committed → Fix Released |
The verification of the Stable Release Update for open-iscsi has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.
This bug was fixed in the package open-iscsi - 2.0.874-5ubuntu9.3
---------------
open-iscsi (2.0.874-
* debian/
(LP: #1807978)
open-iscsi (2.0.874-
[Scott Moser]
* debian/
run ipconfig to gather all DHCP config info, including DNS search
domain, which iBFT can't provide. (LP: #1806777)
-- Scott Moser <email address hidden> Tue, 11 Dec 2018 19:10:50 -0500
Changed in open-iscsi (Ubuntu Cosmic): | |
status: | Fix Committed → Fix Released |
summary: |
- vmtest: iscsi/LVM tests failing on cosmic/disco + vmtest: open-iscsi iscsi/LVM tests failing on cosmic/disco |
I've verified that the bug can be fixed by removing the file rules.d/ 70-iscsi- disk.rules
/lib/udev/
from the image.
For reference, that file looks like: PATH}== "*-iscsi- *", ENV{SYSTEMD_ WANTS}= "iscsid. service"
SUBSYSTEM=="block", ACTION=="add", ENV{ID_
iscsid.service looks like: /git.launchpad. net/ubuntu/ +source/ open-iscsi/ tree/debian/ iscsid. service
https:/
open-iscsi.service looks like: /git.launchpad. net/ubuntu/ +source/ open-iscsi/ tree/debian/ open-iscsi. service
https:/
One interesting thing is that IscsiBasic works fine, bug IscsiLvm (lvm over an iscsi device) fails.
https:/ /jenkins. ubuntu. com/server/ job/curtin- vmtest- devel-amd64/ 604/artifact/ output/ CosmicTestIscsi Basic/logs/ boot-serial. log
The serial console from IscsiLvm test looks like: /jenkins. ubuntu. com/server/ job/curtin- vmtest- devel-amd64/ 604/artifact/ output/ CosmicTestIscsi Lvm/logs/ boot-serial. log
https:/
It has by\x2duuid- a98f706b\ x2db064\ x2d4682\ x2d8eb2\ x2d6c2c1284060c .device: Dependency Before= network- online. target ignored (.device units cannot be delayed) by\x2duuid- a98f706b\ x2db064\ x2d4682\ x2d8eb2\ x2d6c2c1284060c .device: Dependency Before= network. target ignored (.device units cannot be delayed) by\x2duuid- 18bec31c\ x2d09a8\ x2d4a02\ x2d91c6\ x2de9bf6efb6fad .device: Dependency Before= network- online. target ignored (.device units cannot be delayed) by\x2duuid- 18bec31c\ x2d09a8\ x2d4a02\ x2d91c6\ x2de9bf6efb6fad .device: Dependency Before= network. target ignored (.device units cannot be delayed)
[ 17.666655] systemd[1]: dev-disk-
[ 17.669807] systemd[1]: dev-disk-
[ 17.673828] systemd[1]: dev-disk-
[ 17.678043] systemd[1]: dev-disk-
And then ultimately ¦4b\x2d8151\ x2d221b270ec708 .device.
[K[[0;1;31m TIME [0m] Timed out waiting for device dev-diâ€