vmtest: open-iscsi iscsi/LVM tests failing on cosmic/disco

[Ryan Harper]

=== Begin SRU Template ===
[Impact]
This bug results in lvm devices on top of iscsi block devices not being
available to the system. The journal will show messages like:
  [ 17.666655] systemd[1]: dev-disk-by\x2duuid-a98f706b\x2db064\x2d4682\x2d8eb2\x2d6c2c1284060c.device: Dependency Before=network-online.target ignored (.device units cannot be delayed)

And ultimately
 TIME [0m] Timed out waiting for device dev-di...device.

[Test Case]
This can be tested by running curtin's vmtest test harness.
A run like this should suffice:

env CURTIN_VMTEST_ADD_REPOS=proposed \
    CURTIN_VMTEST_UPGRADE_PACKAGES=open-iscsi \
    ./tools/jenkins-runner \
       tests/vmtests/test_iscsi.py \
       tests/vmtests/test_lvm_iscsi.py \
       tests/vmtests/test_mdadm_iscsi.py \
       tests/vmtests/test_basic.py

That is all the iscsi based tests and a 'basic' install for good measure.

[Regression Potential]
Chance of regression is hopefully low, and hopefully lower now than before.
Regression is almost certainly limited to iscsi devices.

[Other Info]
An attempt to fix bug 1802354 added a udev rule for open-iscsi that looked
like:
  SUBSYSTEM=="block", ACTION=="add", ENV{ID_PATH}=="*-iscsi-*", ENV{SYSTEMD_WANTS}="iscsid.service"

The problem was ENV{SYSTEMD_WANTS}="iscsid.service".

The '=' operator would truncate the udev-added "wants". lvm also utilized
SYSTEMD_WANTS, and the two would conflict.

This is the second regression fix to the open-iscsi package as a result
of the fix for bug 1755858.

The regression was caught by curtin's test harness (vmtest). Specifically
the following tests would fail:

 tests/vmtests/test_lvm_iscsi.py:BionicTestIscsiLvm
 tests/vmtests/test_lvm_iscsi.py:CosmicTestIscsiLvm
 tests/vmtests/test_lvm_iscsi.py:DiscoTestIscsiLvm

Related Bugs:
 * bug 1755858: iscsid autostarts on all servers when it has nothing to do
 * bug 1802354: iscsid does not run if there are only initramfs initiated targets

=== End SRU Template ===

TestIscsiBasic and other Iscsi tests on Cosmic/Disco are failing in vmtest on jenkins.

  - looked some at curtin at iscsi failures cosmic, disco
    - 604 was first fail like this.
           cosmic 20181204
           disco 20181129
      https://jenkins.ubuntu.com/server/job/curtin-vmtest-devel-amd64/604/consoleFull
      602 PASSED and did run disco
           cosmic 20181114
           disco 20181109

    - seems likely tied to open-iscsi upload https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1755858 or https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1802354
  I want to revert iscsi and see if it fixes.

[Scott Moser (smoser)]

I've verified that the bug can be fixed by removing the file
 /lib/udev/rules.d/70-iscsi-disk.rules
from the image.

For reference, that file looks like:
SUBSYSTEM=="block", ACTION=="add", ENV{ID_PATH}=="*-iscsi-*", ENV{SYSTEMD_WANTS}="iscsid.service"

iscsid.service looks like:
 https://git.launchpad.net/ubuntu/+source/open-iscsi/tree/debian/iscsid.service

open-iscsi.service looks like:
 https://git.launchpad.net/ubuntu/+source/open-iscsi/tree/debian/open-iscsi.service

One interesting thing is that IscsiBasic works fine, bug IscsiLvm (lvm over an iscsi device) fails.

 https://jenkins.ubuntu.com/server/job/curtin-vmtest-devel-amd64/604/artifact/output/CosmicTestIscsiBasic/logs/boot-serial.log

The serial console from IscsiLvm test looks like:
 https://jenkins.ubuntu.com/server/job/curtin-vmtest-devel-amd64/604/artifact/output/CosmicTestIscsiLvm/logs/boot-serial.log

It has
  [ 17.666655] systemd[1]: dev-disk-by\x2duuid-a98f706b\x2db064\x2d4682\x2d8eb2\x2d6c2c1284060c.device: Dependency Before=network-online.target ignored (.device units cannot be delayed)
  [ 17.669807] systemd[1]: dev-disk-by\x2duuid-a98f706b\x2db064\x2d4682\x2d8eb2\x2d6c2c1284060c.device: Dependency Before=network.target ignored (.device units cannot be delayed)
  [ 17.673828] systemd[1]: dev-disk-by\x2duuid-18bec31c\x2d09a8\x2d4a02\x2d91c6\x2de9bf6efb6fad.device: Dependency Before=network-online.target ignored (.device units cannot be delayed)
  [ 17.678043] systemd[1]: dev-disk-by\x2duuid-18bec31c\x2d09a8\x2d4a02\x2d91c6\x2de9bf6efb6fad.device: Dependency Before=network.target ignored (.device units cannot be delayed)

And then ultimately
 [K[[0;1;31m TIME [0m] Timed out waiting for device dev-di…4b\x2d8151\x2d221b270ec708.device.

[Scott Moser (smoser)]

Here is a full journal of a failing system with a bunch of systemd debug added.
 http://paste.ubuntu.com/p/8DMShKJgZR/

[Steve Langasek (vorlon)]

This is caused by a bug in the added udev rule.

SUBSYSTEM=="block", ACTION=="add", ENV{ID_PATH}=="*-iscsi-*", ENV{SYSTEMD_WANTS}="iscsid.service"

This should be ENV{SYSTEMD_WANTS}+= instead. By overwriting the variable, we are clobbering the declaration in /lib/udev/rules.d/69-lvm-metad.rules that there is a dependency on a pvscan.

[Brian Murray (brian-murray)]

Hello Ryan, or anyone else affected,

Accepted open-iscsi into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-iscsi/2.0.874-5ubuntu9.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Ryan, or anyone else affected,

Accepted open-iscsi into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-iscsi/2.0.874-5ubuntu2.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package open-iscsi - 2.0.874-5ubuntu14

---------------
open-iscsi (2.0.874-5ubuntu14) disco; urgency=medium

  * debian/iscsi-disk.rules: Fix bug with LVM on top of iscsi devices.
    (LP: #1807978)

 -- Scott Moser <email address hidden> Tue, 11 Dec 2018 19:01:14 -0500

[Scott Moser (smoser)]

With respect to the resource-agents autopkgtest failure in bionic,
I had commented on that in a previous SRU:
 https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1802354/comments/18

copied here for eaiser view:

It seems this test fails transiently. It was quite reliable from
 2017-11-27 to 2018-08-09
with only a single failure in 44 runs.
But 2018-08-22 and after there have been only 3 successful runs in 18.

The point of this is to show that there may well have been a regression
that caused resource-agents to fail, but it is not likely this change.

[Launchpad Janitor (janitor)]

This bug was fixed in the package open-iscsi - 2.0.874-5ubuntu2.6

---------------
open-iscsi (2.0.874-5ubuntu2.6) bionic; urgency=medium

  * debian/iscsi-disk.rules: Fix bug with LVM on top of iscsi devices.
    (LP: #1807978)

open-iscsi (2.0.874-5ubuntu2.5) bionic; urgency=medium

  [Scott Moser]
  * debian/extra/initramfs.local-top: handle iSCSI iBFT DHCP to correctly
    run ipconfig to gather all DHCP config info, including DNS search
    domain, which iBFT can't provide. (LP: #1806777)

 -- Scott Moser <email address hidden> Tue, 11 Dec 2018 19:14:17 -0500

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for open-iscsi has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package open-iscsi - 2.0.874-5ubuntu9.3

---------------
open-iscsi (2.0.874-5ubuntu9.3) cosmic; urgency=medium

  * debian/iscsi-disk.rules: Fix bug with LVM on top of iscsi devices.
    (LP: #1807978)

open-iscsi (2.0.874-5ubuntu9.2) cosmic; urgency=medium

  [Scott Moser]
  * debian/extra/initramfs.local-top: handle iSCSI iBFT DHCP to correctly
    run ipconfig to gather all DHCP config info, including DNS search
    domain, which iBFT can't provide. (LP: #1806777)

 -- Scott Moser <email address hidden> Tue, 11 Dec 2018 19:10:50 -0500