[FFe] Support for ISCSI block devices

[Blake Rouse]

==== Begin Curtin SRU Template ====
[Impact]
Curtin currently does not support specifying iSCSI targets in the storage
configuration. That means that MAAS or other users of curtin are not able
to install to use iscsi disks during installation.

The changes in this bug add the ability to specify non-root iSCSI targets
to curtin, and include integration tests (vmtest) and unit tests to excercise
the functionality.

[Test Case]
Run curtin vmtest on tests/vmtests/test_iscsi.py.
 ./tools/jenkins-runner tests/vmtests/test_iscsi.py

The named test sets up an iscsi server with tgt and does an installation
utilizing the disks provided by that iscsi server.

[Regression Potential]
Regression would most likely be a result of the new iscsi functionality
mis-identifying configuration for a local disk as a iscsi disk, and then
failing to do things as a result.

==== End Curtin SRU Template ====

[FFe Justification]

 - Curtin currently does not support specifying iSCSI targets in the storage configuration.

 - Users have iSCSI targets they would like to install to via curtin.

 - The changes in this bug add the ability to specify non-root iSCSI targets in the YAML and include vmtests and unittests for that functionality.

 - Existing users are unaffected by this change, as if they specified a iSCSI compatible value in their YAML currently it would fail to parse. The added code is only used when iSCSI is used.

---

Curtin needs to support the ability to attach an ISCSI block device during the installation and perform disk operations just like a locally attached block device. The disk also needs to be auto mounted when Ubuntu is rebooted after installation.

Possible format:

- id: sdb
  type: disk
  iscsi:
    target: iqn.2016-11.io.maas:storage.lun1
    portal: 192.168.122.2:3260

[Scott Moser (smoser)]

It might be sane or nice to support specification of the iscsi target in RFC 4173 format.
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=804162 for some more information.

[Scott Moser (smoser)]

In order to test this through vmtest, we'll need some iscsi server.
I've done some work at [1] which at minimum will show how to register a new device into iscsi, and add it and then remove it within a given xkvm. the 'tgt-boot-test' there handles adding a block device to tgt and then tearing it down.

[1] https://git.launchpad.net/~smoser/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md?h=diglett-master

[Jon Grimm (jgrimm)]

I'm assuming this will need FFe, might as well file it now.

[Launchpad Janitor (janitor)]

This bug was fixed in the package curtin - 0.1.0~bzr470-0ubuntu1

---------------
curtin (0.1.0~bzr470-0ubuntu1) zesty; urgency=medium

  * New upstream snapshot.
    - vmtest: allow time-based skipping of tests, apply to ZestyMdadmBcache
      (LP: #1667078)
    - Add iSCSI disk support. (LP: #1645515)
    - vmtest: handle change in btrfs-progs when finding fs UUID. (LP: #1665701)
    - pycodestyle: fix bare except in vmtests
    - unittests: add subp coverage for retries, and rcs
    - reporting: Add better descriptions and granularity of events
    - tox: do not bother with pep8 from trusty in trusty-check
    - tests: add two scripts used for running vmtest with installed package.
    - Make add-apt-repository command retry
    - Add tests for network bridging configuration
  - debian/rules: remove debug 'find'

 -- Scott Moser <email address hidden> Tue, 28 Feb 2017 10:30:43 -0500

[Jon Grimm (jgrimm)]

This will also be needed in Xenial.

[Brian Murray (brian-murray)]

Hello Blake, or anyone else affected,

Accepted curtin into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr470-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Blake, or anyone else affected,

Accepted curtin into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr470-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

Ubuntu server jenkins now runs integration tests with for yakkety-proposed.
It's latest run at [1] passed and ran the tests listed below that verify
this bug if fixed.

  bug 1645515 tests/vmtests/test_iscsi.py
  bug 1665701 tests/vmtests/test_nvme.py tests/vmtests/test_basic.py

I've attached 2 files for posterity
  curtin-vmtest-proposed-y-23-archive.zip [2]
  curtin-vmtest-proposed-y-23-console.log [3]
artifacts of curtin vmtest-proposed-y build 23

You can verify that the installation of curtin used the version shown by:
$ grep -r "Installation started" archive/ | head -n 1
archive/output/ZestyTestNetworkIPV6Vlan/logs/install-serial.log:[ 10.529395] cloud-init[1503]: curtin: Installation started. (0.1.0~bzr470-0ubuntu1~16.10.1)

--
[1] https://jenkins.ubuntu.com/server/view/Curtin/job/curtin-vmtest-proposed-y/23/
[2] https://bugs.launchpad.net/ubuntu/+source/btrfs-progs/+bug/1665701/+attachment/4834793/+files/curtin-vmtest-proposed-y-23-archive.zip
[3] https://bugs.launchpad.net/ubuntu/+source/btrfs-progs/+bug/1665701/+attachment/4834794/+files/curtin-vmtest-proposed-y-23-console.log

[Scott Moser (smoser)]

Ubuntu server jenkins now runs integration tests with for xenial-proposed.
It's latest run at [1] passed and ran the tests listed below that verify
this bug if fixed.

  bug 1645515 tests/vmtests/test_iscsi.py
  bug 1665701 tests/vmtests/test_nvme.py tests/vmtests/test_basic.py

I've attached 2 files to bug 1665701 for posterity
  curtin-vmtest-proposed-x-22-archive.zip [2]
  curtin-vmtest-proposed-x-22-console.log [3]
artifacts of curtin vmtest-proposed-x build 22

You can verify that the installation of curtin used the version shown by:
$ grep -r "Installation started" archive/ | head -n 1
archive/output/ZestyTestNetworkIPV6Vlan/logs/install-serial.log:[ 9.954624] cloud-init[1518]: curtin: Installation started. (0.1.0~bzr470-0ubuntu1~16.04.1)

--
[1] https://jenkins.ubuntu.com/server/view/Curtin/job/curtin-vmtest-proposed-x/22/
[2] https://bugs.launchpad.net/ubuntu/+source/btrfs-progs/+bug/1665701/+attachment/4834791/+files/curtin-vmtest-proposed-x-22-archive.zip
[3] https://bugs.launchpad.net/ubuntu/+source/btrfs-progs/+bug/1665701/+attachment/4834792/+files/curtin-vmtest-proposed-x-22-console.log

[Launchpad Janitor (janitor)]

This bug was fixed in the package curtin - 0.1.0~bzr470-0ubuntu1~16.04.1

---------------
curtin (0.1.0~bzr470-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/rules: record debian package version in installed package.
    (LP: #1666986)
  * New upstream snapshot.
    - vmtest: allow time-based skipping of tests, apply to ZestyMdadmBcache
      to temporarily avoid bug 1667078.
    - Add iSCSI disk support. (LP: #1645515)
    - vmtest: handle change in btrfs-progs when finding fs UUID. (LP: #1665701)
    - pycodestyle: fix bare except in vmtests
    - unittests: add subp coverage for retries, and rcs
    - reporting: Add better descriptions and granularity of events
    - tox: do not bother with pep8 from trusty in trusty-check
    - tests: add two scripts used for running vmtest with installed package.
    - Make add-apt-repository command retry
    - Add tests for network bridging configuration

 -- Scott Moser <email address hidden> Wed, 01 Mar 2017 11:13:58 -0500

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for curtin has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package curtin - 0.1.0~bzr470-0ubuntu1~16.10.1

---------------
curtin (0.1.0~bzr470-0ubuntu1~16.10.1) yakkety-proposed; urgency=medium

  * debian/rules: record debian package version in installed package.
    (LP: #1666986)
  * New upstream snapshot.
    - vmtest: allow time-based skipping of tests, apply to ZestyMdadmBcache
      to temporarily avoid bug 1667078.
    - Add iSCSI disk support. (LP: #1645515)
    - vmtest: handle change in btrfs-progs when finding fs UUID. (LP: #1665701)
    - pycodestyle: fix bare except in vmtests
    - unittests: add subp coverage for retries, and rcs
    - reporting: Add better descriptions and granularity of events
    - tox: do not bother with pep8 from trusty in trusty-check
    - tests: add two scripts used for running vmtest with installed package.
    - Make add-apt-repository command retry
    - Add tests for network bridging configuration

 -- Scott Moser <email address hidden> Wed, 01 Mar 2017 11:17:12 -0500

[Scott Moser (smoser)]

This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.