fastpath install duplicates iSCSI initiator names, blocking iSCSI HW

[Mark Brown]

[Impact]

* All Xenial cloud images currently share a fixed iSCSI Initiator Name by default when booted.

* The Initiator Name is intended to be a globally unique identifier of a given initiator.

[Test Case]

* Start two instances (LXD, VMs) using cloud images built using the current livecd-rootfs.
  - Both instances will have the same InitiatorName in /etc/iscsi/initiatorname.iscsi.

* Start two instances (LXD, VMs) using cloud images built using an updated livecd-rootfs.
  - The two instances should have distinct InitiatorName values in /etc/iscsi/initiatorname.iscsi.

* Alternatively, the resulting cloud image from using the old and new livecd-rootfs can be compared. In the older case, /etc/iscsi/initiatorname.iscsi will contain a iqn value. In the fixed case, the file will container "Generate=yes".

[Regression Potential]

* It is currently an error that all cloud images of a given version share the same initiator name. (Even mentioned in the file in question!)

* The likelihood of this regressing any real world iSCSI deployments is very low.

* I imagine the primary source of regressions would be end-users/sysadmins already working around the broken iSCSI behavior in 16.04 by manually rewriting this file. If they rely on detecting the fixed Debian initiatornname to do so, that detection will be broken. However, since in those cases, the initiator name is being generated at boot, that should be sufficient.

* Note that this does not solve the known issue for iSCSI that if a snapshot of a booted image is used to launch more instances, they will share iSCSI initiator names.

---

When using fastpath install, each host is given an identical iSCSI initiator name. This does not happen with Debian install mode. The result is HW SANs that use iSCSI get confused. This is an actual customer/partner issue at present.

It would appear to be fallout of the image based approach to installation.
The /etc/iscsi/initiatorname.iscsi file contains something like:
 ## DO NOT EDIT OR REMOVE THIS FILE!
 ## If you remove this file, the iSCSI daemon will not start.
 ## If you change the InitiatorName, existing access control lists
 ## may reject this initiator. The InitiatorName must be unique
 ## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.
 InitiatorName=iqn.1993-08.org.debian:01:dcafddf1c8b

That is what gets used as the initiator, and is seemingly generated at
package-install time. It needs to be re-created
uniquely per installed host.

[Jeff Lane  (bladernr)]

This could become a huge issue for certification scenarios.

[Mark Brown (mstevenbrown)]

This could also potentially cause problems in OIL.

[Gavin Panella (allenap)]

This will generate a new name, rewriting initiatorname.iscsi:

  echo GenerateName=yes >> /etc/iscsi/initiatorname.iscsi
  service open-iscsi restart

[Blake Rouse (blake-rouse)]

This also could be done in the maas images instead of having curtin perform this action.

[Scott Moser (smoser)]

marked this as triaged.
although i'm not really sure where it should be fixed.

[Nobuto Murata (nobuto)]

This affected me too. Initiator name must be unique all the time. It was hard to find this bug in the past unless we had a control on storage array side reporting duplicate initiator names. We always assumed initiator name was generated uniquely by postinst of open-iscsi package except pre-installed MAAS image.

[Nobuto Murata (nobuto)]

A workaround to make sure every MAAS node's IQN unique would be using late_commands of curtin:

[/etc/maas/preseeds/curtin_userdata]
...
late_commands:
  iscsi_initiator_name: curtin in-target -- sed -i -e "s|^InitiatorName=.*|InitiatorName=$(iscsi-iname -p iqn.1993-08.org.debian:01)|" /etc/iscsi/initiatorname.iscsi
  iscsi_update_initramfs: curtin in-target -- update-initramfs -u
...

Now that open-iscsi is in cloud-image as well(LP: #1511008), this issue is not MAAS image specific, and more environments would be affected.

[Blake Rouse (blake-rouse)]

Adding cloud-images since its now provided from cloud-images.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.440

---------------
livecd-rootfs (2.440) zesty; urgency=medium

  [ Steve Langasek ]
  * live-build/ubuntu-cpc/functions: don't call sleep as part of
    umount_settle; udevadm settle is synchronous and there's no reason to
    sleep after it.
  * live-build/ubuntu-cpc/functions: umount_partition: don't call
    umount_settle for each submount; just unmount them one by one and
    settle at the end.
  * Support passing IMAGE_TARGETS in the environment, to limit which images
    are output as part of a build; particularly useful for ubuntu-cpc builds
    which output multiple image types.

  [ Nishanth Aravamudan ]
  * live-build/ubuntu-cpc/hooks/061-open-iscsi.chroot: generate iSCSI
    Initiator Name at first iscsid run for cloud images to ensure it is
    unique (LP: #1444992).

  [ Brian Murray ]
  * Fix the OVF's metadata to include Ubuntu specific identifiers and
    descriptions instead of the generic Linux ones. Thanks to Chris Glass.
    (LP: #1656293)
  * Add replace_grub_root_with_label function thereby consolidating multiple
    uses of the same calls to sed. Thanks to Dan Watkins.

  [ Adam Conrad ]
  * Set device_tree_address for the new kernel in the RPi bootloader config.txt

 -- Adam Conrad <email address hidden> Thu, 23 Mar 2017 11:39:22 -0600

[Nish Aravamudan (nacc)]

Note that even with this fix, end users if they use their own image based off
an instance that used a cloud image to start will probably trigger: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1677726.

I am thinking we should revisit the overall design a bit and ensure that behavior makes sense and is necessary for all use cases. I'll try and set something up but will follow-up in the other bug, FYI.

[Dan Watkins (oddbloke)]

The livecd-rootfs changes fixed this for cloud-images.

[Victor Tapia (vtapia)]

The cloud image for Xenial is still affected by this bug. Deploying with nova:

$ nova boot --flavor m1.small --image ubuntu-xenial-daily-amd64-server-20170830.1-disk1.img xenial-iscsi1
$ nova boot --flavor m1.small --image ubuntu-xenial-daily-amd64-server-20170830.1-disk1.img xenial-iscsi2

$ ssh xenial-iscsi1 'sudo grep ^Init /etc/iscsi/initiatorname.iscsi'
Warning: Permanently added 'xenial-iscsi1,10.5.1.142' (ECDSA) to the list of known hosts.
InitiatorName=iqn.1993-08.org.debian:01:87d0bcc7edba
$ ssh xenial-iscsi2 'sudo grep ^Init /etc/iscsi/initiatorname.iscsi'
Warning: Permanently added 'xenial-iscsi2,10.5.1.143' (ECDSA) to the list of known hosts.
InitiatorName=iqn.1993-08.org.debian:01:87d0bcc7edba

Checking the image:
$ wget https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64.squashfs
$ unsquashfs xenial-server-cloudimg-amd64.squashfs
$ cat squashfs-root/etc/iscsi/initiatorname.iscsi
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator. The InitiatorName must be unique
## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.1993-08.org.debian:01:87d0bcc7edba

Zesty works fine (the image generates a new name):
$ cat squashfs-root/etc/iscsi/initiatorname.iscsi
GenerateName=yes

[Balint Reczey (rbalint)]

The xenial fix is attached to LP: #1721261, but it is already committed to:
http://bazaar.launchpad.net/~ubuntu-core-dev/livecd-rootfs/xenial-proposed/revision/1422

[Brian Murray (brian-murray)]

Hello Mark, or anyone else affected,

Accepted livecd-rootfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.408.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Build of cloud images using the livecd-rootfs from xenial-proposed is here:

https://launchpad.net/~canonical-foundations/+livefs/ubuntu/xenial/ubuntu-cpc/+build/113163

Downloaded
<https://launchpad.net/~canonical-foundations/+livefs/ubuntu/xenial/ubuntu-cpc/+build/113163/+files/livecd.ubuntu-cpc.disk1.img>. Booted the image twice in a VM.

Boot #1:
$ grep -v ^# /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1993-08.org.debian:01:bfad19c26c80
$

Boot #2:
$ grep -v ^# /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1993-08.org.debian:01:7291ea4b148
$

Marking v-done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.408.21

---------------
livecd-rootfs (2.408.21) xenial; urgency=medium

  [ Balint Reczey ]
  * Use kvm kernel only on amd64.
  * Make non-x86 minimized images consistent with x86 by not explicitly
    installing the server task.
  * Clean up dangling /boot/initrd.img symlink left behind on minimized
    builds.

 -- Steve Langasek <email address hidden> Fri, 20 Oct 2017 23:10:29 -0700

[Andres Rodriguez (andreserl)]

We believe this is no longer an issue in the latest releases of MAAS. Please upgrade to the latest version of MAAS, and If you believe this issue is still present, please re-open this bug report or file a new one.