iscsi-targets don't quit session on shutdown

[Ryan Harper]

1. Artful (MAAS Image)[a]
2. open-iscsi 2.0.874-4ubuntu1
3. On shutdown, all iscsi sessions to be stopped and unmounted
4. Iscsi stops but does not stop sessions and shutdown is blocked/hung

[^[[0;32m OK ^[[0m] Reached target Shutdown.
[ 125.666350] connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294921130, last ping 4294922432, now 4294923712
[ 125.922334] connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294921211, last ping 4294922496, now 4294923776
[ 126.178553] connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294921292, last ping 4294922560, now 4294923840
[ 126.434334] connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294921371, last ping 4294922624, now 4294923904

Note, previously released Artful MAAS images work fine:
http://images.maas.io/ephemeral-v2/daily/artful/amd64/20170721/
which contain

open-iscsi 2.0.873+git0.3b4b4500-14ubuntu17

a. http://images.maas.io/ephemeral-v2/daily/artful/amd64/20170826/

[Scott Moser (smoser)]

When I first saw this, I assumed that it was a result of ifupdown being
dropped from the image (bug 1698181). However, it seems (as of the
20170826 cloud image) that that has not yet happened.

Additionally, I've not been able to reproduce this with the dep8 tests
that are in open-iscsi. And those tests *do* a boot and shutdown
on a iscsi target root.

In an effort to describe what happened previously in zesty and how that
might have gone wrong, I wrote the info below.

In iscsi root environment, the initrams brings up the network connection,
and then connects to the iscsi target. It has to communicate with the target
root in order to acheive the following things:

  a. make the network device not be bounced on the re-played coldplug after
     pivot.
  b. do not take the network device *down* on shutdown.
  c. update DNS information from the result of a dhcp or other network
     configuration.

Artful dropped ifupdown, so the methods which were present in xenial-zesty
to accomplish those things have broken.
I suspect it is possible that 'a' and 'b' can be accomplished by somehow
informing systemd that the root mount was a '_netdev' (mount -O _netdev).
An untested patch for this might look like:
  http://paste.ubuntu.com/25418884/

Its possible that resolvconf integration will still work.

iscsi root works in xenial -> zesty like this:
 * in its local-top [1] initramfs hooks, open-iscsi writes the interface
   that the root device lives on to /run/initramfs/open-iscsi.interface.
 * on interface add (coldplug) or remove (shutdown)
   udev rules /lib/udev/rules.d/70-iscsi-network-interface.rules [2]
   call /lib/open-iscsi/net-interface-handler [3].
 * net-interface-handler is then responsible for basically mocking the
   interface up and interface down for the root network device. This
   includes all of 'a', 'b', and 'c'.
   For dns configuration, see the lines around 'command -v resolvconf' at [3].

[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/extra/initramfs.local-top?h=applied/ubuntu/zesty
[2] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/iscsi-network-interface.rules?h=applied/ubuntu/zesty
[3] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/net-interface-handler?h=applied/ubuntu/zesty

[Ryan Harper (raharper)]

Due to some change in behavior of iscsi, we now trip up over not disconnecting the targets after unmounting them from the target. The disconnect code is call with the target dir path (Which isn't valid any more due to being unmounted). We should definitely umount the targets first (to flush data on the device) but then call disconnect with a path of / to use the config on the ephemeral host (which was copied into the target anyhow).

[Ryan Harper (raharper)]

Well bummer, pre-artful, it doesn't find the session data.
Let's try logging out before umounting (which still syncs devices).

On Mon, Aug 28, 2017 at 7:21 PM, Launchpad Bug Tracker <
<email address hidden>> wrote:

> ** Branch linked: lp:~raharper/curtin/trunk.fix-iscsi-shutdown
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1713537
>
> Title:
> iscsi-targets don't quit session on shutdown
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/curtin/+bug/
> 1713537/+subscriptions
>

[Ryan Harper (raharper)]

The previous fix was not sufficient for raid over iscsi and I missed running that test case.

[Launchpad Janitor (janitor)]

This bug was fixed in the package curtin - 0.1.0~bzr532-0ubuntu1

---------------
curtin (0.1.0~bzr532-0ubuntu1) artful; urgency=medium

  * New upstream snapshot.
    - vmtest: fix artful networking (LP: #1714028, LP: #1718216, LP: #1706744)
    - docs: Trivial doc fix for enabling proposed.
    - setup.py: fix to allow installation into a virtualenv (LP: #1703755)
    - doc: update documentation on curtin-hooks and non-ubuntu installation.
    - reporter: Add journald reporter to send events to journald
    - vmtests: add option to tar disk images after test run
    - install: ensure iscsi service is running to handle shutdown properly
    - mdadm: handle write failures to sysfs entries when stopping mdadm
      (LP: #1708052)
    - vmtest: catch exceptions in curtin-log-print
    - iscsi: use curtin storage config to disconnect iscsi targets
      (LP: #1713537)
    - vmtests: bump skip_by_date values out to give cloud-init SRU more time
    - vmtest: get info about collected symlinks and then delete them.
    - Update network cloud-init related skiptest dates, SRU still pending

 -- Ryan Harper <email address hidden> Thu, 05 Oct 2017 19:15:28 -0500

[Scott Moser (smoser)]

This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.