iscsi-targets don't quit session on shutdown

Bug #1713537 reported by Ryan Harper on 2017-08-28
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
curtin
Medium
Unassigned
curtin (Ubuntu)
Medium
Unassigned
Xenial
Medium
Unassigned
Zesty
Medium
Unassigned
Artful
Medium
Unassigned

Bug Description

1. Artful (MAAS Image)[a]
2. open-iscsi 2.0.874-4ubuntu1
3. On shutdown, all iscsi sessions to be stopped and unmounted
4. Iscsi stops but does not stop sessions and shutdown is blocked/hung

[^[[0;32m OK ^[[0m] Reached target Shutdown.
[ 125.666350] connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294921130, last ping 4294922432, now 4294923712
[ 125.922334] connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294921211, last ping 4294922496, now 4294923776
[ 126.178553] connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294921292, last ping 4294922560, now 4294923840
[ 126.434334] connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294921371, last ping 4294922624, now 4294923904

Note, previously released Artful MAAS images work fine:
http://images.maas.io/ephemeral-v2/daily/artful/amd64/20170721/
which contain

open-iscsi 2.0.873+git0.3b4b4500-14ubuntu17

a. http://images.maas.io/ephemeral-v2/daily/artful/amd64/20170826/

Related branches

Scott Moser (smoser) wrote :

When I first saw this, I assumed that it was a result of ifupdown being
dropped from the image (bug 1698181). However, it seems (as of the
20170826 cloud image) that that has not yet happened.

Additionally, I've not been able to reproduce this with the dep8 tests
that are in open-iscsi. And those tests *do* a boot and shutdown
on a iscsi target root.

In an effort to describe what happened previously in zesty and how that
might have gone wrong, I wrote the info below.

In iscsi root environment, the initrams brings up the network connection,
and then connects to the iscsi target. It has to communicate with the target
root in order to acheive the following things:

  a. make the network device not be bounced on the re-played coldplug after
     pivot.
  b. do not take the network device *down* on shutdown.
  c. update DNS information from the result of a dhcp or other network
     configuration.

Artful dropped ifupdown, so the methods which were present in xenial-zesty
to accomplish those things have broken.
I suspect it is possible that 'a' and 'b' can be accomplished by somehow
informing systemd that the root mount was a '_netdev' (mount -O _netdev).
An untested patch for this might look like:
  http://paste.ubuntu.com/25418884/

Its possible that resolvconf integration will still work.

iscsi root works in xenial -> zesty like this:
 * in its local-top [1] initramfs hooks, open-iscsi writes the interface
   that the root device lives on to /run/initramfs/open-iscsi.interface.
 * on interface add (coldplug) or remove (shutdown)
   udev rules /lib/udev/rules.d/70-iscsi-network-interface.rules [2]
   call /lib/open-iscsi/net-interface-handler [3].
 * net-interface-handler is then responsible for basically mocking the
   interface up and interface down for the root network device. This
   includes all of 'a', 'b', and 'c'.
   For dns configuration, see the lines around 'command -v resolvconf' at [3].

[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/extra/initramfs.local-top?h=applied/ubuntu/zesty
[2] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/iscsi-network-interface.rules?h=applied/ubuntu/zesty
[3] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/net-interface-handler?h=applied/ubuntu/zesty

Ryan Harper (raharper) wrote :

Due to some change in behavior of iscsi, we now trip up over not disconnecting the targets after unmounting them from the target. The disconnect code is call with the target dir path (Which isn't valid any more due to being unmounted). We should definitely umount the targets first (to flush data on the device) but then call disconnect with a path of / to use the config on the ephemeral host (which was copied into the target anyhow).

Ryan Harper (raharper) on 2017-08-29
Changed in open-iscsi (Ubuntu):
status: New → Invalid
Changed in curtin (Ubuntu):
assignee: nobody → Ryan Harper (raharper)
status: New → In Progress

Well bummer, pre-artful, it doesn't find the session data.
Let's try logging out before umounting (which still syncs devices).

On Mon, Aug 28, 2017 at 7:21 PM, Launchpad Bug Tracker <
<email address hidden>> wrote:

> ** Branch linked: lp:~raharper/curtin/trunk.fix-iscsi-shutdown
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1713537
>
> Title:
> iscsi-targets don't quit session on shutdown
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/curtin/+bug/
> 1713537/+subscriptions
>

Ryan Harper (raharper) on 2017-09-01
Changed in curtin (Ubuntu):
assignee: Ryan Harper (raharper) → nobody
status: In Progress → Fix Committed
Ryan Harper (raharper) wrote :

The previous fix was not sufficient for raid over iscsi and I missed running that test case.

Changed in curtin (Ubuntu):
status: Fix Committed → In Progress
Scott Moser (smoser) on 2017-10-05
Changed in curtin:
status: New → Fix Committed
importance: Undecided → Medium
Changed in curtin (Ubuntu):
importance: Undecided → Medium
Changed in curtin (Ubuntu Xenial):
status: New → Confirmed
Changed in curtin (Ubuntu Zesty):
status: New → Confirmed
Changed in curtin (Ubuntu Xenial):
importance: Undecided → Medium
Changed in curtin (Ubuntu Zesty):
importance: Undecided → Medium
no longer affects: open-iscsi (Ubuntu)
Scott Moser (smoser) on 2017-10-05
no longer affects: open-iscsi (Ubuntu Xenial)
no longer affects: open-iscsi (Ubuntu Zesty)
no longer affects: open-iscsi (Ubuntu Artful)
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package curtin - 0.1.0~bzr532-0ubuntu1

---------------
curtin (0.1.0~bzr532-0ubuntu1) artful; urgency=medium

  * New upstream snapshot.
    - vmtest: fix artful networking (LP: #1714028, LP: #1718216, LP: #1706744)
    - docs: Trivial doc fix for enabling proposed.
    - setup.py: fix to allow installation into a virtualenv (LP: #1703755)
    - doc: update documentation on curtin-hooks and non-ubuntu installation.
    - reporter: Add journald reporter to send events to journald
    - vmtests: add option to tar disk images after test run
    - install: ensure iscsi service is running to handle shutdown properly
    - mdadm: handle write failures to sysfs entries when stopping mdadm
      (LP: #1708052)
    - vmtest: catch exceptions in curtin-log-print
    - iscsi: use curtin storage config to disconnect iscsi targets
      (LP: #1713537)
    - vmtests: bump skip_by_date values out to give cloud-init SRU more time
    - vmtest: get info about collected symlinks and then delete them.
    - Update network cloud-init related skiptest dates, SRU still pending

 -- Ryan Harper <email address hidden> Thu, 05 Oct 2017 19:15:28 -0500

Changed in curtin (Ubuntu Artful):
status: In Progress → Fix Released

This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in curtin:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers