Detaching multipath volume doesn't work properly when using different targets with same portal for each multipath device

Bug #1382440 reported by Keiichi KII on 2014-10-17
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Cinder
Undecided
Unassigned
Kilo
Undecided
Patrick East
OpenStack Compute (nova)
High
Hiroyuki Eguchi
Icehouse
High
Hiroyuki Eguchi
Juno
High
Sean Dague
os-brick
High
Danny Wilson

Bug Description

Overview:
On Icehouse(2014.1.2) with "iscsi_use_multipath=true", detaching iSCSI
multipath volume doesn't work properly. When we use different targets(IQNs)
associated with same portal for each different multipath device, all of
the targets will be deleted via disconnect_volume().

This problem is not yet fixed in upstream. However, the attached patch
fixes this problem.

Steps to Reproduce:

We can easily reproduce this issue without any special storage
system in the following Steps:

  1. configure "iscsi_use_multipath=True" in nova.conf on compute node.
  2. configure "volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver"
     in cinder.conf on cinder node.
  2. create an instance.
  3. create 3 volumes and attach them to the instance.
  4. detach one of these volumes.
  5. check "multipath -ll" and "iscsiadm --mode session".

Detail:

This problem was introduced with the following patch which modified
attaching and detaching volume operations for different targets
associated with different portals for the same multipath device.

  commit 429ac4dedd617f8c1f7c88dd8ece6b7d2f2accd0
  Author: Xing Yang <email address hidden>
  Date: Date: Mon Jan 6 17:27:28 2014 -0500

    Fixed a problem in iSCSI multipath

We found out that:

> # Do a discovery to find all targets.
> # Targets for multiple paths for the same multipath device
> # may not be the same.
> out = self._run_iscsiadm_bare(['-m',
> 'discovery',
> '-t',
> 'sendtargets',
> '-p',
> iscsi_properties['target_portal']],
> check_exit_code=[0, 255])[0] \
> or ""
>
> ips_iqns = self._get_target_portals_from_iscsiadm_output(out)
...
> # If no other multipath device attached has the same iqn
> # as the current device
> if not in_use:
> # disconnect if no other multipath devices with same iqn
> self._disconnect_mpath(iscsi_properties, ips_iqns)
> return
> elif multipath_device not in devices:
> # delete the devices associated w/ the unused multipath
> self._delete_mpath(iscsi_properties, multipath_device, ips_iqns)

When we use different targets(IQNs) associated with same portal for each different
multipath device, the ips_iqns has all targets in compute node from the result of
"iscsiadm -m discovery -t sendtargets -p <the same portal>".
Then, the _delete_mpath() deletes all of the targets in the ips_iqns
via /sys/block/sdX/device/delete.

For example, we create an instance and attach 3 volumes to the instance:

  # iscsiadm --mode session
  tcp: [17] 192.168.0.55:3260,1 iqn.2010-10.org.openstack:volume-5c526ffa-ba88-4fe2-a570-9e35c4880d12
  tcp: [18] 192.168.0.55:3260,1 iqn.2010-10.org.openstack:volume-b4495e7e-b611-4406-8cce-4681ac1e36de
  tcp: [19] 192.168.0.55:3260,1 iqn.2010-10.org.openstack:volume-b2c01f6a-5723-40e7-9f21-f6b728021b0e
  # multipath -ll
  33000000300000001 dm-7 IET,VIRTUAL-DISK
  size=4.0G features='0' hwhandler='0' wp=rw
  `-+- policy='round-robin 0' prio=1 status=active
     `- 23:0:0:1 sdd 8:48 active ready running
  33000000100000001 dm-5 IET,VIRTUAL-DISK
  size=2.0G features='0' hwhandler='0' wp=rw
  `-+- policy='round-robin 0' prio=1 status=active
     `- 21:0:0:1 sdb 8:16 active ready running
  33000000200000001 dm-6 IET,VIRTUAL-DISK
  size=3.0G features='0' hwhandler='0' wp=rw
  `-+- policy='round-robin 0' prio=1 status=active
     `- 22:0:0:1 sdc 8:32 active ready running

Then we detach one of these volumes:

  # nova volume-detach 95f959cd-d180-4063-ae03-9d21dbd7cc50 5c526ffa-ba88-4fe2-a570-9e35c4880d12

As a result of detaching the volume, the compute node remains 3 iSCSI sessions
and the instance fails to access the attached multipath devices:

  # iscsiadm --mode session
  tcp: [17] 192.168.0.55:3260,1 iqn.2010-10.org.openstack:volume-5c526ffa-ba88-4fe2-a570-9e35c4880d12
  tcp: [18] 192.168.0.55:3260,1 iqn.2010-10.org.openstack:volume-b4495e7e-b611-4406-8cce-4681ac1e36de
  tcp: [19] 192.168.0.55:3260,1 iqn.2010-10.org.openstack:volume-b2c01f6a-5723-40e7-9f21-f6b728021b0e
  # multipath -ll
  33000000300000001 dm-7 ,
  size=4.0G features='0' hwhandler='0' wp=rw
  `-+- policy='round-robin 0' prio=0 status=enabled
     `- #:#:#:# - #:# failed faulty running
  33000000200000001 dm-6 ,
  size=3.0G features='0' hwhandler='0' wp=rw
  `-+- policy='round-robin 0' prio=0 status=enabled
     `- #:#:#:# - #:# failed faulty running

Ken'ichi Ohmichi (oomichi) wrote :

We need to push a patch on the gerrit.
Could you check https://wiki.openstack.org/wiki/How_To_Contribute ?

Changed in nova:
assignee: nobody → Hiroyuki Eguchi (h-eguchi)
Tracy Jones (tjones-i) on 2014-10-29
tags: added: volumes

Fix proposed to branch: master
Review: https://review.openstack.org/132490

Changed in nova:
status: New → In Progress
Keiichi KII (k-keiichi) on 2014-11-12
tags: added: icehouse-backport-potential juno-backport-potential

Fix proposed to branch: master
Review: https://review.openstack.org/135812

Reviewed: https://review.openstack.org/135812
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fb0de106f2f15604750bafc318ba06c41070cc35
Submitter: Jenkins
Branch: master

commit fb0de106f2f15604750bafc318ba06c41070cc35
Author: Hiroyuki Eguchi <email address hidden>
Date: Thu Nov 20 10:41:36 2014 +0900

    Fix connecting unnecessary iSCSI sessions issue

    In Icehouse with "iscsi_use_multipath=true", attaching a multipath
    iSCSI volume may create unnecessary iSCSI sessions.

    The iscsiadm discovery command in connect_volume() returns all of the
    targets in the Cinder node, not just the ones related to the multipath
    volume which is specified by iqn. If the storage has many targets,
    connecting to all these volumes will also result in many unnecessary
    connections.

    There are two types of iSCSI multipath devices. One which shares
    the same iqn between multiple portals, and the other which use
    different iqns on different portals. connect_volume() needs to
    identify the type by checking iscsiadm the output if the iqn is
    used by multiple portals.

    This patch changes the behavior of attaching volume:

       1. Identify the type by checking the iscsiadm output.
       2. Connect to the correct targets by connect_to_iscsi_portal().

    Closes-Bug: #1382440

    Change-Id: I488ad0c09bf26a609e27d67b9ef60b65bb45e0ad

Changed in nova:
status: In Progress → Fix Committed
Keiichi KII (k-keiichi) on 2014-11-25
Changed in nova:
status: Fix Committed → In Progress

Reviewed: https://review.openstack.org/132490
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=36aeedfd5eeb0345d66fa8456ed6a9447a6514a0
Submitter: Jenkins
Branch: master

commit 36aeedfd5eeb0345d66fa8456ed6a9447a6514a0
Author: Hiroyuki Eguchi <email address hidden>
Date: Thu Dec 4 15:12:11 2014 +0900

    Fix disconnecting necessary iSCSI sessions issue

    In Icehouse with "iscsi_use_multipath=true", detaching a multipath
    iSCSI volume kills all iSCSI volumes visible from the nova compute node.
    When we use different targets(IQNs) associated with same portal for
    each different multipath device, all of the targets will be deleted
    via disconnect_volume().

    This patch fixes the behavior of detaching volume:

      1. Extract the targets for the detached multipath device.
      2. Delete/disconnect the targets for the detached multipath device.

    Closes-Bug: #1382440

    Change-Id: I38eafdaee03d136282cfde1fd013e322a4256cc4

Changed in nova:
status: In Progress → Fix Committed

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/141684

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/141687

Thierry Carrez (ttx) on 2014-12-18
Changed in nova:
milestone: none → kilo-1
status: Fix Committed → Fix Released
Keiichi KII (k-keiichi) on 2014-12-24
Changed in nova:
status: Fix Released → In Progress

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/144064

Change abandoned by Hiroyuki Eguchi (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/141685
Reason: Abandoned because ChangeID is different from the commit on master.
Please refer to new review page,
Review: https://review.openstack.org/144064

Change abandoned by Hiroyuki Eguchi (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/141687
Reason: Abandoned because ChangeID is different from the commit on master.
Please refer to new review page,
Review: https://review.openstack.org/144066

Change abandoned by Hiroyuki Eguchi (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/141682
Reason: Abandoned because ChangeID is different from the commit on master.
Please refer to new review page,
Review: https://review.openstack.org/144065

Change abandoned by Hiroyuki Eguchi (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/141684
Reason: Abandoned because ChangeID is different from the commit on master.
Please refer to new review page,
Review: https://review.openstack.org/144067

Alan Pevec (apevec) on 2015-01-14
Changed in nova:
importance: Undecided → High
tags: removed: icehouse-backport-potential juno-backport-potential

Reviewed: https://review.openstack.org/144065
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cca94d032b7ee72a5ff98aa664353ef50e877454
Submitter: Jenkins
Branch: stable/icehouse

commit cca94d032b7ee72a5ff98aa664353ef50e877454
Author: Hiroyuki Eguchi <email address hidden>
Date: Thu Nov 20 10:41:36 2014 +0900

    Fix connecting unnecessary iSCSI sessions issue

    In Icehouse with "iscsi_use_multipath=true", attaching a multipath
    iSCSI volume may create unnecessary iSCSI sessions.

    The iscsiadm discovery command in connect_volume() returns all of the
    targets in the Cinder node, not just the ones related to the multipath
    volume which is specified by iqn. If the storage has many targets,
    connecting to all these volumes will also result in many unnecessary
    connections.

    There are two types of iSCSI multipath devices. One which shares
    the same iqn between multiple portals, and the other which use
    different iqns on different portals. connect_volume() needs to
    identify the type by checking iscsiadm the output if the iqn is
    used by multiple portals.

    This patch changes the behavior of attaching volume:

       1. Identify the type by checking the iscsiadm output.
       2. Connect to the correct targets by connect_to_iscsi_portal().

    Closes-Bug: #1382440

    (cherry picked from commit fb0de106f2f15604750bafc318ba06c41070cc35)

    Conflicts:
            nova/tests/unit/virt/libvirt/test_volume.py

    Change-Id: I488ad0c09bf26a609e27d67b9ef60b65bb45e0ad

Reviewed: https://review.openstack.org/144066
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7307dbaf6b9734ddd456c7b0c1d2c85e053ce61f
Submitter: Jenkins
Branch: stable/juno

commit 7307dbaf6b9734ddd456c7b0c1d2c85e053ce61f
Author: Hiroyuki Eguchi <email address hidden>
Date: Thu Dec 4 15:12:11 2014 +0900

    Fix disconnecting necessary iSCSI sessions issue

    In Icehouse with "iscsi_use_multipath=true", detaching a multipath
    iSCSI volume kills all iSCSI volumes visible from the nova compute node.
    When we use different targets(IQNs) associated with same portal for
    each different multipath device, all of the targets will be deleted
    via disconnect_volume().

    This patch fixes the behavior of detaching volume:

      1. Extract the targets for the detached multipath device.
      2. Delete/disconnect the targets for the detached multipath device.

    Closes-Bug: #1382440

    (cherry picked from commit 36aeedfd5eeb0345d66fa8456ed6a9447a6514a0)

    Conflicts:
            nova/tests/unit/virt/libvirt/test_volume.py

    Change-Id: I38eafdaee03d136282cfde1fd013e322a4256cc4

Reviewed: https://review.openstack.org/144067
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=52103be3e152b53ea0ec542173305cfe3bb1a08d
Submitter: Jenkins
Branch: stable/icehouse

commit 52103be3e152b53ea0ec542173305cfe3bb1a08d
Author: Hiroyuki Eguchi <email address hidden>
Date: Thu Dec 4 15:12:11 2014 +0900

    Fix disconnecting necessary iSCSI sessions issue

    In Icehouse with "iscsi_use_multipath=true", detaching a multipath
    iSCSI volume kills all iSCSI volumes visible from the nova compute node.
    When we use different targets(IQNs) associated with same portal for
    each different multipath device, all of the targets will be deleted
    via disconnect_volume().

    This patch fixes the behavior of detaching volume:

      1. Extract the targets for the detached multipath device.
      2. Delete/disconnect the targets for the detached multipath device.

    Closes-Bug: #1382440

    (cherry picked from commit 36aeedfd5eeb0345d66fa8456ed6a9447a6514a0)

    Conflicts:
            nova/tests/unit/virt/libvirt/test_volume.py

    Change-Id: I38eafdaee03d136282cfde1fd013e322a4256cc4

Reviewed: https://review.openstack.org/144064
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4c1f28a0f467d749ebf406f970b3cea4888ff1b0
Submitter: Jenkins
Branch: stable/juno

commit 4c1f28a0f467d749ebf406f970b3cea4888ff1b0
Author: Hiroyuki Eguchi <email address hidden>
Date: Thu Nov 20 10:41:36 2014 +0900

    Fix connecting unnecessary iSCSI sessions issue

    In Icehouse with "iscsi_use_multipath=true", attaching a multipath
    iSCSI volume may create unnecessary iSCSI sessions.

    The iscsiadm discovery command in connect_volume() returns all of the
    targets in the Cinder node, not just the ones related to the multipath
    volume which is specified by iqn. If the storage has many targets,
    connecting to all these volumes will also result in many unnecessary
    connections.

    There are two types of iSCSI multipath devices. One which shares
    the same iqn between multiple portals, and the other which use
    different iqns on different portals. connect_volume() needs to
    identify the type by checking iscsiadm the output if the iqn is
    used by multiple portals.

    This patch changes the behavior of attaching volume:

       1. Identify the type by checking the iscsiadm output.
       2. Connect to the correct targets by connect_to_iscsi_portal().

    Closes-Bug: #1382440

    (cherry picked from commit fb0de106f2f15604750bafc318ba06c41070cc35)

    Conflicts:
            nova/tests/unit/virt/libvirt/test_volume.py

    Change-Id: I488ad0c09bf26a609e27d67b9ef60b65bb45e0ad

Looks like this is fixed (https://review.openstack.org/#/c/132490/)

Changed in nova:
status: In Progress → Fix Committed

Related fix proposed to branch: master
Review: https://review.openstack.org/165373

Thierry Carrez (ttx) on 2015-04-30
Changed in nova:
milestone: kilo-1 → 2015.1.0
Thierry Carrez (ttx) on 2015-04-30
Changed in nova:
status: Fix Committed → Fix Released
Changed in os-brick:
assignee: nobody → Danny Wilson (daniel-wilson)
Changed in os-brick:
importance: Undecided → High
status: New → Confirmed
Kyrylo Romanenko (kromanenko) wrote :

>>>instance fails to access the attached multipath devices

Could you clarify how to access volume from instance in test reasons?
Should i make filesystem on attached blockdevice and mount it to instance VM?

Changed in os-brick:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/188158
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=531398504fcdcffb37e73a254f5a3da1854ef41e
Submitter: Jenkins
Branch: master

commit 531398504fcdcffb37e73a254f5a3da1854ef41e
Author: Tomoki Sekiyama <email address hidden>
Date: Tue Jun 2 18:38:34 2015 -0400

    Fix disconnecting necessary iSCSI sessions issue

    With "iscsi_use_multipath=true", detaching a multipath iSCSI
    volume kills all iSCSI volumes visible from the attached node.
    When we use different targets(IQNs) associated with same portal
    for each different multipath device, all of the targets will
    be deleted via disconnect_volume().

    This patch fixes the behavior of detaching volume:

      1. Extract the targets for the detached multipath device.
      2. Delete/disconnect the targets for the detached multipath
         device.

    (Ported from I38eafdaee03d136282cfde1fd013e322a4256cc4)

    Co-Authored-By: Hiroyuki Eguchi <email address hidden>
    Closes-Bug: #1382440
    Change-Id: I5fdbb5765642bac7263ff616ae4b006d9504547a

Changed in os-brick:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/188159
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=a8386d5cbe732a4e3f4daf836a6efdf69ee6e11f
Submitter: Jenkins
Branch: master

commit a8386d5cbe732a4e3f4daf836a6efdf69ee6e11f
Author: Tomoki Sekiyama <email address hidden>
Date: Wed Jun 3 13:42:34 2015 -0400

    Fix connecting unnecessary iSCSI sessions issue

    With "iscsi_use_multipath=true", attaching a multipath
    iSCSI volume may create unnecessary iSCSI sessions.
    The iscsiadm discovery command in connect_volume() returns all of
    the targets in the Cinder node, not just the ones related to
    the multipath volume which is specified by iqn. If the storage
    has many targets, connecting to all these volumes will also
    result in many unnecessary connections.

    There are two types of iSCSI multipath devices.
    One which shares the same iqn between multiple portals,
    and the other which use different iqns on different portals.
    connect_volume() needs to identify the type by checking iscsiadm
    the output if the iqn is used by multiple portals.

    This patch changes the behavior of attaching volume:

       1. Identify the type by checking the iscsiadm output.
       2. Connect to the correct targets by connect_to_iscsi_portal().

    (Ported from I488ad0c09bf26a609e27d67b9ef60b65bb45e0ad)

    Co-Authored-By: Hiroyuki Eguchi <email address hidden>
    Related-Bug: #1382440
    Change-Id: I4c9b173cfab387a55772709242eed69c10168c0f

Changed in os-brick:
milestone: none → 0.3.0
status: Fix Committed → Fix Released

Change abandoned by Keiichi KII (<email address hidden>) on branch: master
Review: https://review.openstack.org/165370
Reason: This patch is not needed because of switching to os-brick https://review.openstack.org/#/c/175569/32

Change abandoned by Keiichi KII (<email address hidden>) on branch: master
Review: https://review.openstack.org/165373
Reason: This patch is not needed because of switching to os-brick https://review.openstack.org/#/c/175569/32

Eric Harney (eharney) on 2016-02-04
Changed in cinder:
status: New → Invalid
Eric Harney (eharney) wrote :
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers