iSCSI volume detach does not correctly remove the multipath device descriptors

[SamP]

[Impact]

iSCSI volume detach does not correctly remove the multipath device descriptors.

The multipath devices are left on the compute node and multipath tools will occaisionally send IOs to known multipath devices.

[Test Case]

tested environment:
nova-compute on Ubuntu 14.04.1, iscsi_use_multipath=True and iSCSI volume backend is EMC VNX 5300.

 I created 3 cinder volumes and attached them to a nova instance. Then I detach them one by one. First 2 volumes volumes detached successfully. 3rd volume also successfully detached but ended up with failed multipaths.
Here is the terminal log for last volume detach.

openstack@W1DEV103:~/devstack$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
|
 ID
 | Status | Name | Size | Volume Type | Bootable |
 Attached to
 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 56a63288-5cc0-4f5c-9197-cde731172dd8 | in-use | None | 1 |
 None
 | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
openstack@W1CN103:/etc/iscsi$ date;sudo multipath -l
Fri Sep 19 21:38:13 JST 2014
360060160cf0036002d1475f6e73fe411 dm-2 DGC,VRAID
size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| |- 4:0:0:42 sdb 8:16 active undef running
| |- 5:0:0:42 sdd 8:48 active undef running
| |- 6:0:0:42 sdf 8:80 active undef running
| `- 7:0:0:42 sdh 8:112 active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
|- 11:0:0:42 sdp 8:240 active undef running
|- 8:0:0:42 sdj 8:144 active undef running
|- 9:0:0:42 sdl 8:176 active undef running
`- 10:0:0:42 sdn 8:208 active undef running
openstack@W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
Fri Sep 19 21:38:19 JST 2014
tcp: [10] 172.23.58.228:3260,4 iqn.1992-04.com.emc:cx.fcn00133400150.a7
tcp: [3] 172.23.58.238:3260,8 iqn.1992-04.com.emc:cx.fcn00133400150.b7
tcp: [4] 172.23.58.235:3260,20 iqn.1992-04.com.emc:cx.fcn00133400150.b4
tcp: [5] 172.23.58.236:3260,6 iqn.1992-04.com.emc:cx.fcn00133400150.b5
tcp: [6] 172.23.58.237:3260,19 iqn.1992-04.com.emc:cx.fcn00133400150.b6
tcp: [7] 172.23.58.225:3260,16 iqn.1992-04.com.emc:cx.fcn00133400150.a4
tcp: [8] 172.23.58.226:3260,2 iqn.1992-04.com.emc:cx.fcn00133400150.a5
tcp: [9] 172.23.58.227:3260,17 iqn.1992-04.com.emc:cx.fcn00133400150.a6

openstack@W1DEV103:~/devstack$ nova volume-detach 5bd68785-4acf-43ab-ae13-11b1edc3a62e
56a63288-5cc0-4f5c-9197-cde731172dd8
openstack@W1DEV103:~/devstack$
openstack@W1DEV103:~/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
|
 ID
 | Status | Name | Size | Volume Type | Bootable |
 Attached to
 |
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
| 56a63288-5cc0-4f5c-9197-cde731172dd8 | detaching | None | 1 |
 None
 | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e|

+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
openstack@W1DEV103:~/devstack$
openstack@W1DEV103:~/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
|
 ID
 | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 56a63288-5cc0-4f5c-9197-cde731172dd8 | available | None | 1 |
 None
 | false |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
|
openstack@W1CN103:/etc/iscsi$ date;sudo multipath -l
Fri Sep 19 21:39:23 JST 2014
360060160cf0036002d1475f6e73fe411 dm-2 ,
size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| |- #:#:#:# - #:# active undef running
| |- #:#:#:# - #:# active undef running
| |- #:#:#:# - #:# active undef running
| `- #:#:#:# - #:# active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
|- #:#:#:# - #:# active undef running
|- #:#:#:# - #:# active undef running
|- #:#:#:# - #:# active undef running
`- #:#:#:# - #:# active undef running
openstack@W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
Fri Sep 19 21:39:27 JST 2014
iscsiadm: No active sessions.

Then I manually removed the multipaths,
openstack@W1CN103:/etc/iscsi$ sudo multipath -f 360060160cf0036002d1475f6e73fe411
openstack@W1CN103:/etc/iscsi$ sudo multipath -l
openstack@W1CN103:/etc/iscsi$

 I think the problem is in,
virt/libvirt/volume.py:LibvirtISCSIVolumeDriver
 def _disconnect_volume_multipath_iscsi(self, iscsi_properties, multipath_device):

End of this method executes following code to call remove_multipath_device_descriptor for remove the multipaths by multipath -f before return.
        self._remove_multipath_device_descriptor(multipath_device)
        return
Therefore, first two volumes worked fine. However, when it comes to the last device (in this case 3rd one), this method return without calling _remove_multipath_device_descriptor due to following if statement.

        if not in_use:
            # disconnect if no other multipath devices with same iqn
            self._disconnect_mpath(iscsi_properties, ips_iqns)
            return
It just disconnect them but not remove them.
One of the reasons why we have to remove them is,
https://bugs.launchpad.net/nova/+bug/1223975

IMO, we should call _remove_multipath_device_descriptor in above if statement before return.

[Regression Potential]

- Low: the change is relatively minor and allows the code to also remove the device when the last path is removed. Should a regression occur, it should be limited to iscsi-multipath device detachment, which is a small portion of installations.

[Mark Brown (mstevenbrown)]

Adding archive with the openstack cinder and nova logs for Control and Compute nodes.
And also their kernel logs.

W1DEV103 is the Controller node. (Cinder-vol, scheduler, api are also in this host.)
W1CN103 is the Compute node. (nova-compute and its needed other process are running on this host.)

Following detach API request caused the problem.
The request id: req-1f8a6f05-3751-4bdc-a9bf-a54318b2d985.

The PW for the zip file is: "lkjse89'&3" (without the quotes).

[Rafael David Tinoco (inaddy)]

Reading statement above, It looks like the following commit:

https://github.com/openstack/nova/commit/8a2a10807f22b68b521dd02174cd06648fd1dc56

To fix this specific bug:

https://bugs.launchpad.net/nova/+bug/1223975

Has a minor error on the logic, not removing latest path of one particular removed device (from cinder).

^^^ to summarize and make things clear.

Providing a hotfix soon so we can check if it solves this issues.

Thank you

-Rafael Tinoco

[Rafael David Tinoco (inaddy)]

Finally, after some time, I could reproduce this issue using regular lvm+iscsi backend:

root@ostacktrustycomp:~# multipath -F
root@ostacktrustycomp:~# multipath -ll
<nothing>

root@ostacktrustycontrol:~# nova volume-attach demo-instance1 `nova volume-list | grep avai | awk '{print $2}'`

root@ostacktrustycomp:~# multipath -ll
33000000100000001 dm-1 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 3:0:0:1 sda 8:0 active ready running

root@ostacktrustycontrol:~# nova volume-detach demo-instance1 `nova volume-list | grep "volume" | awk '{print $2}'`

root@ostacktrustycomp:~# multipath -ll
33000000100000001 dm-1 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  `- #:#:#:# - #:# active faulty running

Although latest path was "faulty"... it was still there.

I'm having some problems on getting the second path activated on the nova-compute node (using cinder lvm backend + libvirt multipath) but this is showing the behavior..

My next step is to review libvirt multipath code (for the addition of the second path) AND the removal issue (reported here).

Thank you very much

Rafael Tinoco

[Rafael David Tinoco (inaddy)]

Actually the code is returning on:

        if not devices:
            # disconnect if no other multipath devices
            self._disconnect_mpath(iscsi_properties, ips_iqns)
            return

So it never gets beyond that (at least for my test case).

For two devices, it passes the statement above...

root@ostacktrustycomp:~# multipath -ll
33000000100000001 dm-1 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 7:0:0:1 sda 8:0 active ready running
33000000200000001 dm-2 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 5:0:0:1 sdc 8:32 active ready running

Removing multipath device:

root@ostacktrustycontrol:~# nova volume-detach 7165ad88-8000-4c9b-be19-faecbc3e8638 a6359a32-1263-4778-bc7c-3ba3158b208a

root@ostacktrustycomp:~# multipath -ll
33000000100000001 dm-1 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  `- #:#:#:# - #:# active faulty running

Removing latest device:

root@ostacktrustycontrol:~# nova volume-detach 7165ad88-8000-4c9b-be19-faecbc3e8638 f9c46dd2-caa6-468a-a9f1-4c458f2d3323

root@ostacktrustycomp:~# multipath -ll
33000000100000001 dm-1 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=enabled
  `- #:#:#:# - #:# failed faulty running

Latest path stays failed....

[Rafael David Tinoco (inaddy)]

Removing the multipath on this statement:

        if not devices:
            # disconnect if no other multipath devices
            self._remove_multipath_device_descriptor(multipath_device)
            self._disconnect_mpath(iscsi_properties, ips_iqns)
            return

seem to have solved the problem.

Creating one PPA with nova package + this hotfix for feedbacks...

Thank you

Rafael Tinoco

[Rafael David Tinoco (inaddy)]

Attaching proposed fix to this bug.

A "hotfixed" version can be found in the following PPA:

https://launchpad.net/~inaddy/+archive/ubuntu/1374999

Please test fix and provide me feedback.

PS: Meanwhile I'm also proposing this fix upstream.

[SamP (sampath-priyankara)]

Thanks you for the patch.
I include this patch for our testing schedule, and will let you know the result.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "trusty_nova_2014.1.3-0ubuntu2.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[SamP (sampath-priyankara)]

Hi Tinoco,
I tested this patch with EMC VNX 5300 as iSCSI cinder backend.
Confirmed that this patch solve the issue.

Here is the console log verification of
before apply the patch.
http://paste.openstack.org/show/132629/

And after apply the patch,
http://paste.openstack.org/show/132630/

The issue is no longer exist.

i really appreciate it if you could back port this to stable/juno

Thank you.

[Rafael David Tinoco (inaddy)]

I have created the following PPA:

https://launchpad.net/~inaddy/+archive/ubuntu/lp1374999

containing a hotfixed version of nova for Trusty.

I'll keep this PPA available, updating when needed, until we get upstream fix accepted and backported. As soon as the fix is backported we can ask for a "SRU" for Trusty's nova version.

I'm assigning this case temporary to Felipe (responsible to propose the fix upstream) while I'm on vacation.

Let me know if there any issues.

Thank you.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/135382

[Brian Murray (brian-murray)]

Has this been fixed upstream yet?

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/135382
Reason: It sounds like this patch is now obsolete. Please restore it if that isn't the case.

[Matt Riedemann (mriedem)]

Can this be tried against liberty or mitaka nova when we're using the os-brick library which had other fixes for multipath issues than did nova?

[Billy Olsen (billy-olsen)]

Marking this is confirmed against the Ubuntu Cloud Archive for Kilo, Juno, and Trusty which are still supported from the Ubuntu perspective and is known not to include the os-brick library dependencies. Certainly, the testing for the change to os-brick needs to be verified that the problem is fixed there, so leaving that as incomplete.

[Tobias Urdin (tobias-urdin)]

Could you post your multipath.conf configuration? We had the same issue with multipath devices being stuck/not removed which also blocked our ability to do live migrations. See https://bugs.launchpad.net/nova/+bug/1525802

[Lee Yarwood (lyarwood)]

Moving to invalid as this should no longer reproduce against Liberty or Mitaka after the move os-brick. Please reopen and reassign to os-brick if this issue persists.

[Corey Bryant (corey.bryant)]

Thanks for the patches Billy. I've uploaded to kilo-staging and juno-staging and they'll be working their way to *-proposed for testing.

Just a few comments on the icehouse patch. The other bugs that are fixed in the icehouse patch will need to be updated with SRU information (Impact, Test Case, Regression Potential).

Also I moved the drop of 'return' in the 'not in_use path' from d/p/Fix-wrong-used-ProcessExecutionError-exception.patch to d/p/Detach-iSCSI-latest-path-for-latest-disk.patch.

And I added the following to the juno and icehouse detach iscsi patch headers:
Forwarded: https://review.openstack.org/#/c/135382/
Bug: https://bugs.launchpad.net/nova/+bug/1374999

Thanks,
Corey

[Ryan Beisner (1chb1n)]

Hello SamP, or anyone else affected,

Accepted nova into kilo-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:kilo-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-kilo-needed to verification-kilo-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-kilo-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Juno is EOL so we're going to stop working on that.

[Martin Pitt (pitti)]

There is an SRU waiting in the trusty-proposed queue for this. Please clarify in which Ubuntu release(s) this is already fixed, or upload the fix to yakkety, so that the trusty SRU can proceed.

[Ryan Beisner (1chb1n)]

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Ryan Beisner (1chb1n)]

This bug was fixed in the package nova - 1:2015.1.4-0ubuntu2
---------------

 nova (1:2015.1.4-0ubuntu2) trusty-kilo; urgency=medium
 .
   * d/p/fix-iscsi-detach.patch (LP: #1374999)
     - Clear latest path for last remaining iscsi disk to ensure
       disk is properly removed.
 .
 nova (1:2015.1.4-0ubuntu1) trusty-kilo; urgency=medium
 .
   * New upstream stable release (LP: #1580334).
   * d/p/skip-proxy-test.patch: Skip test_ssl_server and test_two_servers as
     they are hitting ProxyError during package builds.
 .
 nova (1:2015.1.3-0ubuntu1) trusty-kilo; urgency=medium
 .
   * New upstream stable release (LP: #1559215).
 .
 nova (1:2015.1.2-0ubuntu2) vivid; urgency=medium
 .
   * d/control: Bump oslo.concurrency to >= 1.8.2 (LP: #1518016).
 .
 nova (1:2015.1.2-0ubuntu1) vivid; urgency=medium
 .
   * Resynchronize with stable/kilo (68e9359) (LP: #1506058):
     - [68e9359] Fix quota update in init_instance on nova-compute restart
     - [d864603] Raise InstanceNotFound when save FK constraint fails
     - [db45b1e] Give instance default hostname if hostname is empty
     - [61f119e] Relax restrictions on server name
     - [2e731eb] Remove unnecessary 'context' param from quotas reserve method
                 call
     - [5579928] Updated from global requirements
     - [08d1153] Don't expect meta attributes in object_compat that aren't in the
                 db obj
     - [5c6f01f] VMware: pass network info to config drive.
     - [17b5052] Allow to use autodetection of volume device path
     - [5642b17] Delete orphaned instance files from compute nodes
     - [8110cdc] Updated from global requirements
     - [1f5b385] Hyper-V: Fixes serial port issue on Windows Threshold
     - [24251df] Handle FC LUN IDs greater 255 correctly on s390x architectures
     - [dcde7e7] Update obj_reset_changes signatures to match
     - [e16fcfa] Unshelving volume backed instance fails
     - [8fccffd] Make pagination tolerate a deleted marker
     - [587092c] Fix live-migrations usage of the wrong connector information
     - [8794b93] Don't check flavor disk size when booting from volume
     - [c1ad497] Updated from global requirements
     - [0b37312] Hyper-V: Removes old instance dirs after live migration
     - [2d571b1] Hyper-V: Fixes live migration configdrive copy operation
     - [07506f5] Hyper-V: Fix SMBFS volume attach race condition
     - [60356bf] Hyper-V: Fix missing WMI namespace issue on Windows 2008 R2
     - [83fb8cc] Hyper-V: Fix virtual hard disk detach
     - [6c857c2] Updated from global requirements
     - [0313351] Compute: replace incorrect instance object with dict
     - [9724d50] Don't pass the service catalog when making glance requests
     - [b5020a0] libvirt: Kill rsync/scp processes before deleting instance
     - [3f337f8] Support host type specific block volume attachment
     - [cb2a8fb] Fix serializer supported version reporting in object_backport
     - [701c889] Execute _poll_shelved_instances only if shelved_offload_time is
                 > 0
     - [eb3b1c8] Fix rebuild of an instance with a volume attached
     - [e459add] Handle unexpected clear events call
     - [8280575] Support ssh-keygen of OpenS...

[Billy Olsen (billy-olsen)]

@pitti - this bug does not apply to openstack releases >= Liberty due to upstream code changes. The only Ubuntu release this applies to that is not EOL is Trusty. The fix is applicable to the following:

- Trusty
- UCA precise-icehouse
- UCA trusty-juno (now EOL, so excluded)
- UCA trusty-kilo

Hope this clarifies!

[Martin Pitt (pitti)]

Hello SamP, or anyone else affected,

Accepted nova into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/1:2014.1.5-0ubuntu1.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[SamP (sampath-priyankara)]

Hello Martin,

 Thank you for the fix.
 I am really sorry to inform that I am not able to test this patch in my current environment, since it does not have iscsi backend for cinder.

[Ryan Beisner (1chb1n)]

Hello SamP, or anyone else affected,

Accepted nova into icehouse-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:icehouse-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-icehouse-needed to verification-icehouse-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-icehouse-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Billy Olsen (billy-olsen)]

Sorry for the delay, but I finally finished the trusty-proposed SRU verification this evening (Yay).

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 1:2014.1.5-0ubuntu1.5

---------------
nova (1:2014.1.5-0ubuntu1.5) trusty; urgency=medium

  * Fix live migration usage of the wrong connector (LP: #1475411)
    - d/p/Fix-live-migrations-usage-of-the-wrong-connector-inf.patch
  * Fix wrong used ProcessExecutionError exception (LP: #1308839)
    - d/p/Fix-wrong-used-ProcessExecutionError-exception.patch
  * Clean up iSCSI multipath devices in Post Live Migration (LP: #1357368)
    - d/p/Clean-up-iSCSI-multipath-devices-in-Post-Live-Migrat.patch
  * Detach iSCSI latest path for latest disk (LP: #1374999)
    - d/p/Detach-iSCSI-latest-path-for-latest-disk.patch

 -- Billy Olsen <email address hidden> Fri, 29 Apr 2016 15:35:01 -0700