Device descriptor not removed with different iqn and multipath enabled.

[Jorge Niedbalski]

[Environment]

OpenStack Kilo
Trusty 14.04.4

[Description]

if the attached multipath devices doesn't have same iqn like regular lvm+iscsi backend, in_use will be false.

In that case,_disconnect_volume_multipath_iscsi() returns without calling _remove_multipath_device_descriptor().

[Reproduction]

1) Enable cinder LVM ISCSI on /etc/cinder/cinder.conf

volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver

2) Enable iscsi_use_multipath on /etc/nova/nova.conf on your compute nodes:

iscsi_use_multipath = True

3) Create 3 cinder volumes

$ cinder create 1
$ cinder create 1
$ cinder create 1

$ cinder list

ubuntu@niedbalski2-bastion:~/specs/1374999$ cinder list
+--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+
| 10844be6-8f86-414f-a10e-e1a31e2ba6e7 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 |
| 1648d24c-0d65-4377-9fa5-6d3aeb8b1291 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 |
| 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 |
+--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+

4) Attach them to nova

$ nova volume-attach instance_id 10844be6-8f86-414f-a10e-e1a31e2ba6e7
$ nova volume-attach instance_id 1648d24c-0d65-4377-9fa5-6d3aeb8b1291
$ nova volume-attach instance_id 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f

5) Check on the nova-compute unit for the current multipath/session status

tcp: [1] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-10844be6-8f86-414f-a10e-e1a31e2ba6e7
tcp: [2] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-1648d24c-0d65-4377-9fa5-6d3aeb8b1291
tcp: [3] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f

Multipath:

root@juju-1374999-machine-10:/home/ubuntu# multipath -ll
33000000300000001 dm-2 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 10:0:0:1 sdb 8:16 active ready running
33000000100000001 dm-0 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 8:0:0:1 sdg 8:96 active ready running
33000000200000001 dm-1 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 9:0:0:1 sda 8:0 active ready running

6) Detach the current volumes.

First.

ubuntu@niedbalski2-bastion:~/specs/1374999$ nova volume-detach b0a14447-5740-408a-b96f-a1e904b229e5 10844be6-8f86-414f-a10e-e1a31e2ba6e7

ubuntu@niedbalski2-bastion:~/specs/1374999$ juju ssh 10 "sudo multipath -ll"
33000000300000001 dm-2 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 10:0:0:1 sdb 8:16 active ready running
33000000100000001 dm-0 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  `- #:#:#:# - #:# active faulty running
33000000200000001 dm-1 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 9:0:0:1 sda 8:0 active ready running

Second raises the faulty state

33000000300000001 dm-2 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 10:0:0:1 sdb 8:16 active ready running
33000000100000001 dm-0 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  `- #:#:#:# - #:# active faulty running
33000000200000001 dm-1 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  `- #:#:#:# - #:# active faulty running

Third, raises the faulty state also

sudo: unable to resolve host juju-1374999-machine-10
33000000300000001 dm-2 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  `- #:#:#:# - #:# active faulty running
33000000100000001 dm-0 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  `- #:#:#:# - #:# active faulty running
33000000200000001 dm-1 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  `- #:#:#:# - #:# active faulty running

Manual fix.

root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000300000001
root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000100000001
root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000200000001

[Keiichi KII (k-keiichi)]

This issue seems same as bug 1374999, which has a patch by Felipe Reyes:
https://review.openstack.org/135382

The issue may be solved by the patch.

[John Garbutt (johngarbutt)]

Looks like this is invalid now os-brick has merged.

[Jorge Niedbalski (niedbalski)]

@johngarbutt,

I think this is still affecting stable releases.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/135382
Reason: It sounds like this patch is now obsolete. Please restore it if that isn't the case.

[John Garbutt (johngarbutt)]

So to fix this on stable releases, the bug should be targeted for that stable release.this is still invalid for master.