Disconnecting a volume with multipath generates excessive multipath calls

[Sam Morrison]

I have a compute node with 20 volumes attached using iscsi and multipath.
Each multipath device has 4 iscsi devices.

When I disconnect a volume it generates 779 multipath -ll calls.

iscsiadm -m node --rescan
iscsiadm -m session --rescan
multipath - r

multipath -ll /dev/sdch
multipath -ll /dev/sdcg
multipath -ll /dev/sdcf
multipath -ll /dev/sdce
multipath -ll /dev/sdcd
multipath -ll /dev/sdcc
multipath -ll /dev/sdcb
multipath -ll /dev/sdca
multipath -ll /dev/sdbz
multipath -ll /dev/sdby
multipath -ll /dev/sdbx
multipath -ll /dev/sdbw
multipath -ll /dev/sdbv
multipath -ll /dev/sdbu
multipath -ll /dev/sdbt
multipath -ll /dev/sdbs
multipath -ll /dev/sdbr
multipath -ll /dev/sdbq
multipath -ll /dev/sdbp
multipath -ll /dev/sdbo
multipath -ll /dev/sdbn
multipath -ll /dev/sdbm
multipath -ll /dev/sdbl
multipath -ll /dev/sdbk
multipath -ll /dev/sdbj
multipath -ll /dev/sdbi
multipath -ll /dev/sdbh
multipath -ll /dev/sdbg
multipath -ll /dev/sdbf
multipath -ll /dev/sdbe
multipath -ll /dev/sdbd
multipath -ll /dev/sdbc
multipath -ll /dev/sdbb
multipath -ll /dev/sdba
----
...... And so on for 779 times
cp /dev/stdin /sys/block/sdcd/device/delete
cp /dev/stdin /sys/block/sdcc/device/delete
cp /dev/stdin /sys/block/sdcb/device/delete
cp /dev/stdin /sys/block/sdca/device/delete
multipath - r

----

[John Garbutt (johngarbutt)]

Sounds more like a cinder issue, although the detach probably gets called through nova.

[Sam Morrison (sorrison)]

This is due to inefficient code in nova.virt.libvirt.volume

The massive amount of multipath calls is from the code to figure out other IQNs attached to the compute node.
There is basically a nested for loop, this code can be changed to make it more efficient.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/163733

[Keiichi KII (k-keiichi)]

I confirmed the same issue occurred in the latest code
and found that detaching a volume takes too much time
when many volumes(multipath devices) are attached.

In my environment(lvm+iscsi backend for testing) with 100 volumes
attached, detaching a volume takes more than 8 hours to be finished.
Because a multipath -ll call against many multipath devices takes
a few seconds and massive multipath calls are generated as Sam said
in comment #2.

I have a solution to fix this issue. Please review my patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Keiichi KII (<email address hidden>) on branch: master
Review: https://review.openstack.org/163733
Reason: This patch is already merged into os-brick library:
https://review.openstack.org/#/c/190864/

[Keiichi KII (k-keiichi)]

I tested it in my test environment.
It seems this performance issue is already resolved in master for
Liberty after migrating to os-brick library.