Libvirt: Detaching volume from instance on host with many attached volumes is very slow

[Alex Meade]

When many volumes are attached to instances on the same compute host (with multipath enabled), volume detach is very slow and get slower as more volumes are attached.

For example:
1. compute1 is a compute node with instance1 and instance2.
2. instance1 has 10 volumes attached while instance2 has a single volume attached.
3. Issue a detach for the volume attached to instance2
4. Nova spends >20 minutes executing the 'multipath -ll' command for every device on the hypervisor
5. Finally the detach completes successfully

The following log is output in n-cpu many, many times during the detach call. Repeated many times for each volume device.
http://paste.openstack.org/show/192981/

Environment details:
nova.conf virt driver
[libvirt]
iscsi_use_multipath = True
vif_driver = nova.virt.libvirt.vif.LibvirtGenericVIFDriver
inject_partition = -2
live_migration_uri = qemu+ssh://ameade@%s/system
use_usb_tablet = False
cpu_mode = none
virt_type = kvm

cinder.conf backend
[eseries]
volume_backend_name = eseries
volume_driver = cinder.volume.drivers.netapp.common.NetAppDriver
netapp_storage_family = eseries
netapp_storage_protocol = iscsi
netapp_server_hostname = localhost
netapp_server_port = 8081
netapp_webservice_path = /devmgr/v2
netapp_controller_ips = 10.78.152.114,10.78.152.115
netapp_login = rw
netapp_password = xx
netapp_storage_pools = DDP
use_multipath_for_image_xfer = True
netapp_sa_password = password
netapp_enable_multi_attach=True

[Sean Dague (sdague)]

Is this reproducable with an open source backend?

[John Griffith (john-griffith)]

I just loaded this up in a fresh devstack wit LVM, used 10 volumes and didn't run into any issues. Seems like it's an issue with the specific multi-path implementation maybe?

http://paste.openstack.org/show/196142/

Anyway, not enough info to see why this is marked as a Cinder bug but happy to open it back up and help figure things out if you get some more info on it.

Thanks!

[Keiichi KII (k-keiichi)]

We can reproduce this issue with "iscsi_use_multipath=true" and
lvm+iscsi backend(LVMISCSIDriver).
In my environment with 30 volumes attached, detaching a volume takes
about 10 minutes. The time to detach a volume depends on the number
of iSCSI devices and paths for multipath device.

This bug is related to bug 1277316.
And I'm working for this issue(https://review.openstack.org/163733).

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]