libvirt: connect_volume scans all LUNs, which takes >2 mins when host is connected with about 900 Luns

[Shen Wang]

Tested OpenStack version: IceHouse 2014.1, master branch still has this issue.
Host version: CentOS 6, 2.6.32-431.el6.x86_64

I have done some work to test the performance of LUN scanning with multipath, use the way like what Nova dose.
In my test, The host was connected with almost 900 LUNs.
1. I use 'iscsiadm' with '--rescan' to discover LUNs, which takes almost 15s. It seems '--rescan' cause kernel to rescan all the LUNs which has already been connected to the host.
2. I use 'multipath -r' to construct multipath devices, which takes almost 2 minutes. I found that 'multipath -r' will reconstructs all multipath devices against the LUNs
The two steps scans all of the LUNs, and totally costs more then 2 minutes.

According to "connect_volume" in nova.virt.libvirt.volume.py: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume.py#L252, Nova also uses the tow steps to detect new multipath volume, this two steps will scan all of the LUNs, including all the others which already connected. So if a host has a large number of LUNs connected to it, the connect_volume will be very slow.

I think connect_volume needn't scan all of the LUNs, only need scan the LUN specified by connection_info.

[Shen Wang (peter.w)]

at https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume.py#L590
brackets also missed for '-r'

[Thang Pham (thang-pham)]

This bug seems related to bug 1359617, which has a potential patch up for review: https://review.openstack.org/#/c/116998/

[Shen Wang (peter.w)]

Hi, Thang
I have read "bug 1359617" before I report this bug, the two bugs are not related to the same issue.

This bug is related to the LUN scanning, not twice calls of connect_volume

[Sean Dague (sdague)]

Honestly, I consider this a performance wishlist item. If there are patches that's cool, but this seems like a very edge case configuration.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/124978

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/124978
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Davanum Srinivas (DIMS) (dims-v)]

Removing "In Progress" status and assignee as change is abandoned.

[Markus Zoeller (markus_z) (mzoeller)]

This wishlist bug has been open a year without any activity. I'm going to move it to "Opinion / Wishlist", which is an easily-obtainable queue of older requests that have come on.

In case you want to work on that, consider writing a blueprints [1] and spec [2]. I'll recommend to read [3] if not yet done. The effort to implement the requested feature is then driven only by the blueprint (and spec).

References:
[1] https://blueprints.launchpad.net/nova/
[2] https://github.com/openstack/nova-specs
[3] https://wiki.openstack.org/wiki/Blueprints