3PAR driver performance under high load

[Anthony Lee]

During high load situations where multiple requests for volume creation/attachment and volume deletion/detachment are happening errors can occur.

This is being caused by the locks the 3PAR FC and iSCSI drivers have in place. By removing the locks and having separate connections for each volume request the problem can be avoided.

Example of an error that occurs under high load:

2014-10-14 11:44:09.900 ERROR oslo.messaging.rpc.dispatcher [req-2eb2202b-2e70-4c8a-a9b4-e29e23f1d167 1b199c6813fa4742bd20be7866979947 8aa0bd4fd1c347a0881c214babc5e4e0] Exception during message handling:
Bad or unexpected response from the storage volume backend API: Failure in clear_volume_key_value_pair:Not found (HTTP 404) 180 - Object key does not exist
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 127, in lvo_inner1
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher return lvo_inner2(inst, context, volume_id, **kwargs)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/openstack/common/lockutils.py", line 233, in inner
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher retval = f(*args, **kwargs)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 126, in lvo_inner2
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher return f(*_args, **_kwargs)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 731, in detach_volume
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher {'status': 'error_detaching'})
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/openstack/common/excutils.py", line 82, in __exit__
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 726, in detach_volume
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher self.driver.detach_volume(context, volume)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/openstack/common/lockutils.py", line 233, in inner
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher retval = f(*args, **kwargs)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_3par_fc.py", line 449, in detach_volume
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher self.common.detach_volume(volume)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_3par_common.py", line 1352, in detach_volume
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher LOG.error(_("Error detaching volume %s") % volume)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/openstack/common/excutils.py", line 82, in __exit__
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_3par_common.py", line 1349, in detach_volume
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher self.clear_volume_key_value_pair(volume, 'HPQ-CS-instance_uuid')
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_3par_common.py", line 1334, in clear_volume_key_value_pair
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher raise exception.VolumeBackendAPIException(data=msg)
2014-10-14 11:44:09.900 TRACE oslo.messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Failure in clear_volume_key_value_pair:Not found (HTT
P 404) 180 - Object key does not exist

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/128767

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/128767
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=9ef47c280b03b971f84b2d12918e55be0f8804a3
Submitter: Jenkins
Branch: master

commit 9ef47c280b03b971f84b2d12918e55be0f8804a3
Author: Anthony Lee <email address hidden>
Date: Wed Oct 15 14:16:32 2014 -0700

    Removing locks from 3PAR FC and iSCSI drivers

    Removed locks from the 3PAR FC and iSCSI drivers. In high load
    environments where many simultaneous volume creations/deletions
    and attaches/detaches are happening errors occur periodically.

    By changing the drivers to create a new connection to the 3PAR
    backend whenever a volume request is made the errors are avoided
    and performance is improved.

    Closes-Bug: 1381190
    Change-Id: Ie588a1d87cf5a22ddf2e890c440582e1fe67f2cb

[Ilja Livenson (ilja-livenson)]

Hello,

is there a plan to backport this fix to icehouse as well?

thanks,
I.