HPE Lefthand driver fails to create volume backup concurrently

[Vijay Ladani]

While running rally test "create_and_restore_volume_backup" concurrently, test failed with below error

2017-01-11 00:36:26.176 _[00;32mDEBUG cinder.volume.drive...2017-01-11 00:36:26.176 DEBUG cinder.volume.drivers.hpe.hpe_lefthand_iscsi [req-7392f27b-ad00-4272-9940-f2b693e039d6 3c6f12c2c200420ab203999a7cf4b9a5 c606c14351a04fccb8210f2a9b8743ba] <== initialize_connection: exception (209ms) VolumeBackendAPIException(u'Error (HTTP 500) SERVER_ALREADY_EXISTS - The server with the name "csimbe13-b14" already exists. Use a unique name and try again.',) from (pid=5717) trace_logging_wrapper /opt/stack/cinder/cinder/utils.py:901
2017-01-11 00:36:26.176 ERROR cinder.volume.manager [req-7392f27b-ad00-4272-9940-f2b693e039d6 3c6f12c2c200420ab203999a7cf4b9a5 c606c14351a04fccb8210f2a9b8743ba] Driver initialize connection failed (error: Error (HTTP 500) SERVER_ALREADY_EXISTS - The server with the name "csimbe13-b14" already exists. Use a unique name and try again.).
2017-01-11 00:36:26.176 TRACE cinder.volume.manager Traceback (most recent call last):
2017-01-11 00:36:26.176 TRACE cinder.volume.manager  File "/opt/stack/cinder/cinder/volume/manager.py", line 1429, in initialize_connection
2017-01-11 00:36:26.176 TRACE cinder.volume.manager  conn_info = self.driver.initialize_connection(volume, connector)
2017-01-11 00:36:26.176 TRACE cinder.volume.manager  File "/opt/stack/cinder/cinder/utils.py", line 895, in trace_logging_wrapper
2017-01-11 00:36:26.176 TRACE cinder.volume.manager  result = f(*args, **kwargs)
2017-01-11 00:36:26.176 TRACE cinder.volume.manager  File "/opt/stack/cinder/cinder/volume/drivers/hpe/hpe_lefthand_iscsi.py", line 746, in initialize_connection
2017-01-11 00:36:26.176 TRACE cinder.volume.manager  raise exception.VolumeBackendAPIException(ex)
2017-01-11 00:36:26.176 TRACE cinder.volume.manager VolumeBackendAPIException: Error (HTTP 500) SERVER_ALREADY_EXISTS - The server with the name "csimbe13-b14" already exists. Use a unique name and try again.
2017-01-11 00:36:26.176 TRACE cinder.volume.manager 
2017-01-11 00:36:26.178 ERROR oslo_messaging.rpc.server [req-7392f27b-ad00-4272-9940-f2b693e039d6 3c6f12c2c200420ab203999a7cf4b9a5 c606c14351a04fccb8210f2a9b8743ba] Exception during message handling
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  res = self.dispatcher.dispatch(message)
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  return self._do_dispatch(endpoint, method, ctxt, args)
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  result = func(ctxt, **new_args)
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  File "/opt/stack/cinder/cinder/volume/manager.py", line 4400, in initialize_connection
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  connector, volume=volume)
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  File "/opt/stack/cinder/cinder/volume/manager.py", line 1437, in initialize_connection
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  raise exception.VolumeBackendAPIException(data=err_msg)
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Driver initialize connection failed (error: Error (HTTP 500) SERVER_ALREADY_EXISTS - The server with the name "csimbe13-b14" already exists. Use a unique name and try again.).
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server 

[Vijay Ladani (vijay-ladani)]

As per the previous bug https://bugs.launchpad.net/cinder/+bug/1658888 filed against HPE 3PAR driver. Cause of failure is, multiple threads are trying to enter in initialize_connection OR terminate_connection method at the same time which leads to an above error.

Fix is to apply lock on those methods.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/430060

[Sean McGinnis (sean-mcginnis)]

Unassigning due to no activity for > 6 months.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.openstack.org/430060
Reason: No response for several months. Feel free to restore and address comments if you wish to pursue.