concurrent instance creation using cinder 3par FC snapshot volumes are failing

[Jay Mehta]

A typical usage involves steps: creating instances from the cinder snapshot (3Par FC) volumes and tear off the instances and snapshots once the work is done.

Simulation can be done in script as:
1. Create the snapshots from the Bootable cinder volume
2. Create the instance using the snapshot volume ((--block-device-mapping vda=$SNAPSHOT_ID:snapshot:40:1)
3. Verify the instance is active and accessible via SSH
4. Delete the instance (which should delete the created cinder bootable volume from the snapshot due to delete-terminate set to True
5. Delete the snapshot volume

When running this script concurrently 10 times (to simulate 10 concurrent requests), some of the instances are going to ERROR mode.

cinder-volume.log contains errors:
---------------------------------

2016-06-23 09:41:47.414 4376 ERROR cinder.volume.manager [req-355de05b-83d3-46a3-9867-b22cf22bc5e1 - - - - -] Driver initialize connection failed (error: Conflict (HTTP 409) 73 - host WWN/iSCSI name already used by another host).
2016-06-23 09:41:47.416 4376 ERROR oslo_messaging.rpc.dispatcher [req-355de05b-83d3-46a3-9867-b22cf22bc5e1 - - - - -] Exception during message handling: Bad or unexpected response from the storage volume backend API: Driver initialize connection failed (error: Conflict (HTTP 409) 73 - host WWN/iSCSI name already used by another host).
2016-06-23 09:41:47.416 4376 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-06-23 09:41:47.416 4376 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/cinder-20160503T082438Z/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-06-23 09:41:47.416 4376 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2016-06-23 09:41:47.416 4376 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/cinder-20160503T082438Z/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-06-23 09:41:47.416 4376 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2016-06-23 09:41:47.416 4376 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/cinder-20160503T082438Z/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-06-23 09:41:47.416 4376 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-06-23 09:41:47.416 4376 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/cinder-20160503T082438Z/lib/python2.7/site-packages/osprofiler/profiler.py", line 117, in wrapper
2016-06-23 09:41:47.416 4376 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-06-23 09:41:47.416 4376 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/cinder-20160503T082438Z/lib/python2.7/site-packages/cinder/volume/manager.py", line 1349, in initialize_connection
2016-06-23 09:41:47.416 4376 ERROR oslo_messaging.rpc.dispatcher raise exception.VolumeBackendAPIException(data=err_msg)
2016-06-23 09:41:47.416 4376 ERROR oslo_messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Driver initialize connection failed (error: Conflict (HTTP 409) 73 - host WWN/iSCSI name already used by another host).

nova-compute.log has errors:
----------------------------
2016-06-22 20:36:37.802 15603 ERROR nova.volume.cinder [req-98b3fd7f-3051-46b1-a132-6e159f7cc9b6 b1892bd8a33847a9a65f3448749c143f 88e8fc151946405cb8054758e938d346] Initialize connection failed for volume a66ff4b9-0698-45ad-b763-c84f4ab67fbe on host hos30-swift-3par-cp1-compute-ephe-0001-mgmt. Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9963c0c5-a7d2-403d-9985-f3f8d2ccaf56) Code: 500. Attempting to terminate connection.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/335695

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/335695
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=3c18a2e41ace1144fd53a1223f612e8a15d0b9ff
Submitter: Jenkins
Branch: master

commit 3c18a2e41ace1144fd53a1223f612e8a15d0b9ff
Author: Jay Mehta <email address hidden>
Date: Wed Jun 29 14:35:37 2016 -0700

    3par driver handles concurrent host create conflict

    One of the step in creating a FC VLUN is to create a host on array.
    When there happens to have simultaneous request for creating same host,
    hpe3parclient will create host for one and return HTTP 409 conflict for others,
    with error code 73, message "host WWN/iSCSI name already used by another host".
    This was not handled by 3PAR driver and the HTTP exception would propagate up to
    volume manager where it would log it. This patch addresses this bug by catching
    this exception and reusing the same host.

    Change-Id: I66e4f80696c782ab869ef0d7848fc6f3b3100c04
    Closes-Bug: #1597454

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/343869

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/343869
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=00e9cf1c11ce793bf3e86ad5012230262a8861bb
Submitter: Jenkins
Branch: stable/mitaka

commit 00e9cf1c11ce793bf3e86ad5012230262a8861bb
Author: Jay Mehta <email address hidden>
Date: Wed Jun 29 14:35:37 2016 -0700

    3par driver handles concurrent host create conflict

    One of the step in creating a FC VLUN is to create a host on array.
    When there happens to have simultaneous request for creating same
    host, hpe3parclient will create host for one and return HTTP 409
    conflict for others, with error code 73, message "host WWN/iSCSI
    name already used by another host". This was not handled by 3PAR
    driver and the HTTP exception would propagate up to volume manager
    where it would log it. This patch addresses this bug by catching
    this exception and reusing the same host.

    Change-Id: I66e4f80696c782ab869ef0d7848fc6f3b3100c04
    Closes-Bug: #1597454
    (cherry picked from commit 3c18a2e41ace1144fd53a1223f612e8a15d0b9ff)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/351925

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/cinder 8.1.0 release.

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/cinder 9.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mike Perez (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/351925