Pure driver sometimes fails to terminate connection to host

[Patrick East]

Sometimes we can get into a state where a Purity host is deleted before we try and disconnect a volume from it. The result is an error message like:

2015-07-08 04:34:49.327 ERROR cinder.volume.driver [req-41b483e7-08bf-4941-9f03-5342385c22aa f46924c112a14c80ab0a24a613d95eef b7455b8974bb4064ad247c8f375eae6c] Unable to terminate volume connection: PureHTTPError status code 400 returned by REST version 1.4 at cinder-fa1.dev.purestorage.com: BAD REQUEST
[{"msg": "Host does not exist.", "ctx": "image-cache-tester-37f7a2041ef143b28f62306810f4b580-cinder"}]
2015-07-08 04:34:49.328 ERROR cinder.volume.flows.manager.create_volume [req-41b483e7-08bf-4941-9f03-5342385c22aa f46924c112a14c80ab0a24a613d95eef b7455b8974bb4064ad247c8f375eae6c] Failed to copy image 58524ab5-31b8-430b-a0d0-dbf5b673941c to volume: e6d994fc-c68d-4a40-bdd9-d501fbffec38
2015-07-08 04:34:49.328 TRACE cinder.volume.flows.manager.create_volume Traceback (most recent call last):
2015-07-08 04:34:49.328 TRACE cinder.volume.flows.manager.create_volume File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 517, in _copy_image_to_volume
2015-07-08 04:34:49.328 TRACE cinder.volume.flows.manager.create_volume copy_image_to_volume(context, volume_ref, image_service, image_id)
2015-07-08 04:34:49.328 TRACE cinder.volume.flows.manager.create_volume File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-07-08 04:34:49.328 TRACE cinder.volume.flows.manager.create_volume return f(*args, **kwargs)
2015-07-08 04:34:49.328 TRACE cinder.volume.flows.manager.create_volume File "/opt/stack/cinder/cinder/volume/driver.py", line 589, in copy_image_to_volume
2015-07-08 04:34:49.328 TRACE cinder.volume.flows.manager.create_volume self._detach_volume(context, attach_info, volume, properties)
2015-07-08 04:34:49.328 TRACE cinder.volume.flows.manager.create_volume File "/opt/stack/cinder/cinder/volume/driver.py", line 381, in _detach_volume
2015-07-08 04:34:49.328 TRACE cinder.volume.flows.manager.create_volume raise exception.VolumeBackendAPIException(data=err_msg)
2015-07-08 04:34:49.328 TRACE cinder.volume.flows.manager.create_volume VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to terminate volume connection: PureHTTPError status code 400 returned by REST version 1.4 at cinder-fa1.dev.purestorage.com: BAD REQUEST
2015-07-08 04:34:49.328 TRACE cinder.volume.flows.manager.create_volume [{"msg": "Host does not exist.", "ctx": "image-cache-tester-37f7a2041ef143b28f62306810f4b580-cinder"}]
2015-07-08 04:34:49.328 TRACE cinder.volume.flows.manager.create_volume

If this happens while creating a volume from an image (ie when detaching from the cinder node) it will cause the volume to go into an error state... even though it is actually fine.

There should be some protection for this case in terminate_connection for this type of scenario, as far as we should be concerned if the host doesn't exist anymore the volume *is* already disconnected from it, and there is nothing left to do.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/201334

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/201393

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/201393
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c18e1c7cad6f0209a322a0db1ec653bc7940a0d8
Submitter: Jenkins
Branch: master

commit c18e1c7cad6f0209a322a0db1ec653bc7940a0d8
Author: Patrick East <email address hidden>
Date: Mon Jul 13 15:49:40 2015 -0700

    Prevent missing Purity hosts from raising errors

    Occasionally a Purity host can be deleted out from underneath Cinder.
    When this happens if cinder is attempting to disconnect a volume from
    that host we should treat it as a ‘success’ since if the host is gone,
    it is disconnected already (as far as Purity is concerned).

    To prevent one cause of this, the disconnect method will now be locked
    with the connect ones so that can’t delete a host we intend to re-use.

    Change-Id: Iba82c93f9e101709cf5532225068474688187b29
    Closes-bug: #1472710

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/228219

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/228219
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=01056185cacd9bee8ee077ba3c8782e64decbe5a
Submitter: Jenkins
Branch: stable/kilo

commit 01056185cacd9bee8ee077ba3c8782e64decbe5a
Author: Patrick East <email address hidden>
Date: Mon Jul 13 15:49:40 2015 -0700

    Prevent missing Purity hosts from raising errors

    Occasionally a Purity host can be deleted out from underneath Cinder.
    When this happens if cinder is attempting to disconnect a volume from
    that host we should treat it as a ‘success’ since if the host is gone,
    it is disconnected already (as far as Purity is concerned).

    To prevent one cause of this, the disconnect method will now be locked
    with the connect ones so that can’t delete a host we intend to re-use.

    Change-Id: Iba82c93f9e101709cf5532225068474688187b29
    Closes-bug: #1472710
    (cherry picked from commit c18e1c7cad6f0209a322a0db1ec653bc7940a0d8)