HPE 3PAR: In multi host env, multi-detach works partially if volume is attached to instances from separate hosts

[Raghavendra Tilay]

1. IMPACT:
In multi host environment, multi-detach works partially & later volume cannot be deleted.

2. DETAILS & ROOT CAUSE:
Consider below scenario:

   Host-A Host-B
        | |
Instance1 Instance2
        \ /
      Volume
         |
     HPE Array

The volume is attached to instances present on different hosts.
During attach operation, separate vlun entry is created on array for each host.

When detach operation is performed from Host-A, it is checked if this volume has any other attachments.
If yes, then vlun entry of Host-A is not deleted.
This is becuase as per existing logic, it is assumed that volume is attached to multiple instances on same host;
and vlun entries would be deleted on last detach operation only.
Thus vlun entry of Host-A remains as it is on array.

When detach operation is performed from Host-B, it is observed that this volume does NOT have any other attachment.
vlun entry of Host-B is deleted.

Later when someone tries to delete volume, error is displayed:
Since vlun entry of Host-A exists, the volume cannot be deleted.

3. LOGS:
(i) detach from Host-A:
2021-11-23 09:33:18.860 42 INFO cinder.volume.drivers.hpe.hpe_3par_common [req-fc7b0c30-311c-421d-b2ba-9a1560d0a405 e40cffe4d35a4bcc80b9df6a71fe177b 1d8dd76c68034b2d820be0484f660114 - default default] Volume volume-14a58b8c-f301-4f54-a6a4-4a9f4a77f7ea is attached to multiple instances on host gjpctrl.kr.hmgcloud.com, skip terminate volume connection

2021-11-23 09:33:19.206 42 INFO cinder.volume.manager [req-fc7b0c30-311c-421d-b2ba-9a1560d0a405 e40cffe4d35a4bcc80b9df6a71fe177b 1d8dd76c68034b2d820be0484f660114 - default default] Terminate volume connection completed successfully.

(ii) detach from Host-B:
Note: vlun entry is deleted, but it is not captured in INFO logs.

2021-11-23 09:48:15.633 44 INFO cinder.volume.manager [req-99343fb3-4b5e-4806-8fbe-b5499b224c0e e40cffe4d35a4bcc80b9df6a71fe177b fb7706f8ec5b4391a96e0c92c8c20c4e - default default] Terminate volume connection completed successfully.

(iii) During delete volume:
2021-11-23 09:59:42.326 42 ERROR cinder.volume.drivers.hpe.hpe_3par_common [req-d40d22a6-4c04-43e5-977e-54809af275c6 e40cffe4d35a4bcc80b9df6a71fe177b 1d8dd76c68034b2d820be0484f660114 - default default] Exception: Conflict (HTTP 409) 34 - resource in use: HTTPConflict: Conflict (HTTP 409) 34 - resource in use

2021-11-23 09:59:42.350 42 ERROR cinder.volume.manager [req-d40d22a6-4c04-43e5-977e-54809af275c6 e40cffe4d35a4bcc80b9df6a71fe177b 1d8dd76c68034b2d820be0484f660114 - default default] Unable to delete busy volume.: VolumeIsBusy: resource in use

4. SOLUTION:
During detach operation, hostname is checked.
If volume is attached to instances present on same host, then flow is same as earlier i.e vlun entry is deleted only if this is last detach.
If volume is attached to instances present on separate hosts, vlun entry of only that host is deleted.
Thus in multi host env, vlun entries of each host would be deleted separately upon each detach operation; and later volume can be deleted.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/824911

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/824911
Committed: https://opendev.org/openstack/cinder/commit/3ed2f38e54b84073f03ce1148c6dfcbbd37fc032
Submitter: "Zuul (22348)"
Branch: master

commit 3ed2f38e54b84073f03ce1148c6dfcbbd37fc032
Author: raghavendrat <email address hidden>
Date: Mon Jan 17 10:17:47 2022 +0000

    HPE 3PAR: In multi host env, fix multi-detach operation

    In multi host environment, if volume is attached to instances present
    on different hosts, detach operation works partially.
    As a result, at later time the volume cannot be deleted.

    Details are explained in launchpad bug.

    This patch performs following:
    During detach volume from instance, both possibilities are considered:
    the instances can reside:
    [1] either on same host.
    [2] or on different hosts.

    case [1]:
    In such case, behaviour is same as earlier i.e vlun is not deleted
    upon each detach operation i.e skip remainder of terminate volume
    connection. The vluns are deleted only on last detach operation.

    case [2]:
    In such case, vlun of that host on 3par array is deleted separately
    upon each detach operation.

    Closes-Bug: #1958122
    Change-Id: I47e8e86a495802a21570e23ecf7428fccc8b3d60

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/cinder/+/842947

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/842947
Committed: https://opendev.org/openstack/cinder/commit/f6d8353fd13fed3d2aecfc813fd9b66cb1dfdc78
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit f6d8353fd13fed3d2aecfc813fd9b66cb1dfdc78
Author: raghavendrat <email address hidden>
Date: Mon Jan 17 10:17:47 2022 +0000

    HPE 3PAR: In multi host env, fix multi-detach operation

    In multi host environment, if volume is attached to instances present
    on different hosts, detach operation works partially.
    As a result, at later time the volume cannot be deleted.

    Details are explained in launchpad bug.

    This patch performs following:
    During detach volume from instance, both possibilities are considered:
    the instances can reside:
    [1] either on same host.
    [2] or on different hosts.

    case [1]:
    In such case, behaviour is same as earlier i.e vlun is not deleted
    upon each detach operation i.e skip remainder of terminate volume
    connection. The vluns are deleted only on last detach operation.

    case [2]:
    In such case, vlun of that host on 3par array is deleted separately
    upon each detach operation.

    Closes-Bug: #1958122
    Change-Id: I47e8e86a495802a21570e23ecf7428fccc8b3d60
    (cherry picked from commit 3ed2f38e54b84073f03ce1148c6dfcbbd37fc032)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/cinder/+/844793

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 20.0.1 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/844793
Committed: https://opendev.org/openstack/cinder/commit/b25e15828fc062959ce898cab1a4ea02b53654cb
Submitter: "Zuul (22348)"
Branch: stable/xena

commit b25e15828fc062959ce898cab1a4ea02b53654cb
Author: raghavendrat <email address hidden>
Date: Mon Jan 17 10:17:47 2022 +0000

    HPE 3PAR: In multi host env, fix multi-detach operation

    In multi host environment, if volume is attached to instances present
    on different hosts, detach operation works partially.
    As a result, at later time the volume cannot be deleted.

    Details are explained in launchpad bug.

    This patch performs following:
    During detach volume from instance, both possibilities are considered:
    the instances can reside:
    [1] either on same host.
    [2] or on different hosts.

    case [1]:
    In such case, behaviour is same as earlier i.e vlun is not deleted
    upon each detach operation i.e skip remainder of terminate volume
    connection. The vluns are deleted only on last detach operation.

    case [2]:
    In such case, vlun of that host on 3par array is deleted separately
    upon each detach operation.

    Closes-Bug: #1958122
    Change-Id: I47e8e86a495802a21570e23ecf7428fccc8b3d60
    (cherry picked from commit 3ed2f38e54b84073f03ce1148c6dfcbbd37fc032)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 21.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 19.2.0 release.