VMAX initialize_conn failure when concurrent terminate req for different volume

[Carl Pecinovsky]

This concurrency issue has the following pieces:

* A storage masking group with a single volume - Vol1
* An initialize connection request is in process for the masking view associated with the storage group above. The request is to add a second volume to the masking - Vol2.
* A terminate_connection request is also in progress for Vol1 and same masking view.
* The initialize_connection request gets to the point of looking up the masking view and storage group to use.
* The terminate_connection then runs and since Vol1 is the last volume in the storage group, the SG is removed and the masking view is removed.
* Now the initialize connection continues processing but blows up when it tries to add Vol2 to the now non-existent storage group.

So the use case here is concurrent cloud-based deployments when a nova host just begins to be utilized, and a new VM is deployed around the same time the only current existing VM is deleted from the host.

It seems like initialize_ and terminate_ need to introduce some level of locking surrounding the masking view/group name such that terminate_ will not remove those entities unless the SG is both empty AND not locked.

[Xing Yang (xing-yang)]

I wonder if this problem still exists with all the bug fixes we submitted lately. Locking could potentially introduce performance problems and other issues. We are actually trying to get rid of locks in Cinder. We will investigate this issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/157683

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/157683
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=8113c8e1b153ad20ebf1692e8a8b903515d974c6
Submitter: Jenkins
Branch: master

commit 8113c8e1b153ad20ebf1692e8a8b903515d974c6
Author: Xing Yang <email address hidden>
Date: Fri Feb 20 01:38:09 2015 -0500

    Fixed a concurrency issue in VMAX driver

    This patch fixed the following problem:

    When trying to add a second volume to the same masking view,
    the first volume got removed at the same time, causing
    the operation on the second volume to fail.

    When two attach requests happen at the same time on the same
    volume, the second one will fail.

    Also fixed a W503 pep8 issue (line break before binary operator)
    in emc_vmax_common.py.

    Closes-Bug: #1416035
    Closes-Bug: #1403160
    Change-Id: I52975b399c2bd8e2a91bdd09004ee277e54c9a89