detach_volume races with delete

[Matthew Booth]

If a client does:

nova volume-detach foo vol
nova delete foo

Assuming the volume-detach takes a moment, which it normally does, the delete will race with it also also attempt to detach the same volume. It's possible there are no side effects from this other than untidy log messages, but this is difficult to prove.

I found this looking through CI logs.

Note that volume-detach can also race with other instance operations, including itself. I'm almost certain that if you poke hard enough you'll find some combination that breaks things badly.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/430213

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/430213
Reason: This review is > 4 weeks without comment, and is not mergable in it's current state. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Sean Dague (sdague)]

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/430213
Reason: Nevermind I just updated https://review.openstack.org/#/c/636701/ to add mdbooth as co-author.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/636701
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7
Submitter: Zuul
Branch: master

commit 2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7
Author: Gary Kotton <email address hidden>
Date: Wed Feb 13 09:50:14 2019 -0800

    Lock detach_volume

    A detach_volume operation may take place during a attach_volume or
    delete_instance opertaion. This may lead to a race condition which
    would result in an instance going into an error state as the volume
    operation is not atomic.

    Commit 4c4dc3a6d added the lock around the attach_volume. This
    follows that same pattern.

    Co-Authored-By: Matthew Booth <email address hidden>

    Change-Id: I7261135a44f0c2eca4732c8360b9c74b729991a8
    Closes-bug: #1662483

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/637534

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/637535

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/637536

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637534
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3310c3cbf534f5d75477ed206a8fb68eb53c6c10
Submitter: Zuul
Branch: stable/rocky

commit 3310c3cbf534f5d75477ed206a8fb68eb53c6c10
Author: Gary Kotton <email address hidden>
Date: Wed Feb 13 09:50:14 2019 -0800

    Lock detach_volume

    A detach_volume operation may take place during a attach_volume or
    delete_instance opertaion. This may lead to a race condition which
    would result in an instance going into an error state as the volume
    operation is not atomic.

    Commit 4c4dc3a6d added the lock around the attach_volume. This
    follows that same pattern.

    Co-Authored-By: Matthew Booth <email address hidden>

    Change-Id: I7261135a44f0c2eca4732c8360b9c74b729991a8
    Closes-bug: #1662483
    (cherry picked from commit 2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637535
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=17b3d5734af9c7b916cfc7007e98f43df975b8e8
Submitter: Zuul
Branch: stable/queens

commit 17b3d5734af9c7b916cfc7007e98f43df975b8e8
Author: Gary Kotton <email address hidden>
Date: Wed Feb 13 09:50:14 2019 -0800

    Lock detach_volume

    A detach_volume operation may take place during a attach_volume or
    delete_instance opertaion. This may lead to a race condition which
    would result in an instance going into an error state as the volume
    operation is not atomic.

    Commit 4c4dc3a6d added the lock around the attach_volume. This
    follows that same pattern.

    Co-Authored-By: Matthew Booth <email address hidden>

    Change-Id: I7261135a44f0c2eca4732c8360b9c74b729991a8
    Closes-bug: #1662483
    (cherry picked from commit 2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637536
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f0b833a352f8859c7b7e299d90440acfbd42e9db
Submitter: Zuul
Branch: stable/pike

commit f0b833a352f8859c7b7e299d90440acfbd42e9db
Author: Gary Kotton <email address hidden>
Date: Wed Feb 13 09:50:14 2019 -0800

    Lock detach_volume

    A detach_volume operation may take place during a attach_volume or
    delete_instance opertaion. This may lead to a race condition which
    would result in an instance going into an error state as the volume
    operation is not atomic.

    Commit 4c4dc3a6d added the lock around the attach_volume. This
    follows that same pattern.

    Co-Authored-By: Matthew Booth <email address hidden>

    Change-Id: I7261135a44f0c2eca4732c8360b9c74b729991a8
    Closes-bug: #1662483
    (cherry picked from commit 2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.10 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.8 release.