detach_volume races with delete

Bug #1662483 reported by Matthew Booth on 2017-02-07
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Gary Kotton
Ocata
Undecided
Unassigned
Pike
Undecided
Chris Dent
Queens
Undecided
Chris Dent
Rocky
Undecided
Chris Dent

Bug Description

If a client does:

nova volume-detach foo vol
nova delete foo

Assuming the volume-detach takes a moment, which it normally does, the delete will race with it also also attempt to detach the same volume. It's possible there are no side effects from this other than untidy log messages, but this is difficult to prove.

I found this looking through CI logs.

Note that volume-detach can also race with other instance operations, including itself. I'm almost certain that if you poke hard enough you'll find some combination that breaks things badly.

Fix proposed to branch: master
Review: https://review.openstack.org/430213

Changed in nova:
assignee: nobody → Matthew Booth (mbooth-9)
status: New → In Progress
Matt Riedemann (mriedem) on 2017-04-19
tags: added: volumes

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/430213
Reason: This review is > 4 weeks without comment, and is not mergable in it's current state. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Sean Dague (sdague) wrote :

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

Changed in nova:
status: In Progress → New
assignee: Matthew Booth (mbooth-9) → nobody

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/430213
Reason: Nevermind I just updated https://review.openstack.org/#/c/636701/ to add mdbooth as co-author.

Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
status: New → In Progress
Matt Riedemann (mriedem) on 2019-02-13
Changed in nova:
assignee: Matt Riedemann (mriedem) → Gary Kotton (garyk)
importance: Undecided → Medium
Changed in nova:
assignee: Gary Kotton (garyk) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem) on 2019-02-13
Changed in nova:
assignee: Matt Riedemann (mriedem) → Gary Kotton (garyk)
Changed in nova:
assignee: Gary Kotton (garyk) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem) on 2019-02-13
Changed in nova:
assignee: Matt Riedemann (mriedem) → Gary Kotton (garyk)

Reviewed: https://review.openstack.org/636701
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7
Submitter: Zuul
Branch: master

commit 2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7
Author: Gary Kotton <email address hidden>
Date: Wed Feb 13 09:50:14 2019 -0800

    Lock detach_volume

    A detach_volume operation may take place during a attach_volume or
    delete_instance opertaion. This may lead to a race condition which
    would result in an instance going into an error state as the volume
    operation is not atomic.

    Commit 4c4dc3a6d added the lock around the attach_volume. This
    follows that same pattern.

    Co-Authored-By: Matthew Booth <email address hidden>

    Change-Id: I7261135a44f0c2eca4732c8360b9c74b729991a8
    Closes-bug: #1662483

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/637534
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3310c3cbf534f5d75477ed206a8fb68eb53c6c10
Submitter: Zuul
Branch: stable/rocky

commit 3310c3cbf534f5d75477ed206a8fb68eb53c6c10
Author: Gary Kotton <email address hidden>
Date: Wed Feb 13 09:50:14 2019 -0800

    Lock detach_volume

    A detach_volume operation may take place during a attach_volume or
    delete_instance opertaion. This may lead to a race condition which
    would result in an instance going into an error state as the volume
    operation is not atomic.

    Commit 4c4dc3a6d added the lock around the attach_volume. This
    follows that same pattern.

    Co-Authored-By: Matthew Booth <email address hidden>

    Change-Id: I7261135a44f0c2eca4732c8360b9c74b729991a8
    Closes-bug: #1662483
    (cherry picked from commit 2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7)

Reviewed: https://review.openstack.org/637535
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=17b3d5734af9c7b916cfc7007e98f43df975b8e8
Submitter: Zuul
Branch: stable/queens

commit 17b3d5734af9c7b916cfc7007e98f43df975b8e8
Author: Gary Kotton <email address hidden>
Date: Wed Feb 13 09:50:14 2019 -0800

    Lock detach_volume

    A detach_volume operation may take place during a attach_volume or
    delete_instance opertaion. This may lead to a race condition which
    would result in an instance going into an error state as the volume
    operation is not atomic.

    Commit 4c4dc3a6d added the lock around the attach_volume. This
    follows that same pattern.

    Co-Authored-By: Matthew Booth <email address hidden>

    Change-Id: I7261135a44f0c2eca4732c8360b9c74b729991a8
    Closes-bug: #1662483
    (cherry picked from commit 2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7)

Reviewed: https://review.openstack.org/637536
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f0b833a352f8859c7b7e299d90440acfbd42e9db
Submitter: Zuul
Branch: stable/pike

commit f0b833a352f8859c7b7e299d90440acfbd42e9db
Author: Gary Kotton <email address hidden>
Date: Wed Feb 13 09:50:14 2019 -0800

    Lock detach_volume

    A detach_volume operation may take place during a attach_volume or
    delete_instance opertaion. This may lead to a race condition which
    would result in an instance going into an error state as the volume
    operation is not atomic.

    Commit 4c4dc3a6d added the lock around the attach_volume. This
    follows that same pattern.

    Co-Authored-By: Matthew Booth <email address hidden>

    Change-Id: I7261135a44f0c2eca4732c8360b9c74b729991a8
    Closes-bug: #1662483
    (cherry picked from commit 2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7)

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

This issue was fixed in the openstack/nova 17.0.10 release.

This issue was fixed in the openstack/nova 18.2.0 release.

This issue was fixed in the openstack/nova 16.1.8 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers