OpenStack Compute (nova)

HyperV: shelve vm deadlock

Bug #1611321 reported by Lucian Petrut
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Lucian Petrut
Mitaka
Fix Released
Medium
Matt Riedemann
compute-hyperv
Fix Released
Undecided
Unassigned

Bug Description

At the moment, the instance snapshot operation is synchronized using
the instance uuid. This was added some time ago, as the instance
destroy operation was failing when an instance snapshot was in
proggress.

This is now causing a deadlock, as a similar lock was recently
introduced in the manager for the shelve operation by this change:
Id36b3b9516d72d28519c18c38d98b646b47d288d

We can safely remove the lock from the HyperV driver as we now stop
pending jobs when destroying instances.

Tags: hyper-v
Revision history for this message
Maciej Szankin (mszankin) wrote :

Can you provide logs / steps to reproduce?

Changed in nova:
status: New → Incomplete
Revision history for this message
Lucian Petrut (petrutlucian94) wrote :

This is really easy to reproduce, all you have to do is shelve an instance and it will always fail.

As for the logs, the CI failed on the above mentioned change, the one that introduced this issue:
http://64.119.130.115/nova/352554/1/results.html.gz
https://review.openstack.org/#/c/352554/

Revision history for this message
Lucian Petrut (petrutlucian94) wrote :

This bug is addressed by: https://review.openstack.org/#/c/352837/

Matt Riedemann (mriedem)
Changed in nova:
status: Incomplete → In Progress
importance: Undecided → High
assignee: nobody → Lucian Petrut (petrutlucian94)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/352837
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c7af24ca8279226adc5cd8fa0984c6fd79e26d67
Submitter: Jenkins
Branch: master

commit c7af24ca8279226adc5cd8fa0984c6fd79e26d67
Author: Lucian Petrut <email address hidden>
Date: Tue Aug 9 13:21:48 2016 +0300

    HyperV: remove instance snapshot lock

    At the moment, the instance snapshot operation is synchronized using
    the instance uuid. This was added some time ago, as the instance
    destroy operation was failing when an instance snapshot was in
    proggress.

    This is now causing a deadlock, as a similar lock was recently
    introduced in the manager for the shelve operation by this change:
    Id36b3b9516d72d28519c18c38d98b646b47d288d

    We can safely remove the lock from the HyperV driver as we now stop
    pending jobs when destroying instances.

    Closes-Bug: #1611321

    Change-Id: I1c2ca0d24c195ebaba442bbb7091dcecc0a7e781

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to compute-hyperv (master)

Reviewed: https://review.openstack.org/352841
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=e4208017feb92de8bfbe1c3b28d6b6e3cea7ab2c
Submitter: Jenkins
Branch: master

commit e4208017feb92de8bfbe1c3b28d6b6e3cea7ab2c
Author: Lucian Petrut <email address hidden>
Date: Tue Aug 9 13:37:01 2016 +0300

    HyperV: remove instance snapshot lock

    At the moment, the instance snapshot operation is synchronized using
    the instance uuid. This was added some time ago, as the instance
    destroy operation was failing when an instance snapshot was in
    proggress.

    This is now causing a deadlock, as a similar lock was recently
    introduced in the manager for the shelve operation by this change:
    Id36b3b9516d72d28519c18c38d98b646b47d288d

    We can safely remove the lock from the HyperV driver as we now stop
    pending jobs when destroying instances.

    Closes-Bug: #1611321

    Change-Id: I1c2ca0d24c195ebaba442bbb7091dcecc0a7e781

Changed in compute-hyperv:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.0.0b3

This issue was fixed in the openstack/nova 14.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/mitaka)

Reviewed: https://review.openstack.org/357566
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8f20e7067fbd2c784cd65876f58dd3cde59f966c
Submitter: Jenkins
Branch: stable/mitaka

commit 8f20e7067fbd2c784cd65876f58dd3cde59f966c
Author: Lucian Petrut <email address hidden>
Date: Tue Aug 9 13:21:48 2016 +0300

    HyperV: remove instance snapshot lock

    At the moment, the instance snapshot operation is synchronized using
    the instance uuid. This was added some time ago, as the instance
    destroy operation was failing when an instance snapshot was in
    proggress.

    This is now causing a deadlock, as a similar lock was recently
    introduced in the manager for the shelve operation by this change:
    Id36b3b9516d72d28519c18c38d98b646b47d288d

    We can safely remove the lock from the HyperV driver as we now stop
    pending jobs when destroying instances.

    Closes-Bug: #1611321

    Change-Id: I1c2ca0d24c195ebaba442bbb7091dcecc0a7e781
    (cherry picked from commit c7af24ca8279226adc5cd8fa0984c6fd79e26d67)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to compute-hyperv (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/375601

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to compute-hyperv (stable/mitaka)

Reviewed: https://review.openstack.org/375601
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=590e678fcf4fc8eaabf2af51fa3b76a137adbd61
Submitter: Jenkins
Branch: stable/mitaka

commit 590e678fcf4fc8eaabf2af51fa3b76a137adbd61
Author: Lucian Petrut <email address hidden>
Date: Tue Aug 9 13:37:01 2016 +0300

    HyperV: remove instance snapshot lock

    At the moment, the instance snapshot operation is synchronized using
    the instance uuid. This was added some time ago, as the instance
    destroy operation was failing when an instance snapshot was in
    proggress.

    This is now causing a deadlock, as a similar lock was recently
    introduced in the manager for the shelve operation by this change:
    Id36b3b9516d72d28519c18c38d98b646b47d288d

    We can safely remove the lock from the HyperV driver as we now stop
    pending jobs when destroying instances.

    Closes-Bug: #1611321

    Change-Id: I1c2ca0d24c195ebaba442bbb7091dcecc0a7e781
    (cherry picked from commit e4208017feb92de8bfbe1c3b28d6b6e3cea7ab2c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 13.1.2

This issue was fixed in the openstack/nova 13.1.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.