Allowing operations during RESIZE_VERIFY leaks original instance

Bug #965630 reported by Johannes Erdfelt
22
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Dan Prince

Bug Description

Resizing an instance is a two-step process. First the instance is resized and the task_state ends up set to RESIZE_VERIFY. Then a second step is required to confirm or revert the resize, eventually cleaning up one instance and clearing the task_state.

However, a variety of commands are allowed in RESIZE_VERIFY that will end up clearing the task_state without cleaning up one of the instances (the original since the new instance is the one running at that time).

These operations appear to result in this scenario happening:

stop
reboot
rebuild
pause
suspend
rescue
changePassword
createImage

(I've personally seen it and tested it with rescue and changePassword, the others I found from reading the code, there may be more too).

I'm not sure if the API intended for any of these operations to occur when in RESIZE_VERIFY. If not, the simplest fix would be to prevent any of these from executing while in RESIZE_VERIFY.

If it is intended, some redesigning of how states are tracked is necessary to avoid task_state being cleared.

summary: - Rescue from RESIZE_VERIFY leaks original instance
+ Allowing operations during RESIZE_VERIFY leaks original instance
description: updated
Revision history for this message
Rick Harris (rconradharris) wrote :

I'd vote that we prevent any of these actions until the migration is either verified or reverted.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/6391

Changed in nova:
assignee: nobody → Rick Harris (rconradharris)
status: New → In Progress
Dan Prince (dan-prince)
Changed in nova:
assignee: Rick Harris (rconradharris) → Dan Prince (dan-prince)
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/7976

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/7976
Committed: http://github.com/openstack/nova/commit/cc87be923ff83fed349c0da4b285a180d0644db8
Submitter: Jenkins
Branch: master

commit cc87be923ff83fed349c0da4b285a180d0644db8
Author: Dan Prince <email address hidden>
Date: Thu May 31 11:40:27 2012 -0400

    Prevent task_state changes during VERIFY_RESIZE.

    Don't allow operations that cause task_state to change
    when an instance is in VERIFY_RESIZE.

    Fixes LP Bug #965630.

    Change-Id: I7e222985c199d619c5cb248c04e1670c92d5587b

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → folsom-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-2 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.