Resize a vm that vm_state is "stopped" failure, vm's task_state rollback to "active"

Bug #1551703 reported by Jinquan Ni
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Matt Riedemann
Train
Fix Committed
Low
Matt Riedemann

Bug Description

1. version
kilo 2015.1.0

2. Reproduce steps
2.1 create a instance, then stop it.

[root@SBCJNailSlot3 ~(keystone_admin)]# nova list
+--------------------------------------+--------+---------+--------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+---------+--------------+-------------+-------------------+
| 6fe59445-fb89-47ab-9ead-2476c4522a61 | njq | SHUTOFF | - | Shutdown | test=192.168.1.52 |
+--------------------------------------+--------+---------+--------------+-------------+-------------------+

2.2 resize the instance use a new flavor which it's disk less than current flavor's disk
[root@SBCJNailSlot3 ~(keystone_admin)]# nova resize 6fe59445-fb89-47ab-9ead-2476c4522a61 45

disk value in the current flavor of instance “njq” is 20
disk value in the flavor which id equal 45 is 18.
So this resize action will trigger ResizeError that msg is unable to resize disk down.
Then enter the rollback process

2.3 rollback result:
[root@SBCJNailSlot3 ~(keystone_admin)]# nova list
+--------------------------------------+--------+---------+--------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+---------+--------------+-------------+-------------------+
| 6fe59445-fb89-47ab-9ead-2476c4522a61 | njq | ACTIVE | - | Shutdown | test=192.168.1.52 |
+--------------------------------------+--------+---------+--------------+-------------+-------------------+

Although the finally vm_state of instance will be set to stoped by heal_instance_state.
But the process often takes some time.

IMO, This process is not reasonable, and need fix.

Tags: resize
Jinquan Ni (ni-jinquan)
Changed in nova:
assignee: nobody → jinquanni(ZTE) (ni-jinquan)
tags: added: resize
Jinquan Ni (ni-jinquan)
description: updated
Jinquan Ni (ni-jinquan)
summary: - Resize a vm that task_state is "stoped" failure, vm's task_state
- rollback to "active"
+ Resize a vm that vm_state is "stoped" failure, vm's task_state rollback
+ to "active"
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/287774

Changed in nova:
status: New → In Progress
Changed in nova:
assignee: jinquanni(ZTE) (ni-jinquan) → Alex Xu (xuhj)
Jinquan Ni (ni-jinquan)
Changed in nova:
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/287774
Reason: This has had a -1 for several months now so I'm going to abandon it.

Matt Riedemann (mriedem)
summary: - Resize a vm that vm_state is "stoped" failure, vm's task_state rollback
- to "active"
+ Resize a vm that vm_state is "stopped" failure, vm's task_state
+ rollback to "active"
Changed in nova:
assignee: Alex Xu (xuhj) → nobody
status: In Progress → Triaged
Chris Martin (cm876n)
Changed in nova:
assignee: nobody → Chris Martin (cm876n)
Chris Martin (cm876n)
Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/370861

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/287774
Reason: Moving forward with https://review.openstack.org/#/c/370861/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/370861
Reason: This review is > 4 weeks without comment, and is not mergable in it's current state. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
Matt Riedemann (mriedem) wrote :

I think this is close enough to bug 1811235 that I'm going to duplicate it.

Changed in nova:
assignee: Chris Martin (cm876n) → nobody
status: In Progress → Triaged
Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/691908

Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/691908
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5a20996405c5788855a2457283bbbe7d78140a9c
Submitter: Zuul
Branch: master

commit 5a20996405c5788855a2457283bbbe7d78140a9c
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 29 12:10:08 2019 -0400

    Reset instance to current vm_state if rolling back in resize_instance

    You can resize a stopped instance and if the compute driver raises
    InstanceFaultRollback from migrate_disk_and_power_off, the
    _error_out_instance_on_exception decorator, used in the _resize_instance
    method, will by default reset the instance vm_state to ACTIVE even though
    the guest is stopped. The driver could raise InstanceFaultRollback if you
    try resizing the root disk down on a non-volume-backed instance.

    This builds on [1] and does the same thing as prep_resize [2] for
    making sure the original vm_state is reset on InstanceFaultRollback.

    [1] Ie4f9177f4d54cbc7dbcf58bd107fd5f24c60d8bb
    [2] I17543ecb572934ecc7d0bbc7a4ad2f537fa499bc

    Change-Id: Iff1f9f28a1e4ecf00368cbcac27b7687a5eb0dcf
    Closes-Bug: #1551703

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/train)

Reviewed: https://review.opendev.org/696085
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f3763778dd59c78f68bedd124c653b6744acbc9c
Submitter: Zuul
Branch: stable/train

commit f3763778dd59c78f68bedd124c653b6744acbc9c
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 29 12:10:08 2019 -0400

    Reset instance to current vm_state if rolling back in resize_instance

    You can resize a stopped instance and if the compute driver raises
    InstanceFaultRollback from migrate_disk_and_power_off, the
    _error_out_instance_on_exception decorator, used in the _resize_instance
    method, will by default reset the instance vm_state to ACTIVE even though
    the guest is stopped. The driver could raise InstanceFaultRollback if you
    try resizing the root disk down on a non-volume-backed instance.

    This builds on [1] and does the same thing as prep_resize [2] for
    making sure the original vm_state is reset on InstanceFaultRollback.

    [1] Ie4f9177f4d54cbc7dbcf58bd107fd5f24c60d8bb
    [2] I17543ecb572934ecc7d0bbc7a4ad2f537fa499bc

    Change-Id: Iff1f9f28a1e4ecf00368cbcac27b7687a5eb0dcf
    Closes-Bug: #1551703
    (cherry picked from commit 5a20996405c5788855a2457283bbbe7d78140a9c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 20.1.0

This issue was fixed in the openstack/nova 20.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.