Add expected_task for instance.save() used by block_device mapping in compute.manager._build_resources

Bug #1512563 reported by Zhenyu Zheng
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
Low
Unassigned

Bug Description

For large scale deployment, command can be sent concurrently, and for several times, when an instance was asked to be delete during it is launching, we observed that the vm_state and task_state of that instance has changed abnormally like this:

When we delete the instance while its' task state is networking:
scheduling->none->(networking)->deleting->block_device_mapping-->spawing-->none
The expected task_state should be:
networking->deleting->deleted
and the vm_state changes like this:
BUILD-ACIVE-disappear , which is also very strange for user.

After we dive deeper, we found out that in the _build_resource code, the instance.save() for block_device_mapping doesn't contain expected_task_state:
https://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n2135 , so it also saved "deleting" and two process keep working which causes the above situation.

How about we add some expected_task_state also for block_device_mapping?
The expected task states can be NETWORKING, SCHEDULING, and none.

Changed in nova:
assignee: nobody → Zhenyu Zheng (zhengzhenyu)
Changed in nova:
status: New → In Progress
Revision history for this message
Jay Pipes (jaypipes) wrote :

Is there a patch in progress for this? I don't see one linked...

Changed in nova:
importance: Undecided → Medium
Revision history for this message
Zhenyu Zheng (zhengzhenyu) wrote :

Didn't notice this, the patch for this bug is:

https://review.openstack.org/#/c/245038/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/288933

Changed in nova:
assignee: Zhenyu Zheng (zhengzhenyu) → Hans Lindgren (hanlind)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/245038
Reason: This code hasn't been updated in a long time, and is in merge conflict. I am going to abandon this review, but feel free to restore it if you're still working on this.

Changed in nova:
assignee: Hans Lindgren (hanlind) → Stephen Finucane (stephenfinucane)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Stephen Finucane (<email address hidden>) on branch: master
Review: https://review.opendev.org/288933

Changed in nova:
status: In Progress → New
assignee: Stephen Finucane (stephenfinucane) → nobody
Revision history for this message
Artom Lifshitz (notartom) wrote :

Hard to reproduce, as it's essentially a race, but feels valid. I'm bumping this down to Low though, as the problem is mostly cosmetic, and doesn't actually prevent an instance from deleting correctly.

Changed in nova:
status: New → Confirmed
importance: Medium → Low
tags: added: compute race-condition
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.