Build requests can be orphaned without instance mappings
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
melanie witt | ||
Queens |
Fix Committed
|
Medium
|
melanie witt | ||
Rocky |
Fix Committed
|
Medium
|
melanie witt | ||
Stein |
Fix Committed
|
Medium
|
melanie witt |
Bug Description
Mohammed reported this in the nova channel today [1] and the RDO cloud people have run into the same issue too. The deployment got into a situation where instances would show up in a 'nova list' in BUILD/scheduling state but were unable to be deleted. (They show up in 'nova list' because 'nova list' lists build requests and all instances in all cells).
Inspection of the database showed that the "instance" had a build request but *no* instance mapping and *no* instance record in any cell. And the instance could not be deleted even though it appeared in the 'nova list' because the delete API first does a compute API().get in order to get the instance object to pass down to the compute API().delete method. The compute API().get fails with InstanceNotFound because the _get_instance method raises InstanceNotFound if there is no instance mapping for the instance.
Mohammed was able to share this trace [2] which shows the instance_
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
2018-07-25 04:20:12.946 7926 ERROR nova.api.
and the API request returned with a 500:
"HTTP exception thrown: Unexpected API Error. Please report this at http://
Mohammed is going to try a fix to do the build request and instance mapping creates in a single database transaction, so that the build request cannot be orphaned.
Another way to handle it would be to leave the creates as-is and make the API handle deletion of orphaned build requests, but doing that would allow another avenue for instances in ERROR state, whereas doing the build request and instance mapping creates in a single transaction would avoid that.
[1] http://
[2] http://
Changed in nova: | |
assignee: | Mohammed Naser (mnaser) → melanie witt (melwitt) |
Fix proposed to branch: master /review. openstack. org/586742
Review: https:/