nova should not reschedule an instance that has already been deleted

[Chris Friesen]

I'm investigating an issue where an instance with a large disk and an attached cinder volume was booted in a stable/kilo OpenStack setup with the diskFilter disabled.

The timeline looks like this:
scheduler picks initial compute node
nova attempts to boot it up on one compute node, it runs out of disk space and gets rescheduled
 scheduler picks another compute node
user requests instance deletion
user requests cinder volume deletion
nova attempts to boot it up on second compute node, it runs out of disk space and gets rescheduled
scheduler picks a third compute node
nova attempts to boot it up on third compute node, runs into problems due to missing cinder volume

The issue I want to address in this bug is whether it makes sense to reschedule the instance when the instance has already been deleted.

Also, instance deletion sets the task_state to 'deleting' early on. In compute.manager.ComputeManager._do_build_and_run_instance(), if we decide to reschedule then nova-compute will set the task_state to 'scheduling' and then save the instance, which I think could overwrite the 'deleting' state in the DB.

So...would it make sense to have nova-compute put an "expected_task_state" on the instance.save() call that sets the 'scheduling' task_state?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/276783

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/288933

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/276783
Reason: This code hasn't been updated in a long time, and is in merge conflict. I am going to abandon this review, but feel free to restore it if you're still working on this.

[Sean Dague (sdague)]

Automatically discovered version kilo in description. If this is incorrect, please update the description to include 'nova version: ...'

[OpenStack Infra (hudson-openstack)]

Change abandoned by Stephen Finucane (<email address hidden>) on branch: master
Review: https://review.opendev.org/288933