v1 API: image data downloads for images in pending_delete fails

[Nicolas Simonds]

Use case: user spawns an instance, then deletes the boot image, to prevent new instances from being spawned with it. Later, the user wants to live migrate the instance to a new hypervisor.

As part of this, nova will attempt to download all the images used by the instance on the target node to populate its image cache. Because the image used is no longer in an explicit "active" state, the Glance v1 API returns a 404 Not Found.

According to the docs, pending_delete is supposed to be recoverable, which could arguably imply that downloading and re-uploading the image would be allowed; so this smells like a bug.

I will not let the fact that Mark just told me in as many words that it is not dissuade me from this point of view. ;-)

[Nicolas Simonds (nicolas.simonds)]

https://review.openstack.org/93486/

[Erno Kuvaja (jokke)]

I do not think this is a bug. Recovering pending_delete needs intended user action.

Personally I think the described behaviour from Nova side is a Nova bug and would need different approach for the live migration.

[Nicolas Simonds (nicolas.simonds)]

If the v1 API is frozen and not open to behavior modifications, that is an absolutely acceptable answer.

But I think precision in language is important: "a bug in Glance that Nova must work around" is not the same as "a Nova bug". The practical result is the same (the fix happens in Nova, not Glance) but putting the blame on the API consumer rather than the API itself implies that the consumers have unreasonable expectations on how to interact with the API, and I do not see anywhere in the use case's described chain of events where anyone has unreasonable expectations:

Users deleting images while maintaining running instances that use them is uncontroversial.
Nova attempting to reconstruct a bit-for-bit identical copy of the instance runtime on the target node for live-migrations is sensible.
Nova attempting to use Glance to retrieve the backing files it needs to do the above seems logical.

But as an operator/end-user/consumer, I think asking a service for a resource by name, that I know full well exists (because I can look and see the files on disk), and being told "no, you cannot have these", is not a reasonable behavior.

[Erno Kuvaja (jokke)]

I understand your point there, but I'm looking this other way around. If the image is needed for live migration it should not be deleted at the first place as long as it's still in use. I think there must be better ways to solve this issue than rely on something that still might be there. When the data finally gets deleted the live migration will not work anyways at that point anymore if the original image is needed.

Either Nova needs to have image cache for the images it might need for migrating the VMs around (this being just simple fs cache, Swift store, Glance instance or what ever) where Nova is the one controlling when the image gets thrown away or instead of deleting the image using property on it and filtering those images out of the list so that users cannot spawn new VMs out of them.

Even the image recovery bp https://blueprints.launchpad.net/glance/+spec/image-recover points out that the data still being there and any recovery is targeted for emergency situations for accidental deletes. This in mind I do not think any functionality in Glance or in Glance consumers should treat pending_delete as anything else than deleted.

This is the reasoning why I'm saying that this is rather bug in Nova as it relies on something it has no control over to perform the live migrations. Next logical step would be expecting Glance to keep track which images are still in use and not scrubbing them, right?

[OpenStack Infra (hudson-openstack)]

Change abandoned by Nicolas Simonds (<email address hidden>) on branch: master
Review: https://review.openstack.org/93486
Reason: Nova has worked around the problem for all intents and purposes, so this change is no longer needed.

[Abhishek Kekane (abhishek-kekane)]

This bug is reported for v1 api which is deprecated and will be removed during rocky cycle, so marking this as won't fix.