Instance is of no use during "shelve-offloading" operation, when nova-conductor is stopped

[Swati Bansal]

During shelve-offloading operation if nova-conductor is stopped, the instance stuck forever in "SHELVED" state and task state of instance stuck in "SHELVING_OFFLOADING" state.
After restarting the nova-conductor service, the state of instance does not change to "SHELVE_OFFLOADED" from "SHELVED" and the task state remains in "SHELVING_OFFLOADING" state. In this particular state, the instance is of no use because an instance cannot shelve, unshelve and shelve-offload.

Source code should be modified to reset the state of instance.

[Matt Riedemann (mriedem)]

Everything goes through nova-conductor for database updates so if that's down, you have problems. You can reset the state of the instance using the reset state API (as admin):

https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server.html#server-set

nova server set --state error

Although you can't unshelve from that if the instance is really offloaded and not on a host. If the shelve offload image snapshot was successful, then you basically need to delete the old server and create a new one from the shelved offloaded image snapshot.

[Matt Riedemann (mriedem)]

What it sounds like you're asking for is the ability to rebuild a shelved offloaded instance in error state, which would actually trigger something like an evacuate operation to build the instance on a new host (like unshelve). That's not supported today, and to add that support would be an API change to rebuild and require a spec:

https://docs.openstack.org/nova/latest/contributor/blueprints.html

[Ian Kumlien (pomac)]

This is still happening btw, and there is no good way out.

Risking the VM is not always an option.

[Ian Kumlien (pomac)]

Do, in this particular case, it was stuck in "shelved" - however the disk had been shelved properly, so *boom* another vm bites the dust.

Not having a way out of shelve/shelved_offloaded if openstack deems something odd is broken.