Nova should allow HARD_REBOOT to instances in the state REBOOTING_HARD

[Matthew Gilliard]

Currently when trying to issue a hard reboot to an instance, the logic in nova/compute/api.py says:

    if (reboot_type == 'HARD' and instance['task_state'] == task_states.REBOOTING_HARD)):
        raise exception.InstanceInvalidState

This means there's no user-facing way to rescue an instance that is stuck in REBOOTING_HARD except for DELETE.

We should allow hard reboot to happen in the state REBOOTING_HARD. Some new locking code will be required.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/84642

[Thang Pham (thang-pham)]

Can't you issue a "nova reset-state [--active] <instance>" to bring the instance to a state where you can then issue a hard reboot, instead of deleting this section of code?

[Jay Pipes (jaypipes)]

Thang, I agree that is a way that an *admin* user could effectively reset the state of the instance.

Unfortunately, there's no way for a normal user to do such a thing, AFAIK.

That said, I don't think Saranya's patch is really the appropriate fix here. I'd rather see a fix that identifies causes for an instance getting stuck in the REBOOTING_HARD state and attempts to rectify the stuck instance (either in a retry with timeout loop, or calling into the hypervisor itself to see what's up with the instance).

[Saranya Pandian (saranya-pandian)]

My thought is that the virtual instance should appear to the user in the same way as a physical machine. One can reset ('hard reboot') a physical machine multiple times even if a reset is already in progress. Hence the fix.

Well, shouldn't we let hypervisor handle such scenarios for all the booting cases? All hypervisors handle these cases appropriately.

[Jay Pipes (jaypipes)]

Saranya, physical machines cannot be hard-rebooted while being hard-rebooted. It's physically impossible.

I'm actually not even sure this bug is valid. Or rather, it *might* be a valid bug if the instance is somehow getting stuck in REBOOTING_HARD task_state and is unable to be saved other than an admin manually resetting the state. But that is not actually the description of the bug.