Cannot restore soft-deleted instance when node is failure

[Eric Xie]

Description
===========
So far, the soft-deleted instance can not be evacuated.
When the node can not be fixed or exceed the reclaim window,
the instance can not be restored for ever.

Steps to reproduce
==================
* Set `reclaim_instance_interval` to 300s, and restart nova-api and nova-compute
* then delete the instance A
* then shutdown compute node for 500s
* Use `openstack server evacuate xxx` and got the instance can not be evacuated.

Expected result
===============
Can be evacuated and restored

Actual result
=============
Do not be allowed to be evacuated

Environment
===========
# git log
commit 22830d78b91946b108defe26b3a8ddefc2247363 (HEAD -> master, origin/master, origin/HEAD)
Merge: fb020b360b 4d8bf15fec
Author: Zuul <email address hidden>
Date: Wed Jun 16 00:53:08 2021 +0000

    Merge "libvirt: Set driver_iommu when attaching virtio devices to SEV instance"

commit fb020b360b13faa53f64222bc81be0d965b47358
Merge: 7f83cbe9e2 0ac74f4e00
Author: Zuul <email address hidden>
Date: Tue Jun 15 21:47:10 2021 +0000

    Merge "Remove references to 'inst_type'

Logs & Configs
==============
nova evacuate a59b2915-5e2e-4541-8bba-235e05ab83dc
ERROR (Conflict): Cannot 'evacuate' instance a59b2915-5e2e-4541-8bba-235e05ab83dc while it is in vm_state soft-delete (HTTP 409) (Request-ID: req-ab2bcfab-4a2e-41d8-

[Balazs Gibizer (balazs-gibizer)]

One way forward is to allow restore to succeed when the compute host is down, and just set the VM back to ACTIVE in the DB. Then restore + evacuate would work together when host is down.

[Dmitriy Rabotyagov (noonedeadpunk)]

Another related usecase would be when it's not possible evacuate instance that is in `powering-off` state.

That could happen when HV went unresponsive and request to shutdown instance has been sent to API. So you must do reset-state in order to evacuate instance (which would raise bug #1932326 as well though)

[Eric Xie (eric-xie)]

@Balazs, what do you think if can evacuate soft-deleted instance directly?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/800357