Rebuild instance is stuck in rebuilding state when hosting Compute is powered off

[Hu Zhou]

Description
===========
When the instance X's hosting Compute H is down, ry to rebuild instance is stuck is "REBUILD" status forever.

Steps to reproduce
==================
* Create instance X on Compute H
* Power off Compute H
* Rebuild instance X (with same image)

$ openstack server rebuild X

Expected result
===============
The instance X should fails to rebuild on the same host with proper error message.

Actual result
=============
The instance is in "REBUILD" state forever.

Environment
===========
1. Queens release used.

   $ rpm -qa | grep nova-compute
     openstack-nova-compute-17.0.5-1

2. KVM hypervisor
   qemu-kvm-common-ev-2.10.0-21
   libvirt-daemon-kvm-3.9.0-14.1

2. LVM used as storage backend on Compute host.
   lvm2-2.02.177-4

3. Which networking type did you use?
   Neutron with OpenVSwitch

[Balazs Gibizer (balazs-gibizer)]

I can reproduce the reported behavior on recent master in a devstack. I also waited after the compute service stop until the nova controller detected that the compute is down. Still rebuild is accepted and the the instance is stuck in REBUILD state even after I started the compute service back up.

A possible workaround is to use nova reset-state --active to push the state of the server back to ACTIVE state.

When the compute is started up it logged an ERROR for the stuck instance so this is definitely bug.[1]

[1] http://paste.openstack.org/show/793758/

[Akhil Gudise (akhil-g)]

I was also able to reproduce this bug in the latest version. I would like to add my views on this bug since the nova controller can detect that the compute node is down maybe we can add a method to check the status of the compute nodes, that get invoked for all the requests made related to compute services.

[Balazs Gibizer (balazs-gibizer)]

As we discussed on the nova weekly meeting I'm OK with the idea to introduce a service up check in the rebuild code path.

[Balazs Gibizer (balazs-gibizer)]

@Akhil: I saw your ping on IRC. I the rebuild code paths is the following

* https://github.com/openstack/nova/blob/8ca9b7254b50351c0e4b3cf79e76851c20823ecd/nova/api/openstack/compute/servers.py#L1113 (running in the nova-api service)
* https://github.com/openstack/nova/blob/8ca9b7254b50351c0e4b3cf79e76851c20823ecd/nova/compute/api.py#L3596 (still in the nova-api)
* https://github.com/openstack/nova/blob/8ca9b7254b50351c0e4b3cf79e76851c20823ecd/nova/conductor/manager.py#L1223 (running in the nova-conductor service)
* https://github.com/openstack/nova/blob/8ca9b7254b50351c0e4b3cf79e76851c20823ecd/nova/compute/manager.py#L3282 (would run in the nova-compute service)

I think we should introduce the service up check at the start of https://github.com/openstack/nova/blob/8ca9b7254b50351c0e4b3cf79e76851c20823ecd/nova/compute/api.py#L3401

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/749531