Guest remain in origin host after evacuate and unset force-down nova-compute

[huanhongda]

Description
===========
If evacuate an instance from a nova-compute forced down host, after unset force-down the instance's guest will remain in origin host.
That's because while unset force-down nova-compute service, nova-compute will not destroy evacuated instances.
And it will do destroy while restart nova-compute service.

Steps to reproduce
==================
1) Boot an instance on node-1.
2) Force down the nova-compute service on node-1.
   nova service-force-down node-1 nova-compute
3) Evacuate the instance to other host.
4) Unset force-down for node-1 nova-compute.
   nova service-force-down --unset node-1 nova-compute
5) Check the guest on node-1.
   virsh list

Expected result
===============
The guest should be deleted on node-1.

Actual result
=============
The guest still remain on node-1.

Environment
===========
This bug was found in Newton. I think it also exist in master as the code of unset force-down service is the same as Newton.

[huanhongda (hongda)]

I think nova-compute should destroy evacuated instances in unset force-down nova-compute process, like restarting nova-compute service.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/587807

[Balazs Gibizer (balazs-gibizer)]

The intention of forced_down flag is to give a way to the deployer to state that she made sure taht the service is dead. So if the deployer sets forced_down but does not make sure that the service is dead the she does not use the API as intended.

[melanie witt (melwitt)]

I don't think this is a valid bug. The evacuate API use case is when the compute host is dead, powered off, or otherwise completely disconnected from the network such that the nova-compute service and instances cannot be accessed. The force-down API is just a way for an operator to force a compute host as "down" without having to wait for the service update interval to reflect "down".

The operator must completely "fence" or "wall off" the compute host before calling the evacuate API, if the compute host is not already dead or powered off. If you evacuate instances without fencing the compute host, you are in danger of corrupting data because duplicate instances will be running potentially connected to the same disk at the same time (if shared storage) and connected to same network port, same volume, etc.

[melanie witt (melwitt)]

If you need to move instances while the compute host is still up, you can set the service to 'disabled' to prevent new instances being scheduled there and then cold or live migrate the existing instances off of the compute host to another host.

I think the only thing to do for this bug report is enhance the documentation about the evacuate API to have more detail about how to use it.

[melanie witt (melwitt)]

Coincidentally, I just found an old, abandoned attempt to update the documentation:

https://review.openstack.org/492533

[OpenStack Infra (hudson-openstack)]

Change abandoned by huanhongda (<email address hidden>) on branch: master
Review: https://review.openstack.org/587807

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/492533
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8835198b8d09e9a69ea83741fdb1579a98019b51
Submitter: Zuul
Branch: master

commit 8835198b8d09e9a69ea83741fdb1579a98019b51
Author: Sean Dague <email address hidden>
Date: Thu Aug 10 09:34:13 2017 -0400

    Update api-guide and api-ref to be clear about forced-down

    Closes-Bug: #1691871
    Related-Bug: #1784826

    Change-Id: Ifc6f1549d88a1b7d9f6e25c962c8a15dd8e180fb