Guest remain in origin host after evacuate and unset force-down nova-compute

Bug #1784826 reported by huanhongda
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned

Bug Description

Description
===========
If evacuate an instance from a nova-compute forced down host, after unset force-down the instance's guest will remain in origin host.
That's because while unset force-down nova-compute service, nova-compute will not destroy evacuated instances.
And it will do destroy while restart nova-compute service.

Steps to reproduce
==================
1) Boot an instance on node-1.
2) Force down the nova-compute service on node-1.
   nova service-force-down node-1 nova-compute
3) Evacuate the instance to other host.
4) Unset force-down for node-1 nova-compute.
   nova service-force-down --unset node-1 nova-compute
5) Check the guest on node-1.
   virsh list

Expected result
===============
The guest should be deleted on node-1.

Actual result
=============
The guest still remain on node-1.

Environment
===========
This bug was found in Newton. I think it also exist in master as the code of unset force-down service is the same as Newton.

Tags: evacuate
Revision history for this message
huanhongda (hongda) wrote :

I think nova-compute should destroy evacuated instances in unset force-down nova-compute process, like restarting nova-compute service.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/587807

Changed in nova:
assignee: nobody → huanhongda (hongda)
status: New → In Progress
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

The intention of forced_down flag is to give a way to the deployer to state that she made sure taht the service is dead. So if the deployer sets forced_down but does not make sure that the service is dead the she does not use the API as intended.

Changed in nova:
assignee: huanhongda (hongda) → Xiaopengli (xiaopengleee)
assignee: Xiaopengli (xiaopengleee) → nobody
Revision history for this message
melanie witt (melwitt) wrote :

I don't think this is a valid bug. The evacuate API use case is when the compute host is dead, powered off, or otherwise completely disconnected from the network such that the nova-compute service and instances cannot be accessed. The force-down API is just a way for an operator to force a compute host as "down" without having to wait for the service update interval to reflect "down".

The operator must completely "fence" or "wall off" the compute host before calling the evacuate API, if the compute host is not already dead or powered off. If you evacuate instances without fencing the compute host, you are in danger of corrupting data because duplicate instances will be running potentially connected to the same disk at the same time (if shared storage) and connected to same network port, same volume, etc.

Changed in nova:
status: In Progress → Invalid
Revision history for this message
melanie witt (melwitt) wrote :

If you need to move instances while the compute host is still up, you can set the service to 'disabled' to prevent new instances being scheduled there and then cold or live migrate the existing instances off of the compute host to another host.

I think the only thing to do for this bug report is enhance the documentation about the evacuate API to have more detail about how to use it.

Revision history for this message
melanie witt (melwitt) wrote :

Coincidentally, I just found an old, abandoned attempt to update the documentation:

https://review.openstack.org/492533

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by huanhongda (<email address hidden>) on branch: master
Review: https://review.openstack.org/587807

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/492533
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8835198b8d09e9a69ea83741fdb1579a98019b51
Submitter: Zuul
Branch: master

commit 8835198b8d09e9a69ea83741fdb1579a98019b51
Author: Sean Dague <email address hidden>
Date: Thu Aug 10 09:34:13 2017 -0400

    Update api-guide and api-ref to be clear about forced-down

    Closes-Bug: #1691871
    Related-Bug: #1784826

    Change-Id: Ifc6f1549d88a1b7d9f6e25c962c8a15dd8e180fb

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.