Failed Live Block Migration leaves with Inconsistent Instance Status

[Mate Lakat]

I have a master and slave devstack installation. As I issued a live block migrate to an invalid host, the status of the Instance was changed to "Migrating"

CONFIGURATION:
stack@DevStackOSDomU:~$ less /etc/nova/nova.conf | grep compute_driver
compute_driver=xenapi.XenAPIDriver

STEPS TO REPRODUCE:

1.) You have a server
stack@DevStackOSDomU:~/devstack$ nova list
+--------------------------------------+------+--------+------------------+
| ID | Name | Status | Networks |
+--------------------------------------+------+--------+------------------+
| 083a1d11-22f9-4e32-b476-21412b8bd725 | asd | ACTIVE | private=10.0.0.2 |

2.) And multiple compute resources
stack@DevStackOSDomU:~/devstack$ nova-manage service list
Binary Host Zone Status State
nova-compute DevStackOSDomU nova enabled :-)
...
nova-compute DevStackComputeSlave nova enabled :-)

3.) Server is running on DevStackOSDomU:
+-------------------------------------+------------------...
| Property | Value
+-------------------------------------+------------------...
...
| OS-EXT-SRV-ATTR:host | DevStackOSDomU
...

4.) You live migrate the server to an invalid host:
stack@DevStackOSDomU:~/devstack$ nova live-migration 083a1d11-22f9-4e32-b476-21412b8bd725 SomeHostThatDoesNotExist --block-migrate

5.) You get an error as expected:
...
ComputeHostNotFound: Compute host SomeHostThatDoesNotExist could not be found.

 (HTTP 400) (Request-ID: req-0dbf7ad3-c45b-4924-aa3d-9b62edd95fd0)

6.) But the servers state is changed to "MIGRATING":
stack@DevStackOSDomU:~/devstack$ nova list
+--------------------------------------+------+-----------+------------------+
| ID | Name | Status | Networks |
+--------------------------------------+------+-----------+------------------+
| 083a1d11-22f9-4e32-b476-21412b8bd725 | asd | MIGRATING | private=10.0.0.2 |
+--------------------------------------+------+-----------+------------------+

[Dan Prince (dan-prince)]

Hi Mate,

I'm a little confused on this one. Would you mind adding the detailed steps one might need to follow to reproduce this issue to the ticket?

[Mate Lakat (mate-lakat)]

Dan: Sorry for the dodgy description, I updated the description with the appropriate steps. Please note, that I also have a pending change request for a tempest test, that covers this case at:

https://review.openstack.org/#/c/13101/

[Rohit Karajgi (rohitk)]

I could reproduce this for KVM. Trying to migrate to the same host using CLI fails as expected, but the instance's task state moves to "migrating".
The task state should remain unchanged, or re-set to "None" for this use case.

[Jaroslav Henner (jhenner)]

`--> nova list --host node-01...
+--------------------------------------+----------+--------+---------------------+
| ID | Name | Status | Networks |
+--------------------------------------+----------+--------+---------------------+
| 3a6228e0-b87b-4c79-9521-5b1ad439f628 | testicek | ACTIVE | demonet=192.168.0.2 |
+--------------------------------------+----------+--------+---------------------+

`--> nova live-migration 3a6228e0-b87b-4c79-9521-5b1ad439f628 node-01...
ERROR: Live migration of instance 3a6228e0-b87b-4c79-9521-5b1ad439f628 to host node-01... failed (HTTP 400)

`--> nova list --host node-01...
+--------------------------------------+----------+-----------+---------------------+
| ID | Name | Status | Networks |
+--------------------------------------+----------+-----------+---------------------+
| 3a6228e0-b87b-4c79-9521-5b1ad439f628 | testicek | MIGRATING | demonet=192.168.0.2 |
+--------------------------------------+----------+-----------+---------------------+

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/19616

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/19616
Committed: http://github.com/openstack/nova/commit/be62d6a86971abac57a1cc03c985ba1e97fd55cb
Submitter: Jenkins
Branch: master

commit be62d6a86971abac57a1cc03c985ba1e97fd55cb
Author: Jian Wen <email address hidden>
Date: Mon Jan 14 19:13:24 2013 +0800

    Handle compute node not available for live migration

    This patch handles exception.ComputeServiceUnavailable by restoring
    instance's vm_state and instance's task_state after live migration
    failure caused by unavailable source/dest compute node.

    Raises detailed HTTPBadRequest explanation for this exception.

    Fixes bug 973393 and bug 1051881

    Change-Id: If825b61fad9c4e3030f2e6c5002907255eaf3661

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/22873

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/22873
Committed: http://github.com/openstack/nova/commit/20294279ee1d6d82dbb87c4c29e3a8b9fd0cb8bd
Submitter: Jenkins
Branch: stable/folsom

commit 20294279ee1d6d82dbb87c4c29e3a8b9fd0cb8bd
Author: Jian Wen <email address hidden>
Date: Mon Jan 14 19:13:24 2013 +0800

    Handle compute node not available for live migration

    This patch handles exception.ComputeServiceUnavailable by restoring
    instance's vm_state and instance's task_state after live migration
    failure caused by unavailable source/dest compute node.

    Raises detailed HTTPBadRequest explanation for this exception.

    Fixes bug 973393 and bug 1051881

    Conflicts:
     nova/scheduler/driver.py
     nova/scheduler/manager.py
     nova/tests/api/openstack/compute/contrib/test_admin_actions.py
     nova/tests/scheduler/test_scheduler.py

    Change-Id: If825b61fad9c4e3030f2e6c5002907255eaf3661