Live migration failure in API doesn't revert task_state to None

[Loganathan Parthipan]

If API times out on a RPC during the processing of a migrate_server it does not revert the task_state back to NULL before or after sending the error response back to the user. This can prevent further API operations on the VM and leave a good VMs in non-operable state with the exception of perhaps a delete.

This is one possible reproducer. I'm not sure if this is always true, and I'd appreciate if someone else confirm it.

1. Somehow make RPC requests hang
2. Issue a live migration request
3. The call should return an HTTP error (409 perhaps)
4. Check VM. It should be in a good state but the task_state stuck in 'migrating'

[John Garbutt (johngarbutt)]

We should either put the VM into ERROR, if we can't rollback.

Or we should rollback and reset to ACTIVE.

I have recently made sure we now record instance faults, so there is a tiny bit more fault tracking.

[John Garbutt (johngarbutt)]

I have marked this as medium, because this kind of error stops a feature feeling "solid".

[John Garbutt (johngarbutt)]

Certain failures to get handled correctly, see here:
https://github.com/openstack/nova/blob/c85a17447e54ebba192c0dcab1222760319cbe46/nova/scheduler/manager.py#L109

Need to do something similar for when there are RPC timeout issues in other places in the code.

[Loganathan Parthipan (parthipan)]

I can look at trapping MessageTimeout. I'm not sure what to do when the timeout is from conductor itself. Trying to set the task_state in this case may not succeed.

[John Garbutt (johngarbutt)]

Hmm, I think this is the bit that needs attention (generally) is in the conductor manager, look at the second except block, it should be more like what we have in the scheduler manager (given above).

[Bartosz Fic (bartosz-fic)]

I've noticed that hanging other files in conductor leads to the same behaviour.

For example:
- /nova/conductor/rpcapi.py (server_migrate method)
- /nova/conductor/manager.py (_live_migrate method)
- /nova/conductor/tasks/live_migrate.py (execute method)

As a result of live migration we get instance with attributes:
a) vm_state: active
b) task_state: migrating

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/168916

[Matt Riedemann (mriedem)]

This bug was reported over 18 months ago, so not a regression in liberty, so marking as liberty-backport-potential so it doesn't block rc1.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/168916
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f2a1f00829e849e78f850a73489864e57cbd86b3
Submitter: Jenkins
Branch: master

commit f2a1f00829e849e78f850a73489864e57cbd86b3
Author: Maciej Szankin <email address hidden>
Date: Fri Feb 26 11:18:51 2016 +0100

    Live migration failure in API leaves VM in MIGRATING state

    When nova-api calls nova-conductor a RPC MessagingTimeout might
    occur. In such case we shouldn't leave VM in MIGRATING state. Possible
    scenarios are:

    * nova-conductor received message but failed to respond, no additional
    exceptions raised - live migration will start, VM will be moved to
    destination host
    * nova-conductor received message but failed to respond, additional
    exception raised (e.g., LibvirtError) - LM will not start
    * nova-api couldn't reach nova-conductor - LM will not start

    Because we can't predict in API layer what happened below, this patch
    writes instance fault to database when MessagingTimeout is caught.

    Co-Authored-By: Pawel Koniszewski <email address hidden>
                    Bartosz Fic <email address hidden>
    Closes-Bug: #1276214
    Change-Id: Id800e925fbb689d20e7907b698b67c92fd3da979

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/304746

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/304746
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0c4fc7812b349ccdd423168f568fe694325a8e74
Submitter: Jenkins
Branch: stable/mitaka

commit 0c4fc7812b349ccdd423168f568fe694325a8e74
Author: Maciej Szankin <email address hidden>
Date: Fri Feb 26 11:18:51 2016 +0100

    Live migration failure in API leaves VM in MIGRATING state

    When nova-api calls nova-conductor a RPC MessagingTimeout might
    occur. In such case we shouldn't leave VM in MIGRATING state. Possible
    scenarios are:

    * nova-conductor received message but failed to respond, no additional
    exceptions raised - live migration will start, VM will be moved to
    destination host
    * nova-conductor received message but failed to respond, additional
    exception raised (e.g., LibvirtError) - LM will not start
    * nova-api couldn't reach nova-conductor - LM will not start

    Because we can't predict in API layer what happened below, this patch
    writes instance fault to database when MessagingTimeout is caught.

    Co-Authored-By: Pawel Koniszewski <email address hidden>
                    Bartosz Fic <email address hidden>
    Closes-Bug: #1276214
    Change-Id: Id800e925fbb689d20e7907b698b67c92fd3da979
    (cherry picked from commit f2a1f00829e849e78f850a73489864e57cbd86b3)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 13.1.1 release.