live migration (block migration) fails at post_live_migration_at_destination function, but the status of this instance is still "migrating".

[Charlotte Han]

I have two compute node, host opencos114-93 and host opencos179-24.
A migration failed at function of " post_live_migration_at_destination", but the status of this instance is still migrating.
Log is as follows:
http://paste.openstack.org/show/121143/

[Charlotte Han (hanrong)]

A Exception of "NovaException" is thrown and it would be not captured when the function "post_live_migration_at_destination" is in process.
I hope the the Exception of "NovaException" will be captured, and roll back.
We should either put the Instance status into ERROR, if we can't rollback.

[Joe Gordon (jogo)]

can you share the logs on paste.openstack.org so they don't get wrapped and are easier to read. Also what version of nova are you using?

[Joe Gordon (jogo)]

Marking as incomplete since we don't know what version of nova this is

[Charlotte Han (hanrong)]

logs address: http://paste.openstack.org/show/121143/

[Charlotte Han (hanrong)]

the version of nova is icehouse 2014.1.2.

[Joe Gordon (jogo)]

Are you using nova network or neutron? "NovaException: Unexpected vif_type=binding_failed"
 makes me thing this is a networking issue.

[Charlotte Han (hanrong)]

I am using neutron.
if the network is error, I hope the migrating Instance change its status.

[John Garbutt (johngarbutt)]

I am seeing similar issues, when neutron operations fail during post_live_migrate, it leaves things in a very messy state.

I am working on a patch to fix this now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/129881

[OpenStack Infra (hudson-openstack)]

Change abandoned by Joe Gordon (<email address hidden>) on branch: master
Review: https://review.openstack.org/129881
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Charlotte Han (hanrong)]

Hi John Garbutt, since it was over 4 weeks after your last change, I hope you do not mind if I take over this bug and give it try.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/186634

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/186634
Reason: This patch has been stalled for quite a while, so I am going to abandon it to keep the code review queue sane. Please restore the change when it is ready for review.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/235994

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/235994
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=89b1fecce116bc44f558e76cbb5dc43497ea67cc
Submitter: Jenkins
Branch: master

commit 89b1fecce116bc44f558e76cbb5dc43497ea67cc
Author: Pawel Koniszewski <email address hidden>
Date: Mon Feb 1 11:56:59 2016 +0100

    Update instance host in post live migration even when exception occurs

    Currently when, e.g., port binding fails on destination host nova
    loses track of running VM. Operator needs to change record in DB
    manually in order to recover VM in nova and then perform operations
    on destination host to repair such VM. Because VM is on destination
    host already it should be updated regardless of post live migration
    at destination result.

    Change-Id: Ibb5158f453abd9717e6d2ab501295351ca9d0dcf
    Closes-Bug: #1379581

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 13.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/358439

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/358439
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=53fffe02bbabdd71d3cd829d73fba0c19c9b9175
Submitter: Jenkins
Branch: stable/liberty

commit 53fffe02bbabdd71d3cd829d73fba0c19c9b9175
Author: Pawel Koniszewski <email address hidden>
Date: Mon Feb 1 11:56:59 2016 +0100

    Update instance host in post live migration even when exception occurs

    Currently when, e.g., port binding fails on destination host nova
    loses track of running VM. Operator needs to change record in DB
    manually in order to recover VM in nova and then perform operations
    on destination host to repair such VM. Because VM is on destination
    host already it should be updated regardless of post live migration
    at destination result.

    (cherry picked from commit 89b1fecce116bc44f558e76cbb5dc43497ea67cc)

    Change-Id: Ibb5158f453abd9717e6d2ab501295351ca9d0dcf
    Closes-Bug: #1379581
    Conflicts:
     nova/tests/unit/compute/test_compute_mgr.py

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 12.0.5 release.