live-migration Interrupt when get network info failed

[John Smith]

Description
===========
When an exception raised by neutron，live migrate will be interrupt and vm task_state keep in migrating.

Steps to reproduce
==================
1.Set the Token timeout after 5 minutes.

2.Login dashboard, and live-migrate an CentOS instance after 4 minutes(the token will timeout after 1min)

3.bug reproduced,because of neutronclient Authentication required (token timeout)

Expected result
===============
Instance rollback or directly in error state.

Actual result
=============
Live migrate operation stoped，but instance task_state keeping in "migrating"

Environment
===========
version: Mitaka

hypervisor : Libvirt + KVM

storage: LVM

networking type: Neutron with OpenVSwitch

Logs & Configs
==============
In source node,nova-compute log as follow:

2016-07-02 16:14:17.809 10162 INFO nova.compute.manager [req-d322c8ac-2d6d-4960-8275-ac6e3f26f7ac ebe821cc991f4657aa3002054739933c 71acb857b6e34df6bfa2da07b0ce7902 - - -] [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] _post_live_migration() is started..
2016-07-02 16:14:18.072 10162 WARNING nova.virt.libvirt.driver [req-d322c8ac-2d6d-4960-8275-ac6e3f26f7ac ebe821cc991f4657aa3002054739933c 71acb857b6e34df6bfa2da07b0ce7902 - - -] [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] Error monitoring migration: Authentication required
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] Traceback (most recent call last):
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6504, in _live_migration
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] dom, finish_event)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6434, in _live_migration_monitor
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] migrate_data)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] payload)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] six.reraise(self.type_, self.value, self.tb)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] return f(self, context, *args, **kw)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 405, in decorated_function
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] kwargs['instance'], e, sys.exc_info())
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] six.reraise(self.type_, self.value, self.tb)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] return function(self, context, *args, **kwargs)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6329, in _post_live_migration
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] network_info = self._get_instance_nw_info(ctxt, instance)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1647, in _get_instance_nw_info
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] return self.network_api.get_instance_nw_info(context, instance)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 767, in get_instance_nw_info
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] preexisting_port_ids)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 783, in _get_instance_nw_info
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] preexisting_port_ids)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1741, in _rebuild_network_info_model
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] context, instance, current_neutron_ports)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1701, in _gather_networks_by_interface
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] iface['network_id'])
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 251, in _get_available_networks
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] nets = neutron.list_networks(**search_opts).get('networks', [])
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 99, in with_params
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] ret = self.function(instance, *args, **kwargs)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 526, in list_networks
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] **_params)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 304, in list
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] for r in self._pagination(collection, path, **params):
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 317, in _pagination
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] res = self.get(path, params=params)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 290, in get
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] headers=headers, params=params)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 267, in retry_request
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] headers=headers, params=params)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 208, in do_request
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] self._handle_fault_response(status_code, replybody)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 182, in _handle_fault_response
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] exception_handler_v20(status_code, des_error_body)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 80, in exception_handler_v20
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] message=message)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] NeutronClientException: Authentication required

[John Garbutt (johngarbutt)]

Ah quite right, in _do_live_migration in /nova/compute/manager.py you see the error handling sets the migration record to the error state, but it has not put the instance into the ERROR state, as it should have. See:
https://github.com/openstack/nova/blob/bd0ba826e4cc764f257d20c748c390caf73aa619/nova/compute/manager.py#L5218

[Sivasathurappan Radhakrishnan (siva-radhakrishnan)]

Hi Zengzhi Wang!
Please let me know if you are still working on this bug?

Regards,
Siva.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379032

[Luo Gangyi (luogangyi)]

I think we should ensure no such "NeutronClientException: Authentication required " occur instead of set instance to error.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sivasathurappan Radhakrishnan (<email address hidden>) on branch: master
Review: https://review.openstack.org/379032
Reason: We already have a patch for it https://review.openstack.org/#/c/342903/2 in review. Hence abandoning this one.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/342903
Reason: This review is > 6 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Timofey Durakov (tdurakov)]

this is actually the duplicate for 1647451

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sivasathurappan Radhakrishnan (<email address hidden>) on branch: master
Review: https://review.openstack.org/379032