live-migration Interrupt when get network info failed

Bug #1599057 reported by John Smith on 2016-07-05
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Sivasathurappan Radhakrishnan

Bug Description

Description
===========
When an exception raised by neutron,live migrate will be interrupt and vm task_state keep in migrating.

Steps to reproduce
==================
1.Set the Token timeout after 5 minutes.

2.Login dashboard, and live-migrate an CentOS instance after 4 minutes(the token will timeout after 1min)

3.bug reproduced,because of neutronclient Authentication required (token timeout)

Expected result
===============
Instance rollback or directly in error state.

Actual result
=============
Live migrate operation stoped,but instance task_state keeping in "migrating"

Environment
===========
version: Mitaka

hypervisor : Libvirt + KVM

storage: LVM

networking type: Neutron with OpenVSwitch

Logs & Configs
==============
In source node,nova-compute log as follow:

2016-07-02 16:14:17.809 10162 INFO nova.compute.manager [req-d322c8ac-2d6d-4960-8275-ac6e3f26f7ac ebe821cc991f4657aa3002054739933c 71acb857b6e34df6bfa2da07b0ce7902 - - -] [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] _post_live_migration() is started..
2016-07-02 16:14:18.072 10162 WARNING nova.virt.libvirt.driver [req-d322c8ac-2d6d-4960-8275-ac6e3f26f7ac ebe821cc991f4657aa3002054739933c 71acb857b6e34df6bfa2da07b0ce7902 - - -] [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] Error monitoring migration: Authentication required
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] Traceback (most recent call last):
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6504, in _live_migration
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] dom, finish_event)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6434, in _live_migration_monitor
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] migrate_data)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] payload)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] six.reraise(self.type_, self.value, self.tb)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] return f(self, context, *args, **kw)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 405, in decorated_function
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] kwargs['instance'], e, sys.exc_info())
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] six.reraise(self.type_, self.value, self.tb)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] return function(self, context, *args, **kwargs)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6329, in _post_live_migration
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] network_info = self._get_instance_nw_info(ctxt, instance)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1647, in _get_instance_nw_info
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] return self.network_api.get_instance_nw_info(context, instance)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 767, in get_instance_nw_info
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] preexisting_port_ids)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 783, in _get_instance_nw_info
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] preexisting_port_ids)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1741, in _rebuild_network_info_model
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] context, instance, current_neutron_ports)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1701, in _gather_networks_by_interface
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] iface['network_id'])
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 251, in _get_available_networks
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] nets = neutron.list_networks(**search_opts).get('networks', [])
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 99, in with_params
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] ret = self.function(instance, *args, **kwargs)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 526, in list_networks
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] **_params)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 304, in list
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] for r in self._pagination(collection, path, **params):
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 317, in _pagination
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] res = self.get(path, params=params)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 290, in get
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] headers=headers, params=params)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 267, in retry_request
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] headers=headers, params=params)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 208, in do_request
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] self._handle_fault_response(status_code, replybody)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 182, in _handle_fault_response
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] exception_handler_v20(status_code, des_error_body)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 80, in exception_handler_v20
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] message=message)
2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] NeutronClientException: Authentication required

Changed in nova:
assignee: nobody → Zengzhi Wang (wang-zengzhi)
tags: added: live-migration
John Garbutt (johngarbutt) wrote :

Ah quite right, in _do_live_migration in /nova/compute/manager.py you see the error handling sets the migration record to the error state, but it has not put the instance into the ERROR state, as it should have. See:
https://github.com/openstack/nova/blob/bd0ba826e4cc764f257d20c748c390caf73aa619/nova/compute/manager.py#L5218

Changed in nova:
status: New → Triaged
importance: Undecided → Medium

Hi Zengzhi Wang!
Please let me know if you are still working on this bug?

Regards,
Siva.

Changed in nova:
assignee: Zengzhi Wang (wang-zengzhi) → Sivasathurappan Radhakrishnan (siva-radhakrishnan)

Fix proposed to branch: master
Review: https://review.openstack.org/379032

Changed in nova:
status: Triaged → In Progress
Luo Gangyi (luogangyi) wrote :

I think we should ensure no such "NeutronClientException: Authentication required " occur instead of set instance to error.

Change abandoned by Sivasathurappan Radhakrishnan (<email address hidden>) on branch: master
Review: https://review.openstack.org/379032
Reason: We already have a patch for it https://review.openstack.org/#/c/342903/2 in review. Hence abandoning this one.

Changed in nova:
assignee: Sivasathurappan Radhakrishnan (siva-radhakrishnan) → nobody

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/342903
Reason: This review is > 6 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Changed in nova:
assignee: nobody → Sivasathurappan Radhakrishnan (siva-radhakrishnan)
Timofey Durakov (tdurakov) wrote :

this is actually the duplicate for 1647451

Change abandoned by Sivasathurappan Radhakrishnan (<email address hidden>) on branch: master
Review: https://review.openstack.org/379032

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers