OpenStack Compute (nova)

  1. Series newton
  2. Bug #1662626
  3. Activity log

Activity log for bug #1662626

Date Who What changed Old value New value Message
2017-02-07 18:39:35 John Garbutt bug added bug
2017-02-07 18:42:03 John Garbutt description A live-migration stress test was working fine when suddenly a VM stopped migrating. It failed with this error: ERROR nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42 Error=Domain not found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' (instance-00000431) The full stack trace: 2017-02-05 02:33:45.795 19770 INFO nova.compute.manager [req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] VM Paused (Lifecycle Event) 2017-02-05 02:33:45.870 19770 INFO nova.compute.manager [req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] During sync_power_state the instance has a pending task (migrating). Skip. 2017-02-05 02:33:45.883 19770 INFO nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration operation has completed 2017-02-05 02:33:45.884 19770 INFO nova.compute.manager [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] _post_live_migration() is started.. 2017-02-05 02:33:46.156 19770 INFO os_vif [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:a2:90:55,bridge_name='brq476ab6ba-b3',has_traffic_filtering=True,id=98d476b3-0ead-4adb-ad54-1dff63edcd65,network=Network(476ab6ba-b32e-409e-9711-9412e8475ea0),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=True,vif_name='tap98d476b3-0e') 2017-02-05 02:33:46.189 19770 INFO nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Deleting instance files /var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del 2017-02-05 02:33:46.195 19770 INFO nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Deletion of /var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del complete 2017-02-05 02:33:46.334 19770 ERROR nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42 Error=Domain not found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' (instance-00000431) 2017-02-05 02:33:46.363 19770 WARNING nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Error monitoring migration: Domain not found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' (instance-00000431) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Traceback (most recent call last): 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6345, in _live_migration 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] finish_event, disk_paths) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6255, in _live_migration_monitor 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] migrate_data) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] function_name, call_dict, binary) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self.force_reraise() 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(self.type_, self.value, self.tb) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] return f(self, context, *args, **kw) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] kwargs['instance'], e, sys.exc_info()) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self.force_reraise() 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(self.type_, self.value, self.tb) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] return function(self, context, *args, **kwargs) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 5470, in _post_live_migration 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] destroy_vifs=destroy_vifs) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 976, in cleanup 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self._undefine_domain(instance) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 874, in _undefine_domain 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] {'errcode': errcode, 'e': e}, instance=instance) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self.force_reraise() 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(self.type_, self.value, self.tb) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 868, in _undefine_domain 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] guest.delete_configuration() 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 266, in delete_configuration 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self._domain.undefine() 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] result = proxy_call(self._autowrap, f, *args, **kwargs) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] rv = execute(f, *args, **kwargs) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(c, e, tb) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] rv = meth(*args, **kwargs) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/libvirt.py", line 2701, in undefine 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] if ret == -1: raise libvirtError ('virDomainUndefine() failed', dom=self) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] libvirtError: Domain not found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' (instance-00000431) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Live migration failed. 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Traceback (most recent call last): 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 5261, in _do_live_migration 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] block_migration, migrate_data) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5775, in live_migration 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] migrate_data) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6345, in _live_migration 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] finish_event, disk_paths) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6255, in _live_migration_monitor 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] migrate_data) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] function_name, call_dict, binary) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self.force_reraise() 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(self.type_, self.value, self.tb) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] return f(self, context, *args, **kw) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] kwargs['instance'], e, sys.exc_info()) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self.force_reraise() 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(self.type_, self.value, self.tb) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] return function(self, context, *args, **kwargs) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 5470, in _post_live_migration 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] destroy_vifs=destroy_vifs) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 976, in cleanup 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self._undefine_domain(instance) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 874, in _undefine_domain 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] {'errcode': errcode, 'e': e}, instance=instance) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self.force_reraise() 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(self.type_, self.value, self.tb) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 868, in _undefine_domain 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] guest.delete_configuration() 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 266, in delete_configuration 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self._domain.undefine() 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] result = proxy_call(self._autowrap, f, *args, **kwargs) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] rv = execute(f, *args, **kwargs) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(c, e, tb) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] rv = meth(*args, **kwargs) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/libvirt.py", line 2701, in undefine 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] if ret == -1: raise libvirtError ('virDomainUndefine() failed', dom=self) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] libvirtError: Domain not found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' (instance-00000431) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] A live-migration stress test was working fine when suddenly a VM stopped migrating. It failed with this error: ERROR nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42 Error=Domain not found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' (instance-00000431) The full stack trace: 2017-02-05 02:33:41.787 19770 INFO nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration running for 240 secs, memory 9% remaining; (bytes processed=15198240264, remaining=1680875520, total=17314955264) 2017-02-05 02:33:45.795 19770 INFO nova.compute.manager [req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] VM Paused (Lifecycle Event) 2017-02-05 02:33:45.870 19770 INFO nova.compute.manager [req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] During sync_power_state the instance has a pending task (migrating). Skip. 2017-02-05 02:33:45.883 19770 INFO nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration operation has completed 2017-02-05 02:33:45.884 19770 INFO nova.compute.manager [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] _post_live_migration() is started.. 2017-02-05 02:33:46.156 19770 INFO os_vif [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:a2:90:55,bridge_name='brq476ab6ba-b3',has_traffic_filtering=True,id=98d476b3-0ead-4adb-ad54-1dff63edcd65,network=Network(476ab6ba-b32e-409e-9711-9412e8475ea0),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=True,vif_name='tap98d476b3-0e') 2017-02-05 02:33:46.189 19770 INFO nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Deleting instance files /var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del 2017-02-05 02:33:46.195 19770 INFO nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Deletion of /var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del complete 2017-02-05 02:33:46.334 19770 ERROR nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42 Error=Domain not found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' (instance-00000431) 2017-02-05 02:33:46.363 19770 WARNING nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Error monitoring migration: Domain not found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' (instance-00000431) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Traceback (most recent call last): 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6345, in _live_migration 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] finish_event, disk_paths) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6255, in _live_migration_monitor 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] migrate_data) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] function_name, call_dict, binary) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self.force_reraise() 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(self.type_, self.value, self.tb) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] return f(self, context, *args, **kw) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] kwargs['instance'], e, sys.exc_info()) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self.force_reraise() 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(self.type_, self.value, self.tb) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] return function(self, context, *args, **kwargs) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 5470, in _post_live_migration 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] destroy_vifs=destroy_vifs) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 976, in cleanup 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self._undefine_domain(instance) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 874, in _undefine_domain 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] {'errcode': errcode, 'e': e}, instance=instance) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self.force_reraise() 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(self.type_, self.value, self.tb) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 868, in _undefine_domain 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] guest.delete_configuration() 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 266, in delete_configuration 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self._domain.undefine() 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] result = proxy_call(self._autowrap, f, *args, **kwargs) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] rv = execute(f, *args, **kwargs) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(c, e, tb) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] rv = meth(*args, **kwargs) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/libvirt.py", line 2701, in undefine 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] if ret == -1: raise libvirtError ('virDomainUndefine() failed', dom=self) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] libvirtError: Domain not found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' (instance-00000431) 2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Live migration failed. 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] Traceback (most recent call last): 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 5261, in _do_live_migration 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] block_migration, migrate_data) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5775, in live_migration 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] migrate_data) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6345, in _live_migration 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] finish_event, disk_paths) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6255, in _live_migration_monitor 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] migrate_data) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] function_name, call_dict, binary) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self.force_reraise() 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(self.type_, self.value, self.tb) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] return f(self, context, *args, **kw) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] kwargs['instance'], e, sys.exc_info()) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self.force_reraise() 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(self.type_, self.value, self.tb) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] return function(self, context, *args, **kwargs) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/compute/manager.py", line 5470, in _post_live_migration 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] destroy_vifs=destroy_vifs) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 976, in cleanup 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self._undefine_domain(instance) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 874, in _undefine_domain 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] {'errcode': errcode, 'e': e}, instance=instance) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self.force_reraise() 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(self.type_, self.value, self.tb) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 868, in _undefine_domain 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] guest.delete_configuration() 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 266, in delete_configuration 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] self._domain.undefine() 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] result = proxy_call(self._autowrap, f, *args, **kwargs) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] rv = execute(f, *args, **kwargs) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] six.reraise(c, e, tb) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] rv = meth(*args, **kwargs) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] File "/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/libvirt.py", line 2701, in undefine 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] if ret == -1: raise libvirtError ('virDomainUndefine() failed', dom=self) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b] libvirtError: Domain not found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' (instance-00000431) 2017-02-05 02:33:46.364 19770 ERROR nova.compute.manager [instance: 62034d78-3144-4efd-9c2c-8a792aed3d6b]
2017-02-07 19:00:32 OpenStack Infra nova: status New In Progress
2017-02-08 11:50:33 John Garbutt nova: importance Undecided Medium
2017-02-08 11:54:06 John Garbutt tags live-migration live-migration ocata-rc-potential
2017-02-17 02:49:36 Matt Riedemann nominated for series nova/ocata
2017-02-17 02:49:36 Matt Riedemann bug task added nova/ocata
2017-02-17 02:49:52 Matt Riedemann tags live-migration ocata-rc-potential live-migration ocata-backport-potential
2017-03-27 13:30:27 OpenStack Infra nova: status In Progress Fix Released
2017-04-15 18:58:43 OpenStack Infra nova/ocata: status New In Progress
2017-04-15 18:58:43 OpenStack Infra nova/ocata: assignee Matt Riedemann (mriedem)
2017-04-25 21:21:53 OpenStack Infra nova/ocata: status In Progress Fix Committed
2017-08-11 19:34:45 Matt Riedemann nova/ocata: importance Undecided Medium
2017-08-11 19:34:48 Matt Riedemann nominated for series nova/newton
2017-08-11 19:34:48 Matt Riedemann bug task added nova/newton
2017-08-11 19:34:59 Matt Riedemann nova/newton: assignee Shane Peters (shaner)
2017-08-11 19:35:02 Matt Riedemann nova/newton: status New In Progress
2017-08-11 19:35:07 Matt Riedemann nova/newton: importance Undecided Medium
2017-08-11 19:35:18 Matt Riedemann tags live-migration ocata-backport-potential libvirt live-migration
2017-08-12 01:22:26 OpenStack Infra nova/newton: status In Progress Fix Committed
2017-10-05 21:41:11 Shane Peters bug task added cloud-archive
2017-10-05 21:44:50 Billy Olsen nominated for series cloud-archive/mitaka
2017-10-05 21:45:11 Billy Olsen cloud-archive: importance Undecided Medium
2017-10-05 21:45:26 Billy Olsen cloud-archive: assignee Shane Peters (shaner)
2017-10-05 21:46:09 Billy Olsen cloud-archive: status New Confirmed
2017-10-19 18:30:27 Corey Bryant cloud-archive: status Confirmed Fix Released
2017-10-19 18:31:44 Corey Bryant bug task added nova (Ubuntu)
2017-10-19 18:31:54 Corey Bryant nova (Ubuntu): status New Fix Released
2017-10-19 18:31:58 Corey Bryant nova (Ubuntu): importance Undecided Medium
2017-10-19 18:32:14 Corey Bryant nominated for series Ubuntu Artful
2017-10-19 18:32:14 Corey Bryant bug task added nova (Ubuntu Artful)
2017-10-19 18:32:14 Corey Bryant nominated for series Ubuntu Zesty
2017-10-19 18:32:14 Corey Bryant bug task added nova (Ubuntu Zesty)
2017-10-19 18:32:14 Corey Bryant nominated for series Ubuntu Xenial
2017-10-19 18:32:14 Corey Bryant bug task added nova (Ubuntu Xenial)
2017-10-19 19:00:51 Corey Bryant nova (Ubuntu Zesty): status New Fix Released
2017-10-19 19:03:06 Corey Bryant nova (Ubuntu Xenial): status New Triaged
2017-10-19 19:04:34 Corey Bryant bug task added cloud-archive/mitaka
2017-10-19 19:04:40 Corey Bryant cloud-archive/mitaka: status New Triaged
2017-10-19 19:04:43 Corey Bryant cloud-archive/mitaka: importance Undecided Medium
2017-10-19 19:04:46 Corey Bryant nova (Ubuntu Xenial): importance Undecided Medium
2017-10-19 19:04:48 Corey Bryant nova (Ubuntu Zesty): importance Undecided Medium
2017-10-19 19:15:22 Corey Bryant nominated for series cloud-archive/newton
2017-10-19 19:15:22 Corey Bryant bug task added cloud-archive/newton
2017-10-19 19:15:22 Corey Bryant nominated for series cloud-archive/ocata
2017-10-19 19:15:22 Corey Bryant bug task added cloud-archive/ocata
2017-10-19 19:15:22 Corey Bryant nominated for series cloud-archive/pike
2017-10-19 19:15:22 Corey Bryant bug task added cloud-archive/pike
2017-10-19 19:15:37 Corey Bryant cloud-archive/newton: importance Undecided Medium
2017-10-19 19:15:37 Corey Bryant cloud-archive/newton: status New Fix Released
2017-10-19 19:15:49 Corey Bryant cloud-archive/ocata: importance Undecided Medium
2017-10-19 19:15:49 Corey Bryant cloud-archive/ocata: status New Fix Released
2018-02-02 10:04:36 OpenStack Infra tags libvirt live-migration in-stable-ocata libvirt live-migration