Volume based live migration aborted unexpectedly
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Invalid
|
High
|
Unassigned |
Bug Description
Volume based live migration is failing during tempest testing in the check and experimental pipelines
http://
shows 42 failures since 12/8
2015-12-18 15:26:26.411 ERROR nova.virt.
2015-12-18 15:26:26.412 DEBUG nova.virt.
Traceback (most recent call last):
File "/usr/local/
timer()
File "/usr/local/
cb(*args, **kw)
File "/usr/local/
waiter.
File "/usr/local/
result = function(*args, **kwargs)
File "/opt/stack/
return func(*args, **kwargs)
File "/opt/stack/
instance=
File "/usr/local/
six.
File "/opt/stack/
CONF.
File "/usr/local/
result = proxy_call(
File "/usr/local/
rv = execute(f, *args, **kwargs)
File "/usr/local/
six.reraise(c, e, tb)
File "/usr/local/
rv = meth(*args, **kwargs)
File "/usr/local/
if ret == -1: raise libvirtError ('virDomainMigr
libvirtError: operation failed: migration job: unexpectedly failed
summary: |
- Volume based live migration fails unexpectedly + Volume based live migration aborted unexpectedly |
description: | updated |
Changed in nova: | |
status: | Confirmed → In Progress |
Changed in nova: | |
assignee: | Lee Yarwood (lyarwood) → melanie witt (melwitt) |
Changed in nova: | |
status: | In Progress → Invalid |
assignee: | melanie witt (melwitt) → nobody |
http:// logs.openstack. org/17/ 255217/ 5/check/ gate-tempest- dsvm-neutron- dvr-multinode- full/15995d2/ logs/subnode- 2/screen- n-cpu.txt. gz?#_2015- 12-10_08_ 14_34_437
2015-12-10 08:14:34.437 ERROR nova.virt. libvirt. driver [req-e8766399- ceeb-4b6c- 82e9-bb5de1d8ee d3 tempest- LiveBlockMigrat ionTestJSON- 1334467432 tempest- LiveBlockMigrat ionTestJSON- 138512217] [instance: f21cf8ce- fd9e-4fa3- 9cf6-9fb5da8751 ca] Live Migration failure: operation failed: migration job: unexpectedly failed libvirt. driver [req-e8766399- ceeb-4b6c- 82e9-bb5de1d8ee d3 tempest- LiveBlockMigrat ionTestJSON- 1334467432 tempest- LiveBlockMigrat ionTestJSON- 138512217] [instance: f21cf8ce- fd9e-4fa3- 9cf6-9fb5da8751 ca] Migration operation thread notification thread_finished /opt/stack/ new/nova/ nova/virt/ libvirt/ driver. py:6130 lib/python2. 7/dist- packages/ eventlet/ hubs/hub. py", line 457, in fire_timers lib/python2. 7/dist- packages/ eventlet/ hubs/timer. py", line 58, in __call__ lib/python2. 7/dist- packages/ eventlet/ event.py" , line 168, in _do_send switch( result) lib/python2. 7/dist- packages/ eventlet/ greenthread. py", line 214, in main new/nova/ nova/utils. py", line 1178, in context_wrapper new/nova/ nova/virt/ libvirt/ driver. py", line 5732, in _live_migration _operation instance) lib/python2. 7/dist- packages/ oslo_utils/ excutils. py", line 204, in __exit__ reraise( self.type_ , self.value, self.tb) new/nova/ nova/virt/ libvirt/ driver. py", line 5700, in _live_migration _operation libvirt. live_migration_ bandwidth) lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 183, in doit self._autowrap, f, *args, **kwargs) lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 141, in proxy_call lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 122, in execute lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 80, in tworker lib/python2. 7/dist- packages/ libvirt. py", line 1586, in migrateToURI2 ateToURI2( ) failed', dom=self)
2015-12-10 08:14:34.438 DEBUG nova.virt.
Traceback (most recent call last):
File "/usr/local/
timer()
File "/usr/local/
cb(*args, **kw)
File "/usr/local/
waiter.
File "/usr/local/
result = function(*args, **kwargs)
File "/opt/stack/
return func(*args, **kwargs)
File "/opt/stack/
instance=
File "/usr/local/
six.
File "/opt/stack/
CONF.
File "/usr/local/
result = proxy_call(
File "/usr/local/
rv = execute(f, *args, **kwargs)
File "/usr/local/
six.reraise(c, e, tb)
File "/usr/local/
rv = meth(*args, **kwargs)
File "/usr/local/
if ret == -1: raise libvirtError ('virDomainMigr
libvirtError: operation failed: migration job: unexpectedly failed
From the libvirtd logs:
http:// logs.openstack. org/17/ 255217/ 5/check/ gate-tempest- dsvm-neutron- dvr-multinode- full/15995d2/ logs/subnode- 2/libvirt/ libvirtd. txt.gz# _2015-12- 10_08_14_ 33_802
2015-12-10 08:14:33.802+0000: 12436: debug : qemuMonitorIOPr ocess:396 : QEMU_MONITOR_ IO_PROCESS: mon=0x7f70100095e0 buf={"return": {"status": "failed"}, "id": "libvirt-108"}
QEMU log for the domain:
http:// logs.openstack. org/17/ 255217/ 5/check/ gate-tempest- dsvm-neutron- dvr-multinode- full/15995d2/ logs/libvirt/ qemu/instance- 0000000a. txt.gz
From logstash it looks like this started on 12/8, and is isolated to the OVH and BlueBox node...