Comment 1 for bug 1524898

http://logs.openstack.org/17/255217/5/check/gate-tempest-dsvm-neutron-dvr-multinode-full/15995d2/logs/subnode-2/screen-n-cpu.txt.gz?#_2015-12-10_08_14_34_437

2015-12-10 08:14:34.437 ERROR nova.virt.libvirt.driver [req-e8766399-ceeb-4b6c-82e9-bb5de1d8eed3 tempest-LiveBlockMigrationTestJSON-1334467432 tempest-LiveBlockMigrationTestJSON-138512217] [instance: f21cf8ce-fd9e-4fa3-9cf6-9fb5da8751ca] Live Migration failure: operation failed: migration job: unexpectedly failed
2015-12-10 08:14:34.438 DEBUG nova.virt.libvirt.driver [req-e8766399-ceeb-4b6c-82e9-bb5de1d8eed3 tempest-LiveBlockMigrationTestJSON-1334467432 tempest-LiveBlockMigrationTestJSON-138512217] [instance: f21cf8ce-fd9e-4fa3-9cf6-9fb5da8751ca] Migration operation thread notification thread_finished /opt/stack/new/nova/nova/virt/libvirt/driver.py:6130
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send
    waiter.switch(result)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/opt/stack/new/nova/nova/utils.py", line 1178, in context_wrapper
    return func(*args, **kwargs)
  File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5732, in _live_migration_operation
    instance=instance)
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5700, in _live_migration_operation
    CONF.libvirt.live_migration_bandwidth)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
    result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
    rv = execute(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
    six.reraise(c, e, tb)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
    rv = meth(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1586, in migrateToURI2
    if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
libvirtError: operation failed: migration job: unexpectedly failed

From the libvirtd logs:

http://logs.openstack.org/17/255217/5/check/gate-tempest-dsvm-neutron-dvr-multinode-full/15995d2/logs/subnode-2/libvirt/libvirtd.txt.gz#_2015-12-10_08_14_33_802

2015-12-10 08:14:33.802+0000: 12436: debug : qemuMonitorIOProcess:396 : QEMU_MONITOR_IO_PROCESS: mon=0x7f70100095e0 buf={"return": {"status": "failed"}, "id": "libvirt-108"}

QEMU log for the domain:

http://logs.openstack.org/17/255217/5/check/gate-tempest-dsvm-neutron-dvr-multinode-full/15995d2/logs/libvirt/qemu/instance-0000000a.txt.gz

From logstash it looks like this started on 12/8, and is isolated to the OVH and BlueBox nodes, which might correlate with enabling config drive:

https://review.openstack.org/#/c/254858/