libvirt: nova assumed dest emultor path is the same as source and fails to migrate if this is not true.

[sean mooney]

Description
===========
live migartion between compute nodes of different distors
fails due to asstion that the emulator path on dest is
the same as the source compute node. In past releases
it was acidentally posable to migrate between ubuntu and centos
host because the qemu-kvm binary was packaged in the same location.

in centos 7.5 and ubuntu 16.04 this is nolonger true.

Steps to reproduce
==================
deploy a multinode devestack with one ubuntu node and one centos node.
configure ssh keys and the other normal steps for live migration
and boot a vm on one host. then attempt to live migrate the vm.

Expected result
===============
live migration succeds as it did when the emulator paths were the same.

Actual result
=============

migration fails as the emulator path of the source is used to invoke the
emulator on the destination node resulting in a file not found error.

Environment
===========
1. Rock RC1 nova sha: afe4512bf66c89a061b1a7ccd3e7ac8e3b1b284d
   the exact version is not important as i belive we have alway
   had the same behavior.

2. Which hypervisor did you use?
  libvirt with kvm

2. Which storage type did you use?
  boot form volume with default devstack lvm cinder backend.

3. Which networking type did you use?
   ovs

Logs & Configs
==============

Aug 13 22:04:56 devstack2 nova-compute[1720]: Traceback (most recent call last):
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
Aug 13 22:04:56 devstack2 nova-compute[1720]: timer()
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
Aug 13 22:04:56 devstack2 nova-compute[1720]: cb(*args, **kw)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 168, in _do_send
Aug 13 22:04:56 devstack2 nova-compute[1720]: waiter.switch(result)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
Aug 13 22:04:56 devstack2 nova-compute[1720]: result = function(*args, **kwargs)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/opt/stack/nova/nova/utils.py", line 810, in context_wrapper
Aug 13 22:04:56 devstack2 nova-compute[1720]: return func(*args, **kwargs)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7134, in _live_migration_operation
Aug 13 22:04:56 devstack2 nova-compute[1720]: LOG.error("Live Migration failure: %s", e, instance=instance)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 13 22:04:56 devstack2 nova-compute[1720]: self.force_reraise()
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 13 22:04:56 devstack2 nova-compute[1720]: six.reraise(self.type_, self.value, self.tb)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7127, in _live_migration_operation
Aug 13 22:04:56 devstack2 nova-compute[1720]: bandwidth=CONF.libvirt.live_migration_bandwidth)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 673, in migrate
Aug 13 22:04:56 devstack2 nova-compute[1720]: destination, params=params, flags=flags)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
Aug 13 22:04:56 devstack2 nova-compute[1720]: result = proxy_call(self._autowrap, f, *args, **kwargs)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
Aug 13 22:04:56 devstack2 nova-compute[1720]: rv = execute(f, *args, **kwargs)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
Aug 13 22:04:56 devstack2 nova-compute[1720]: six.reraise(c, e, tb)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
Aug 13 22:04:56 devstack2 nova-compute[1720]: rv = meth(*args, **kwargs)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1745, in migrateToURI3
Aug 13 22:04:56 devstack2 nova-compute[1720]: if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
Aug 13 22:04:56 devstack2 nova-compute[1720]: libvirtError: Cannot check QEMU binary /usr/libexec/qemu-kvm: No such file or directory

[sean mooney (sean-k-mooney)]

set status to traige and importance to low.

as far as i am aware this is still the case however i found this why testing live migration and trying to break it and no one
has reported this in produciton or brought it up since i filed this almost a year ago so its a low priority to fix.