Another hit today. https://zuul.opendev.org/t/openstack/build/674a5f21a91d4af0861d6ab3c2cc6834/log/controller/logs/screen-n-cpu.txt#12057 Jun 11 10:31:38.867508 ubuntu-focal-rax-ord-0025071874 nova-compute[108685]: ERROR os_brick.initiator.linuxscsi [None req-1333f0cc-148b-4f4c-a845-460900ec49d7 tempest-LiveMigrationTest-666538249 tempest-LiveMigrationTest-666538249-project-admin] multipathd is not running: exit code None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory and https://zuul.opendev.org/t/openstack/build/674a5f21a91d4af0861d6ab3c2cc6834/log/compute1/logs/screen-n-cpu.txt#11399 Jun 11 10:31:38.471494 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: ERROR nova.virt.libvirt.driver [-] [instance: 34236c1a-1cbf-448b-b36a-9daf1cf760e8] Migration operation has aborted Jun 11 10:31:38.472905 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: ERROR nova.virt.libvirt.driver [-] [instance: 34236c1a-1cbf-448b-b36a-9daf1cf760e8] Live Migration failure: unable to connect to server at 'ubuntu-focal-rax-ord-0025071874:49152': Connection refused: libvirt.libvirtError: unable to connect to server at 'ubuntu-focal-rax-ord-0025071874:49152': Connection refused Jun 11 10:31:38.473132 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: DEBUG nova.virt.libvirt.driver [-] [instance: 34236c1a-1cbf-448b-b36a-9daf1cf760e8] Migration operation thread notification {{(pid=68341) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:9921}} Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: Traceback (most recent call last): Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/hub.py", line 476, in fire_timers Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: timer() Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/timer.py", line 59, in __call__ Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: cb(*args, **kw) Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/usr/local/lib/python3.8/dist-packages/eventlet/event.py", line 175, in _do_send Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: waiter.switch(result) Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenthread.py", line 221, in main Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: result = function(*args, **kwargs) Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/opt/stack/nova/nova/utils.py", line 654, in context_wrapper Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: return func(*args, **kwargs) Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 9576, in _live_migration_operation Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: LOG.error("Live Migration failure: %s", e, instance=instance) Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: self.force_reraise() Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: raise self.value Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 9564, in _live_migration_operation Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: guest.migrate(self._live_migration_uri(dest), Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 617, in migrate Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: self._domain.migrateToURI3( Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 190, in doit Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: result = proxy_call(self._autowrap, f, *args, **kwargs) Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 148, in proxy_call Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: rv = execute(f, *args, **kwargs) Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 129, in execute Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: six.reraise(c, e, tb) Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: raise value Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 83, in tworker Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: rv = meth(*args, **kwargs) Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: File "/usr/local/lib/python3.8/dist-packages/libvirt.py", line 2099, in migrateToURI3 Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: raise libvirtError('virDomainMigrateToURI3() failed') Jun 11 10:31:38.481216 ubuntu-focal-rax-ord-0025071879 nova-compute[68341]: libvirt.libvirtError: unable to connect to server at 'ubuntu-focal-rax-ord-0025071874:49152': Connection refused