Test fails like:-
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 150, in test_live_block_migration
self._test_live_migration()
File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 135, in _test_live_migration
self._live_migrate(server_id, destination_host, state, volume_backed)
File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 94, in _live_migrate
self.assertEqual(target_host, self.get_host_for_server(server_id),
File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/testtools/testcase.py", line 395, in assertEqual
self.assertThat(observed, matcher, message)
File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/testtools/testcase.py", line 482, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: 'np0037423811' != 'np0037423819': Live Migration failed. Migrations list for Instance 490cfc79-aa11-481d-82e3-99ca38fd2a07: [
{'id': 1, 'source_compute': 'np0037423819', 'dest_compute': 'np0037423811', 'source_node': 'np0037423819', 'dest_node': 'np0037423811', 'dest_host': None, 'old_instance_type_id': 11, 'new_instance_type_id': 11, 'instance_uuid': '490cfc79-aa11-481d-82e3-99ca38fd2a07', 'status': 'failed', 'migration_type': 'live-migration', 'created_at': '2024-05-02T10:11:10.000000', 'updated_at': '2024-05-02T10:11:17.000000'}]
sample failures:-
- http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d01/860283/32/check/nova-live-migration/d01ef23/testr_results.html
- https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_397/916112/6/gate/neutron-ovs-tempest-multinode-full/3978b33/testr_results.html
- https://d64b7553b36edb75caae-206dd41ab15ee6b59e703e19aac8d4d8.ssl.cf1.rackcdn.com/918195/8/check/nova-grenade-multinode/506fec9/testr_results.html
Nova compute have Traces:-
May 02 10:11:16.775995 np0037423819 nova-compute[30136]: ERROR nova.virt.libvirt.driver [None req-0ef45e21-38a4-44ae-aa70-6d59daf17b5b tempest-LiveAutoBlockMigrationV225Test-2095721713 tempest-LiveAutoBlockMigrationV225Test-2095721713-project-admin] [instance: 490cfc79-aa11-481d-82e3-99ca38fd2a07] Live Migration failure: unable to connect to server at 'np0037423811:49152': Connection refused: libvirt.libvirtError: unable to connect to server at 'np0037423811:49152': Connection refused
May 02 10:11:16.776306 np0037423819 nova-compute[30136]: DEBUG nova.virt.libvirt.driver [None req-0ef45e21-38a4-44ae-aa70-6d59daf17b5b tempest-LiveAutoBlockMigrationV225Test-2095721713 tempest-LiveAutoBlockMigrationV225Test-2095721713-project-admin] [instance: 490cfc79-aa11-481d-82e3-99ca38fd2a07] Migration operation thread notification {{(pid=30136) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:11059}}
May 02 10:11:16.781772 np0037423819 nova-compute[30136]: Traceback (most recent call last):
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: timer()
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/timer.py", line 59, in __call__
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: cb(*args, **kw)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 173, in _do_send
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: waiter.switch(result)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: result = function(*args, **kwargs)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: return func(*args, **kwargs)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10713, in _live_migration_operation
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: with excutils.save_and_reraise_exception():
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: self.force_reraise()
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: raise self.value
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10702, in _live_migration_operation
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: guest.migrate(self._live_migration_uri(dest),
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 647, in migrate
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: self._domain.migrateToURI3(
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: result = proxy_call(self._autowrap, f, *args, **kwargs)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: rv = execute(f, *args, **kwargs)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: raise e.with_traceback(tb)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: rv = meth(*args, **kwargs)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: File "/usr/lib/python3/dist-packages/libvirt.py", line 2126, in migrateToURI3
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: raise libvirtError('virDomainMigrateToURI3() failed')
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: libvirt.libvirtError: unable to connect to server at 'np0037423811:49152': Connection refused
