As seen in these two test runs: https://solutions.qa.canonical.com/testruns/testRun/2f1dd1c5-0843-4afb-a29f-b8b4a71450a5 and https://solutions.qa.canonical.com/testruns/testRun/350ae74a-c8be-4da4-9174-eae38c081fec
Running Ussuri Focal and Bionic respectively, using OVN. During rally test execution nova fails to launch an instance, failing with:
f6c-4eb2-b3f2-af1ba3359fd2] Timeout waiting for [('network-vif-plugged', '5f994790-54ed-4b42-8e22-2132cc625f1e')] for instance with vm_state building and task_state spawning.: eventlet.timeout.Timeout: 300 seconds
2021-08-18 09:16:48.068 1288687 INFO os_vif [req-13b9f103-19ae-4a6c-9fbe-30073de62e9e 126d233a10704f1da37d674cb3028cc4 88d8532639e84e878163c65f62fd23ec - 1639e3b13e654c55b4eaf9fe5bd7bd6a 1639e3b13e654c55b4eaf9fe5bd7bd6a] Successfully unplugged vif VIFOpenVSwitch(
active=False,address=fa:16:3e:0d:0f:1c,bridge_name='br-int',has_traffic_filtering=True,id=5f994790-54ed-4b42-8e22-2132cc625f1e,network=Network(43da95fc-1396-4115-b564-a4414511eb44),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_n
ame='tap5f994790-54')
2021-08-18 09:16:48.069 1288687 INFO nova.virt.libvirt.driver [req-13b9f103-19ae-4a6c-9fbe-30073de62e9e 126d233a10704f1da37d674cb3028cc4 88d8532639e84e878163c65f62fd23ec - 1639e3b13e654c55b4eaf9fe5bd7bd6a 1639e3b13e654c55b4eaf9fe5bd7bd6a] [instance: 3b2c53d8-ff6c
-4eb2-b3f2-af1ba3359fd2] Deleting instance files /var/lib/nova/instances/3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2_del
2021-08-18 09:16:48.070 1288687 INFO nova.virt.libvirt.driver [req-13b9f103-19ae-4a6c-9fbe-30073de62e9e 126d233a10704f1da37d674cb3028cc4 88d8532639e84e878163c65f62fd23ec - 1639e3b13e654c55b4eaf9fe5bd7bd6a 1639e3b13e654c55b4eaf9fe5bd7bd6a] [instance: 3b2c53d8-ff6c
-4eb2-b3f2-af1ba3359fd2] Deletion of /var/lib/nova/instances/3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2_del complete
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [req-13b9f103-19ae-4a6c-9fbe-30073de62e9e 126d233a10704f1da37d674cb3028cc4 88d8532639e84e878163c65f62fd23ec - 1639e3b13e654c55b4eaf9fe5bd7bd6a 1639e3b13e654c55b4eaf9fe5bd7bd6a] [instance: 3b2c53d8-ff6c-4e
b2-b3f2-af1ba3359fd2] Instance failed to spawn: nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] Traceback (most recent call last):
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6594, in _create_domain_and_network
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] guest = self._create_domain(
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] next(self.gen)
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 513, in wait_for_instance_event
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] actual_event = event.wait()
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] File "/usr/lib/python3/dist-packages/eventlet/event.py", line 125, in wait
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] result = hub.switch()
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] File "/usr/lib/python3/dist-packages/eventlet/hubs/hub.py", line 298, in switch
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] return self.greenlet.switch()
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] eventlet.timeout.Timeout: 300 seconds
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2]
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] During handling of the above exception, another exception occurred:
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2]
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] Traceback (most recent call last):
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2621, in _build_resources
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] yield resources
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2381, in _build_and_run_instance
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] self.driver.spawn(context, instance, image_meta,
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3684, in spawn
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] self._create_domain_and_network(
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6619, in _create_domain_and_network
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] raise exception.VirtualInterfaceCreateException()
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed
2021-08-18 09:16:48.177 1288687 ERROR nova.compute.manager [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2]
2021-08-18 09:16:48.190 1288687 INFO nova.compute.manager [req-13b9f103-19ae-4a6c-9fbe-30073de62e9e 126d233a10704f1da37d674cb3028cc4 88d8532639e84e878163c65f62fd23ec - 1639e3b13e654c55b4eaf9fe5bd7bd6a 1639e3b13e654c55b4eaf9fe5bd7bd6a] [instance: 3b2c53d8-ff6c-4eb
2-b3f2-af1ba3359fd2] Terminating instance
2021-08-18 09:16:48.197 1288687 INFO nova.virt.libvirt.driver [-] [instance: 3b2c53d8-ff6c-4eb2-b3f2-af1ba3359fd2] Instance destroyed successfully.
Poking around in the logs (baremetal machine 4 in this case) for that instance and nothing really jumps out, we're hoping someone can help shed some light on this failure.
Crashdumps and bundles for each run can be found here:
https://oil-jenkins.canonical.com/artifacts/2f1dd1c5-0843-4afb-a29f-b8b4a71450a5/index.html
and here:
https://oil-jenkins.canonical.com/artifacts/350ae74a-c8be-4da4-9174-eae38c081fec/index.html
All instances of this failure can be found at:
https://solutions.qa.canonical.com/bugs/bugs/bug/1940554
I was also hit by this bug in the cloud env I'm working on. The env is available and the issue is reproducible, so if any more information is required - I'd be happy to share.