In testrun https://solutions.qa.canonical.com/v2/testruns/e31df22c-facf-4532-86de-20fb295610ea (yoga on focal), the tempest test tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_instance fails with:
```
tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_instance
fail 728.002
2b3a740c-181a-4149-b500-d552cc133ec9
Traceback (most recent call last):
File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-e3045414-d4cc-496d-90a5-b8bc217ebb73/repo/tempest/common/utils/__init__.py", line 70, in wrapper
return f(*func_args, **func_kwargs)
File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-e3045414-d4cc-496d-90a5-b8bc217ebb73/repo/tempest/scenario/test_shelve_instance.py", line 112, in test_shelve_instance
self._create_server_then_shelve_and_unshelve()
File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-e3045414-d4cc-496d-90a5-b8bc217ebb73/repo/tempest/scenario/test_shelve_instance.py", line 95, in _create_server_then_shelve_and_unshelve
self._shelve_then_unshelve_server(server)
File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-e3045414-d4cc-496d-90a5-b8bc217ebb73/repo/tempest/scenario/test_shelve_instance.py", line 59, in _shelve_then_unshelve_server
waiters.wait_for_server_status(self.servers_client, server['id'],
File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-e3045414-d4cc-496d-90a5-b8bc217ebb73/repo/tempest/common/waiters.py", line 101, in wait_for_server_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (TestShelveInstance:test_shelve_instance) Server c16475f6-417c-4066-8c80-67ac3b45b13d failed to reach ACTIVE status and task state "None" within the required time (600 s). Current status: SHELVED_OFFLOADED. Current task state: None.
```
In the nova logs we see:
```
2022-11-26 18:05:48.027 582295 INFO nova.compute.manager [req-ff74d716-fa9b-4592-9527-f9f2a9f2734e ca3b01fe03d54d42acf5cbc138dd85d0 2131979f9fe0493daca69cb318e50fb6 - bf7977fdc4fd4bc4995396682b86d2a8 bf7977fdc4fd4bc4995396682b86d2a8] [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] Unshelving
2022-11-26 18:05:48.131 582295 INFO nova.compute.claims [req-ff74d716-fa9b-4592-9527-f9f2a9f2734e ca3b01fe03d54d42acf5cbc138dd85d0 2131979f9fe0493daca69cb318e50fb6 - bf7977fdc4fd4bc4995396682b86d2a8 bf7977fdc4fd4bc4995396682b86d2a8] [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] Claim successful on node solqa-lab1-server-44.nosilo.lab1.solutionsqa
2022-11-26 18:05:48.552 582295 INFO nova.network.neutron [req-ff74d716-fa9b-4592-9527-f9f2a9f2734e ca3b01fe03d54d42acf5cbc138dd85d0 2131979f9fe0493daca69cb318e50fb6 - bf7977fdc4fd4bc4995396682b86d2a8 bf7977fdc4fd4bc4995396682b86d2a8] [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] Updating port dd9ad86e-28d2-4e27-836f-5dbce9eca8a1 with attributes {'binding:host_id': 'solqa-lab1-server-44.nosilo.lab1.solutionsqa', 'device_owner': 'compute:nova'}
2022-11-26 18:05:50.741 582295 INFO nova.virt.libvirt.driver [req-ff74d716-fa9b-4592-9527-f9f2a9f2734e ca3b01fe03d54d42acf5cbc138dd85d0 2131979f9fe0493daca69cb318e50fb6 - bf7977fdc4fd4bc4995396682b86d2a8 bf7977fdc4fd4bc4995396682b86d2a8] [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] Creating image
2022-11-26 18:06:15.995 582295 INFO nova.virt.libvirt.driver [req-ff74d716-fa9b-4592-9527-f9f2a9f2734e ca3b01fe03d54d42acf5cbc138dd85d0 2131979f9fe0493daca69cb318e50fb6 - bf7977fdc4fd4bc4995396682b86d2a8 bf7977fdc4fd4bc4995396682b86d2a8] [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] Rebasing disk image.
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [req-ff74d716-fa9b-4592-9527-f9f2a9f2734e ca3b01fe03d54d42acf5cbc138dd85d0 2131979f9fe0493daca69cb318e50fb6 - bf7977fdc4fd4bc4995396682b86d2a8 bf7977fdc4fd4bc4995396682b86d2a8] [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] Instance failed to spawn: nova.exception.InternalError: Unexpected vif_type=binding_failed
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] Traceback (most recent call last):
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6741, in _unshelve_instance
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] self.driver.spawn(context, instance, image_meta,
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4329, in spawn
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] xml = self._get_guest_xml(context, instance, network_info,
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7300, in _get_guest_xml
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] conf = self._get_guest_config(instance, network_info, image_meta,
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6914, in _get_guest_config
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] config = self.vif_driver.get_config(
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 600, in get_config
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] raise exception.InternalError(_('Unexpected vif_type=%s') % vif_type)
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] nova.exception.InternalError: Unexpected vif_type=binding_failed
2022-11-26 18:06:42.452 582295 ERROR nova.compute.manager [instance: c16475f6-417c-4066-8c80-67ac3b45b13d]
2022-11-26 18:06:42.537 582295 INFO nova.scheduler.client.report [req-ff74d716-fa9b-4592-9527-f9f2a9f2734e ca3b01fe03d54d42acf5cbc138dd85d0 2131979f9fe0493daca69cb318e50fb6 - bf7977fdc4fd4bc4995396682b86d2a8 bf7977fdc4fd4bc4995396682b86d2a8] Deleted allocations for instance c16475f6-417c-4066-8c80-67ac3b45b13d
2022-11-26 18:06:42.649 582295 INFO nova.compute.manager [req-ff74d716-fa9b-4592-9527-f9f2a9f2734e ca3b01fe03d54d42acf5cbc138dd85d0 2131979f9fe0493daca69cb318e50fb6 - bf7977fdc4fd4bc4995396682b86d2a8 bf7977fdc4fd4bc4995396682b86d2a8] [instance: c16475f6-417c-4066-8c80-67ac3b45b13d] Successfully reverted task state from spawning on failure for instance.
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server [req-ff74d716-fa9b-4592-9527-f9f2a9f2734e ca3b01fe03d54d42acf5cbc138dd85d0 2131979f9fe0493daca69cb318e50fb6 - bf7977fdc4fd4bc4995396682b86d2a8 bf7977fdc4fd4bc4995396682b86d2a8] Exception during message handling: nova.exception.InternalError: Unexpected vif_type=binding_failed
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 71, in wrapped
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server _emit_versioned_exception_notification(
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server raise self.value
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 63, in wrapped
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 184, in decorated_function
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. "
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server raise self.value
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 155, in decorated_function
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1439, in decorated_function
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 212, in decorated_function
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server raise self.value
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 201, in decorated_function
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6661, in unshelve_instance
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server do_unshelve_instance()
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 391, in inner
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6658, in do_unshelve_instance
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server self._unshelve_instance(
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6767, in _unshelve_instance
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server self._nil_out_instance_obj_host_and_node(instance)
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server raise self.value
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6741, in _unshelve_instance
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server self.driver.spawn(context, instance, image_meta,
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4329, in spawn
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server xml = self._get_guest_xml(context, instance, network_info,
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7300, in _get_guest_xml
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server conf = self._get_guest_config(instance, network_info, image_meta,
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6914, in _get_guest_config
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server config = self.vif_driver.get_config(
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 600, in get_config
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server raise exception.InternalError(_('Unexpected vif_type=%s') % vif_type)
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server nova.exception.InternalError: Unexpected vif_type=binding_failed
2022-11-26 18:06:42.668 582295 ERROR oslo_messaging.rpc.server
```
Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/e31df22c-facf-4532-86de-20fb295610ea/index.html
