On Fs001 CentOS Stream 9 wallaby, Multiple compute server tempest tests are failing with following error [1]:
```
{1} tempest.api.compute.images.test_images.ImagesTestJSON.test_create_image_from_paused_server [335.060967s] ... FAILED
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/api/compute/images/test_images.py", line 99, in test_create_image_from_paused_server
server = self.create_test_server(wait_until='ACTIVE')
File "/usr/lib/python3.9/site-packages/tempest/api/compute/base.py", line 270, in create_test_server
body, servers = compute.create_test_server(
File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 267, in create_test_server LOG.exception('Server %s failed to delete in time',
File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ self.force_reraise()
File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
raise self.value
File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 237, in create_test_server waiters.wait_for_server_status(
File "/usr/lib/python3.9/site-packages/tempest/common/waiters.py", line 100, in wait_for_server_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (ImagesTestJSON:test_create_image_from_paused_server) Server 6d1d8906-46fd-42ad-8b4e-0f89adb25ed1 failed to reach ACTIVE status and task state "None" within the required time (300 s). Server boot request ID: req-4930f047-7f5f-4d08-9ebb-8ac99b29ad7b. Current status: BUILD. Current task state: spawning.
```
Below is the list of other tempest tests failing on the same job.[2]
```
tempest.api.compute.images.test_images.ImagesTestJSON.test_create_image_from_paused_server[id-71bcb732-0261-11e7-9086-fa163e4fa634]
tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive[id-777e468f-17ca-4da4-b93d-b7dbf56c0494]
tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON.test_delete_server_while_in_attached_volume[id-d0f3f0d6-d9b6-4a32-8da4-23015dcab23c,volume]
tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesV270Test.test_create_get_list_interfaces[id-2853f095-8277-4067-92bd-9f10bd4f8e0c,network]
tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON.test_delete_server_while_in_shelved_state[id-bb0cb402-09dd-4947-b6e5-5e7e1cfa61ad]
setUpClass (tempest.api.compute.images.test_images_oneserver_negative.ImagesOneServerNegativeTestJSON)
tempest.api.compute.servers.test_device_tagging.TaggedBootDevicesTest_v242.test_tagged_boot_devices[id-a2e65a6c-66f1-4442-aaa8-498c31778d96,image,network,slow,volume]
tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON.test_delete_server_while_in_suspended_state[id-1f82ebd3-8253-4f4e-b93f-de9b7df56d8b]
tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_create_list_show_delete_interfaces_by_network_port[id-73fe8f02-590d-4bf1-b184-e9ca81065051,network]
setUpClass (tempest.api.compute.servers.test_server_rescue.ServerRescueTestJSONUnderV235)
```
On Fs001 CentOS Stream 9 wallaby, Multiple compute server tempest tests are failing with following error [1]: api.compute. images. test_images. ImagesTestJSON. test_create_ image_from_ paused_ server [335.060967s] ... FAILED
```
{1} tempest.
Captured traceback: python3. 9/site- packages/ tempest/ api/compute/ images/ test_images. py", line 99, in test_create_ image_from_ paused_ server test_server( wait_until= 'ACTIVE' ) python3. 9/site- packages/ tempest/ api/compute/ base.py" , line 270, in create_test_server create_ test_server( python3. 9/site- packages/ tempest/ common/ compute. py", line 267, in create_test_server
LOG.exception( 'Server %s failed to delete in time', python3. 9/site- packages/ oslo_utils/ excutils. py", line 227, in __exit__
self.force_ reraise( ) python3. 9/site- packages/ oslo_utils/ excutils. py", line 200, in force_reraise python3. 9/site- packages/ tempest/ common/ compute. py", line 237, in create_test_server
waiters. wait_for_ server_ status( python3. 9/site- packages/ tempest/ common/ waiters. py", line 100, in wait_for_ server_ status TimeoutExceptio n(message) lib.exceptions. TimeoutExceptio n: Request timed out :test_create_ image_from_ paused_ server) Server 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1 failed to reach ACTIVE status and task state "None" within the required time (300 s). Server boot request ID: req-4930f047- 7f5f-4d08- 9ebb-8ac99b29ad 7b. Current status: BUILD. Current task state: spawning.
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/
server = self.create_
File "/usr/lib/
body, servers = compute.
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
raise self.value
File "/usr/lib/
File "/usr/lib/
raise lib_exc.
tempest.
Details: (ImagesTestJSON
```
Below is the list of other tempest tests failing on the same job.[2] api.compute. images. test_images. ImagesTestJSON. test_create_ image_from_ paused_ server[ id-71bcb732- 0261-11e7- 9086-fa163e4fa6 34] api.compute. admin.test_ volume. AttachSCSIVolum eTestJSON. test_attach_ scsi_disk_ with_config_ drive[id- 777e468f- 17ca-4da4- b93d-b7dbf56c04 94] api.compute. servers. test_delete_ server. DeleteServersTe stJSON. test_delete_ server_ while_in_ attached_ volume[ id-d0f3f0d6- d9b6-4a32- 8da4-23015dcab2 3c,volume] api.compute. servers. test_attach_ interfaces. AttachInterface sV270Test. test_create_ get_list_ interfaces[ id-2853f095- 8277-4067- 92bd-9f10bd4f8e 0c,network] api.compute. servers. test_delete_ server. DeleteServersTe stJSON. test_delete_ server_ while_in_ shelved_ state[id- bb0cb402- 09dd-4947- b6e5-5e7e1cfa61 ad] api.compute. images. test_images_ oneserver_ negative. ImagesOneServer NegativeTestJSO N) api.compute. servers. test_device_ tagging. TaggedBootDevic esTest_ v242.test_ tagged_ boot_devices[ id-a2e65a6c- 66f1-4442- aaa8-498c31778d 96,image, network, slow,volume] api.compute. servers. test_delete_ server. DeleteServersTe stJSON. test_delete_ server_ while_in_ suspended_ state[id- 1f82ebd3- 8253-4f4e- b93f-de9b7df56d 8b] api.compute. servers. test_attach_ interfaces. AttachInterface sTestJSON. test_create_ list_show_ delete_ interfaces_ by_network_ port[id- 73fe8f02- 590d-4bf1- b184-e9ca810650 51,network] api.compute. servers. test_server_ rescue. ServerRescueTes tJSONUnderV235)
```
tempest.
tempest.
tempest.
tempest.
tempest.
setUpClass (tempest.
tempest.
tempest.
tempest.
setUpClass (tempest.
```
Here is the traceback from nova-compute logs [3], manager [req-4930f047- 7f5f-4d08- 9ebb-8ac99b29ad 7b d5ea6c724785473 b8ea1104d70fb0d 14 64c7d31d84284a2 8bc9aaa4eaad2b9 fb - default default] [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] Instance failed to spawn: nova.exception. VirtualInterfac eCreateExceptio n: Virtual Interface creation failed manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] Traceback (most recent call last): manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] File "/usr/lib/ python3. 9/site- packages/ nova/virt/ libvirt/ driver. py", line 7231, in _create_ guest_with_ network manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] guest = self._create_guest( manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] File "/usr/lib64/ python3. 9/contextlib. py", line 126, in __exit__ manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] next(self.gen) manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] File "/usr/lib/ python3. 9/site- packages/ nova/compute/ manager. py", line 479, in wait_for_ instance_ event manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] actual_event = event.wait() manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] File "/usr/lib/ python3. 9/site- packages/ eventlet/ event.py" , line 125, in wait manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] result = hub.switch() manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] File "/usr/lib/ python3. 9/site- packages/ eventlet/ hubs/hub. py", line 313, in switch manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] return self.greenlet. switch( ) manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] eventlet. timeout. Timeout: 300 seconds manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] During handling of the above exception, another exception occurred: manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] Traceback (most recent call last): manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] File "/usr/lib/ python3. 9/site- packages/ nova/compute/ manager. py", line 2640, in _build_resources manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] yield resources manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] File "/usr/lib/ python3. 9/site- packages/ nova/compute/ manager. py", line 2409, in _build_ and_run_ instance manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] self.driver. spawn(context, instance, image_meta, manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] File "/usr/lib/ python3. 9/site- packages/ nova/virt/ libvirt/ driver. py", line 4193, in spawn manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] self._create_ guest_with_ network( manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] File "/usr/lib/ python3. 9/site- packages/ nova/virt/ libvirt/ driver. py", line 7257, in _create_ guest_with_ network manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] raise exception. VirtualInterfac eCreateExceptio n() manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1] nova.exception. VirtualInterfac eCreateExceptio n: Virtual Interface creation failed manager [instance: 6d1d8906- 46fd-42ad- 8b4e-0f89adb25e d1]
```
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
2022-03-15 09:05:39.011 2 ERROR nova.compute.
```
This job https:/ /review. rdoproject. org/zuul/ builds? job_name= periodic- tripleo- ci-centos- 9-ovb-3ctlr_ 1comp-featurese t001-wallaby is broken from 13th Mar, 2021 and earlier /bugs.launchpad .net/tripleo/ +bug/1960310 is also seen on this.
https:/
In each run, tempest tests failures are different. So filing this bug from the last run for debugging.
Logs:
[1]. https:/ /logserver. rdoproject. org/17/ 40517/1/ check/periodic- tripleo- ci-centos- 9-ovb-3ctlr_ 1comp-featurese t001-wallaby/ 94e16ac/ logs/undercloud /var/log/ tempest/ tempest_ run.log. txt.gz
[2]. https:/ /logserver. rdoproject. org/17/ 40517/1/ check/periodic- tripleo- ci-centos- 9-ovb-3ctlr_ 1comp-featurese t001-wallaby/ 94e16ac/ logs/undercloud /var/log/ tempest/ failing_ tests.log. txt.gz
[3]. https:/ /logserver. rdoproject. org/17/ 40517/1/ check/periodic- tripleo- ci-centos- 9-ovb-3ctlr_ 1comp-featurese t001-wallaby/ 94e16ac/ logs/overcloud- novacompute- 0/var/log/ containers/ nova/nova- compute. log.1.gz