periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-wallaby has started failing on multiple tempest tests from 13th June, 2022 [2].
Below is the traceback [1]:
```
{1} tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest.test_server_create_no_allocate [12.799220s] ... FAILED
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/api/compute/admin/test_auto_allocate_network.py", line 151, in test_server_create_no_allocate
server = self.create_test_server(networks='none', wait_until='ACTIVE')
File "/usr/lib/python3.9/site-packages/tempest/api/compute/base.py", line 280, in create_test_server
body, servers = compute.create_test_server(
File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 353, in create_test_server
LOG.exception('Server %s failed to delete in time',
File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
self.force_reraise()
File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
raise self.value
File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 324, in create_test_server
_setup_validation_fip(
File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 99, in _setup_validation_fip
raise ValueError('Unable to setup floating IP for validation: '
ValueError: Unable to setup floating IP for validation: port not found on tenant network
```
Below is the list of failing tempest tests [3]
```
tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest.test_server_create_no_allocate[id-5eb7b8fa-9c23-47a2-9d7d-02ed5809dd34]
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az[id-96be03c7-570d-409c-90f8-e4db3c646996]
setUpClass (tempest.api.compute.admin.test_simple_tenant_usage.TenantUsagesTestJSON)
tempest.api.compute.admin.test_servers.ServersAdmin275Test.test_rebuild_update_server_275[id-bf2b4a00-73a3-4d53-81fa-acbcd97d6339]
tempest.api.compute.floating_ips.test_floating_ips_actions.FloatingIPsAssociationTestJSON.test_associate_already_associated_floating_ip[id-6edef4b2-aaf1-4abc-bbe3-993e2561e0fe]
tempest.api.compute.admin.test_delete_server.DeleteServersAdminTestJSON.test_admin_delete_servers_of_others[id-73177903-6737-4f27-a60c-379e8ae8cf48]
tempest.api.compute.admin.test_delete_server.DeleteServersAdminTestJSON.test_delete_server_while_in_error_state[id-99774678-e072-49d1-9d2a-49a59bc56063]
tempest.api.compute.floating_ips.test_floating_ips_actions_negative.FloatingIPsAssociationNegativeTestJSON.test_associate_ip_to_server_with_floating_ip[id-58a80596-ffb2-11e6-9393-fa163e4fa634,negative]
tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive[id-777e468f-17ca-4da4-b93d-b7dbf56c0494]
tempest.api.compute.admin.test_server_diagnostics.ServerDiagnosticsTest.test_get_server_diagnostics[id-31ff3486-b8a0-4f56-a6c0-aab460531db3]
tempest.api.compute.admin.test_server_diagnostics.ServerDiagnosticsV248Test.test_get_server_diagnostics[id-64d0d48c-dff1-11e6-bf01-fe55135034f3]
tempest.api.compute.admin.test_server_diagnostics_negative.ServerDiagnosticsNegativeTest.test_get_server_diagnostics_by_non_admin[id-e84e2234-60d2-42fa-8b30-e2d3049724ac,negative]
setUpClass (tempest.api.compute.admin.test_servers.ServersAdminTestJSON)
setUpClass (tempest.api.compute.admin.test_servers_negative.ServersAdminNegativeTestJSON)
tempest.api.compute.images.test_images_negative.ImagesNegativeTestJSON.test_create_image_from_deleted_server[id-6cd5a89d-5b47-46a7-93bc-3916f0d84973,negative]
setUpClass (tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON)
setUpClass (tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON)
tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON.test_delete_server_while_in_building_state[id-9e6e0c87-3352-42f7-9faf-5d6210dbd159]
tempest.api.compute.servers.test_create_server_multi_nic.ServersTestMultiNic.test_verify_duplicate_network_nics[id-1678d144-ed74-43f8-8e57-ab10dbf9b3c2]
tempest.api.compute.servers.test_create_server_multi_nic.ServersTestMultiNic.test_verify_multiple_nics_order[id-0578d144-ed74-43f8-8e57-ab10dbf9b3c2]
tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest.test_tagged_attachment[id-3e41c782-2a89-4922-a9d2-9a188c4e7c7c,image,network,volume]
tempest.api.compute.servers.test_multiple_create.MultipleCreateTestJSON.test_multiple_create_with_reservation_return[id-864777fb-2f1e-44e3-b5b9-3eb6fa84f2f7]
setUpClass (tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON)
setUpClass (tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON)
tempest.api.compute.servers.test_servers_microversions.ServerShowV254Test.test_rebuild_server[id-09170a98-4940-4637-add7-1a35121f1a5a]
tempest.api.compute.servers.test_servers_microversions.ServerShowV257Test.test_rebuild_server[id-803df848-080a-4261-8f11-b020cd9b6f60]
tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume[id-52e9045a-e90d-4c0d-9087-79d657faffff,slow]
tempest.api.compute.volumes.test_attach_volume_negative.AttachVolumeNegativeTest.test_attach_attached_volume_to_different_server[id-ee37a796-2afb-11e7-bc0f-fa163e65f5ce,negative]
tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_list_get_volume_attachments[id-7fa563fe-f0f7-43eb-9e22-a1ece036b513]
tempest.api.compute.servers.test_multiple_create_negative.MultipleCreateNegativeTestJSON.test_max_count_less_than_min_count[id-476da616-f1ef-4271-a9b1-b9fc87727cdf,negative]
tempest.api.compute.servers.test_multiple_create_negative.MultipleCreateNegativeTestJSON.test_max_count_non_integer[id-9c5698d1-d7af-4c80-b971-9d403135eea2,negative]
tempest.api.compute.servers.test_multiple_create_negative.MultipleCreateNegativeTestJSON.test_min_count_non_integer[id-999aa722-d624-4423-b813-0d1ac9884d7a,negative]
tempest.api.compute.servers.test_server_actions.ServersAaction247Test.test_create_backup[id-252a4bdd-6366-4dae-9994-8c30aa660f23,image]
tempest.api.compute.volumes.test_attach_volume_negative.AttachVolumeNegativeTest.test_attach_attached_volume_to_same_server[id-aab919e2-d992-4cbb-a4ed-745c2475398c,negative]
tempest.api.compute.volumes.test_attach_volume_negative.AttachVolumeNegativeTest.test_delete_attached_volume[id-a313b5cd-fbd0-49cc-94de-870e99f763c7,negative]
setUpClass (tempest.api.compute.servers.test_server_rescue_negative.ServerRescueNegativeTestJSON)
tempest.api.volume.admin.test_volumes_actions.VolumesActionsTest.test_force_detach_volume[compute,id-d38285d9-929d-478f-96a5-00e66a115b81]
tempest.api.volume.test_volumes_actions.VolumesActionsTest.test_attach_detach_volume_to_instance[compute,id-fff42874-7db5-4487-a8e1-ddda5fb5288d,smoke]
tempest.api.volume.test_volumes_actions.VolumesActionsTest.test_get_volume_attachment[compute,id-9516a2c8-9135-488c-8dd6-5677a7e5f371]
tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_snapshot_create_delete_with_volume_in_use[compute,id-8567b54c-4455-446d-a1cf-651ddeaa3ff2]
tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_snapshot_create_offline_delete_online[compute,id-5210a1de-85a0-11e6-bb21-641c676a5d61]
tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_attach_volumes_with_nonexistent_volume_id[compute,id-f5e56b0a-5d02-43c1-a2a7-c9b792c2e3f6,negative]
tempest.api.compute.servers.test_server_rescue.ServerRescueTestJSONUnderV235.test_rescued_vm_associate_dissociate_floating_ip[id-4842e0cf-e87d-4d9d-b61f-f4791da3cacc]
tempest.api.compute.servers.test_servers.ServerShowV247Test.test_show_server[id-88b0bdb2-494c-11e7-a919-92ebcb67fe33]
tempest.api.compute.servers.test_servers.ServerShowV247Test.test_update_rebuild_list_server[id-8de397c2-57d0-4b90-aa30-e5d668f21a8b]
tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON.test_attach_volume_shelved_or_offload_server[id-13a940b6-3474-4c3c-b03f-29b89112bfee,slow]
tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON.test_detach_volume_shelved_or_offload_server[id-b54e86dd-a070-49c4-9c07-59ae6dae15aa,slow]
```
It is failing in tempest component line. It is a tracker bug. I will add more information based on the investigation.
I have filed another bug https://bugs.launchpad.net/tripleo/+bug/1978553 which contains other errors from this job.
Links:
[1]. https://logserver.rdoproject.org/54/36254/116/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-wallaby/79eac6e/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
[2]. https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-wallaby&skip=0
[3]. https://logserver.rdoproject.org/54/36254/116/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-wallaby/79eac6e/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
https:/
```
Uping tempest-31.0.0 to test with wait_until_
https:/
Depends-On: https:/
Depends-On: https:/
Depends-On: https:/
```
Similar tests are failing here during backport patches.
https:/