running on sunbeam latest/candidate, rev 530, sunbeam failed to run the tempest test "test_update_access_server_address" because it failed to get a machine to deploy in 300 seconds. The traceback from the temptest test is below. There were 150 other tempest tests that ran and all succeeded in getting a system, so I wonder if there is some load issue against the deployment when requesting a bunch of systems at once.
2024-06-18 04:24:47,153 19737 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
Body: None
Response - Headers: {'content-length': '111', 'content-type': 'application/json; charset=UTF-8', 'date': 'Tue, 18 Jun 2024 04:24:47 GMT', 'openstack-api-version': 'compute 2.1', 'server': 'Apache/2.4.52 (Ubuntu)', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-compute-request-id': 'req-62c0f03c-9690-4b35-9701-8a1bd13ac575', 'x-openstack-nova-api-version': '2.1', 'x-openstack-request-id': 'req-62c0f03c-9690-4b35-9701-8a1bd13ac575', 'connection': 'close', 'status': '404', 'content-location': 'http://10.246.167.166:80/openstack-nova/v2.1/servers/554927c7-c44a-498b-9a48-71dfc069367d'}
Body: b'{"itemNotFound": {"code": 404, "message": "Instance 554927c7-c44a-498b-9a48-71dfc069367d could not be found."}}'
}}}
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/tempest/api/compute/servers/test_servers.py", line 130, in test_update_access_server_address
server = self.create_test_server(wait_until='ACTIVE')
File "/home/ubuntu/.local/lib/python3.10/site-packages/tempest/api/compute/base.py", line 243, in create_test_server
body, servers = compute.create_test_server(
File "/home/ubuntu/.local/lib/python3.10/site-packages/tempest/common/compute.py", line 340, in create_test_server
with excutils.save_and_reraise_exception():
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
self.force_reraise()
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
raise self.value
File "/home/ubuntu/.local/lib/python3.10/site-packages/tempest/common/compute.py", line 323, in create_test_server
server = waiters.wait_for_server_status(
File "/home/ubuntu/.local/lib/python3.10/site-packages/tempest/common/waiters.py", line 101, in wait_for_server_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (ServersTestJSON:test_update_access_server_address) Server 554927c7-c44a-498b-9a48-71dfc069367d failed to reach ACTIVE status and task state "None" within the required time (300 s). Request ID of server operation performed before checking the server status req-8070d489-b3e9-41cd-89f3-44f074374a34. Current status: BUILD. Current task state: scheduling.
grepping for the request id in the deployment, i find some notes that it failed to connect to mysql because the name or service is not known.
generated/sunbeam/logs-openstack-nova-0.txt:2024-06-18T04:19:50.425Z [nova-conductor] 2024-06-18 04:19:50.423 37 ERROR nova.conductor.manager [None req-8070d489-b3e9-41cd-89f3-44f074374a34 cbdcb53205864a07b61805382cf1cc8f 5e4250861e8847afbb6a1b35d36cb352 - - 5bc46576b9ab482a87e0fe2918ea1b3c 5bc46576b9ab482a87e0fe2918ea1b3c] Failed to schedule instances: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'nova-api-mysql-router.openstack.svc.cluster.local' ([Errno -2] Name or service not known)")
generated/sunbeam/logs-openstack-nova-0.txt:2024-06-18T04:19:50.431Z [nova-conductor] 2024-06-18 04:19:50.429 37 ERROR oslo_messaging.rpc.server [None req-8070d489-b3e9-41cd-89f3-44f074374a34 cbdcb53205864a07b61805382cf1cc8f 5e4250861e8847afbb6a1b35d36cb352 - - 5bc46576b9ab482a87e0fe2918ea1b3c 5bc46576b9ab482a87e0fe2918ea1b3c] Exception during message handling: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'nova-api-mysql-router.openstack.svc.cluster.local' ([Errno -2] Name or service not known)")
the testrun can be found at:
https://solutions.qa.canonical.com/testruns/9ee210fd-8b31-457c-a990-cefc7d5cd3fb/
with tempest logs at:
https://oil-jenkins.canonical.com/artifacts/9ee210fd-8b31-457c-a990-cefc7d5cd3fb/generated/generated/sunbeam/results.html
and other relevant logs:
https://oil-jenkins.canonical.com/artifacts/9ee210fd-8b31-457c-a990-cefc7d5cd3fb/index.html
