periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby job is failing with 'tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received' error on all tempest tests
Following is the traceback observed within the tests:-
test:- neutron_tempest_plugin.api.admin.test_logging.LoggingTestJSON.test_list_supported_logging_types
failure:-
ft1.1: neutron_tempest_plugin.api.admin.test_logging.LoggingTestJSON.test_list_supported_logging_types[id-1af6cdab-0eb0-4e13-8027-d89cf1c7a87a]testtools.testresult.real._StringException: pythonlogging:'': {{{
2022-03-06 18:37:01,285 342048 INFO [tempest.lib.common.rest_client] Request (LoggingTestJSON:test_list_supported_logging_types): 201 POST http://192.168.24.3:5000/v3/auth/tokens 0.616s
2022-03-06 18:37:01,286 342048 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}
Body: <omitted>
Response - Headers: {'date': 'Sun, 06 Mar 2022 18:37:00 GMT', 'server': 'Apache', 'content-length': '4831', 'x-subject-token': '<omitted>', 'vary': 'X-Auth-Token', 'x-openstack-request-id': 'req-0ac9eb60-3505-4143-8716-080c423ed901', 'connection': 'close', 'content-type': 'application/json', 'status': '201', 'content-location': 'http://192.168.24.3:5000/v3/auth/tokens'}
Body: b'{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "6bde94a33cdd4e23a37ed18954c882dd", "name": "tempest-LoggingTestJSON-1231266580-project-admin", "password_expires_at": null}, "audit_ids": ["aLXLQKAqQPKz4XtEPJPXPQ"], "expires_at": "2022-03-06T19:37:01.000000Z", "issued_at": "2022-03-06T18:37:01.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "570ac593035d482c96a317f14dd8b0e4", "name": "tempest-LoggingTestJSON-1231266580"}, "is_domain": false, "roles": [{"id": "a5d7c13a634940308fd689eb6051bf6d", "name": "admin"}, {"id": "5ddf4d20318841099ccdf758deed11bd", "name": "reader"}, {"id": "f868d32a96a64e0c8f0812313d83c603", "name": "member"}], "catalog": [{"endpoints": [{"id": "4a9e236c4bc648d696b89b48730dec57", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8778/placement", "region": "regionOne"}, {"id": "a68482d8962b4ec2bea72ddfc1e26762", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:8778/placement", "region": "regionOne"}, {"id": "fe8ae0b634684f79b3a8863deedac305", "interface": "public", "region_id": "regionOne", "url": "http://192.168.24.3:8778/placement", "region": "regionOne"}], "id": "04b507583a1746ffae872b842055755e", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "1c2ee42fca4a4b8bb9c2f6264e029edb", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:9292", "region": "regionOne"}, {"id": "a8f87bc96ba2449698889abe1c0788d7", "interface": "public", "region_id": "regionOne", "url": "http://192.168.24.3:9292", "region": "regionOne"}, {"id": "f8785c9b892a4115bc9568332a44107d", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:9292", "region": "regionOne"}], "id": "12ae11712c8049a7920f2852c9c11c42", "type": "image", "name": "glance"}, {"endpoints": [{"id": "19d0f8bf5dfa4755a743f6308103a7ff", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8080/v1/AUTH_570ac593035d482c96a317f14dd8b0e4", "region": "regionOne"}, {"id": "9eb7473bd00f459aa1e565e05d9d31da", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:8080", "region": "regionOne"}, {"id": "eed97f3bab7445a5a0bf6bad029647b8", "interface": "public", "region_id": "regionOne", "url": "http://192.168.24.3:8080/v1/AUTH_570ac593035d482c96a317f14dd8b0e4", "region": "regionOne"}], "id": "2a2e0a6b529a42019485d483b6f5e3e8", "type": "object-store", "name": "swift"}, {"endpoints": [{"id": "4809d3ef7591415f99e51de8d4d331f9", "interface": "public", "region_id": "regionOne", "url": "http://192.168.24.3:8776/v3/570ac593035d482c96a317f14dd8b0e4", "region": "regionOne"}, {"id": "7b622e8f9f0b46b895608029b2b93579", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8776/v3/570ac593035d482c96a317f14dd8b0e4", "region": "regionOne"}, {"id": "816e5ef3dcd74fc1baf1b5139bd70672", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:8776/v3/570ac593035d482c96a317f14dd8b0e4", "region": "regionOne"}], "id": "47a522fa741842cf82cf515a86a25c6b", "type": "volumev3", "name": "cinderv3"}, {"endpoints": [{"id": "4228380747eb4fa9b2638887e946d434", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8774/v2.1", "region": "regionOne"}, {"id": "4e4b586e3c4b450e8e9f352fd1fcf3d2", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:8774/v2.1", "region": "regionOne"}, {"id": "b4238d6f40054259ab3081bedccfe6a0", "interface": "public", "region_id": "regionOne", "url": "http://192.168.24.3:8774/v2.1", "region": "regionOne"}], "id": "57b36f175e10404b82d61a9b9147d368", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "6974cc5aaed44b20abfdcb3cb604db9b", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:35357", "region": "regionOne"}, {"id": "78c2539e96b2435eb7492fd72dd7eec6", "interface": "public", "region_id": "regionOne", "url": "http://192.168.24.3:5000", "region": "regionOne"}, {"id": "a8fce8cf33b7465c96bc129c5fd1c880", "interfac
2022-03-06 18:37:01,288 342048 INFO [tempest.lib.common.rest_client] Request (LoggingTestJSON:test_list_supported_logging_types): 503 GET http://192.168.24.3:9696/v2.0/log/loggable-resources 0.001s
2022-03-06 18:37:01,288 342048 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
Body: None
Response - Headers: {'cache-control': 'no-cache', 'connection': 'close', 'content-type': 'text/html', 'status': '503', 'content-location': 'http://192.168.24.3:9696/v2.0/log/loggable-resources'}
Body: b'<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n'
}}}
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/admin/test_logging.py", line 69, in test_list_supported_logging_types
log_types = self.admin_client.list_loggable_resources()
File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 101, in _list
resp, body = self.get(uri)
File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 314, in get
return self.request('GET', url, extra_headers, headers)
File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 703, in request
self._error_checker(resp, resp_body)
File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 884, in _error_checker
resp=resp)
tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received
Details: 503
test:- neutron_tempest_plugin.api.admin.test_logging.LoggingTestJSON.test_log_lifecycle
failure:-
ft1.2: neutron_tempest_plugin.api.admin.test_logging.LoggingTestJSON.test_log_lifecycle[id-8d2e1ba5-455b-4519-a88e-e587002faba6]testtools.testresult.real._StringException: pythonlogging:'': {{{
2022-03-06 18:37:01,298 342048 INFO [tempest.lib.common.rest_client] Request (LoggingTestJSON:test_log_lifecycle): 503 POST http://192.168.24.3:9696/v2.0/log/logs 0.005s
2022-03-06 18:37:01,298 342048 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
Body: {"log": {"name": "tempest-test-log-2083917668", "description": "tempest-test-log-desc-1280640026", "resource_type": "security_group", "resource_id": null, "target_id": null, "event": "ALL", "enabled": true}}
Response - Headers: {'cache-control': 'no-cache', 'connection': 'close', 'content-type': 'text/html', 'status': '503', 'content-location': 'http://192.168.24.3:9696/v2.0/log/logs'}
Body: b'<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n'
}}}
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/admin/test_logging.py", line 34, in test_log_lifecycle
resource_type='security_group', enabled=True)
File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/base.py", line 1126, in create_log
body = cls.admin_client.create_log(**log_args)
File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 145, in _create
resp, body = self.post(uri, post_data)
File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 299, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 703, in request
self._error_checker(resp, resp_body)
File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 884, in _error_checker
resp=resp)
tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received
Details: 503
test:- neutron_tempest_plugin.api.admin.test_quotas_negative.QuotasAdminNegativeTestJSON.test_create_floatingip_when_quotas_is_full
failure:-
ft4.1: neutron_tempest_plugin.api.admin.test_quotas_negative.QuotasAdminNegativeTestJSON.test_create_floatingip_when_quotas_is_full[id-d00fe5bb-9db8-4e1a-9c31-490f52897e6f,negative]testtools.testresult.real._StringException: pythonlogging:'': {{{
2022-03-06 18:34:30,891 342047 INFO [tempest.lib.common.rest_client] Request (QuotasAdminNegativeTestJSON:test_create_floatingip_when_quotas_is_full): 201 POST http://192.168.24.3:5000/v3/auth/tokens 0.332s
2022-03-06 18:34:30,892 342047 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}
Body: <omitted>
Response - Headers: {'date': 'Sun, 06 Mar 2022 18:34:30 GMT', 'server': 'Apache', 'content-length': '4853', 'x-subject-token': '<omitted>', 'vary': 'X-Auth-Token', 'x-openstack-request-id': 'req-edee05cd-3a05-4516-9043-53a76f5e73d2', 'connection': 'close', 'content-type': 'application/json', 'status': '201', 'content-location': 'http://192.168.24.3:5000/v3/auth/tokens'}
Body: b'{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "241e533cfba1433486b1ad2c7c1cc7ca", "name": "tempest-QuotasAdminNegativeTestJSON-320254474-project-admin", "password_expires_at": null}, "audit_ids": ["oMNaUcHjRrWEwtn0xWgOsQ"], "expires_at": "2022-03-06T19:34:30.000000Z", "issued_at": "2022-03-06T18:34:30.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "0003d3c51fd54cc78dfe4d2d9fe88a33", "name": "tempest-QuotasAdminNegativeTestJSON-320254474"}, "is_domain": false, "roles": [{"id": "a5d7c13a634940308fd689eb6051bf6d", "name": "admin"}, {"id": "5ddf4d20318841099ccdf758deed11bd", "name": "reader"}, {"id": "f868d32a96a64e0c8f0812313d83c603", "name": "member"}], "catalog": [{"endpoints": [{"id": "4a9e236c4bc648d696b89b48730dec57", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8778/placement", "region": "regionOne"}, {"id": "a68482d8962b4ec2bea72ddfc1e26762", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:8778/placement", "region": "regionOne"}, {"id": "fe8ae0b634684f79b3a8863deedac305", "interface": "public", "region_id": "regionOne", "url": "http://192.168.24.3:8778/placement", "region": "regionOne"}], "id": "04b507583a1746ffae872b842055755e", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "1c2ee42fca4a4b8bb9c2f6264e029edb", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:9292", "region": "regionOne"}, {"id": "a8f87bc96ba2449698889abe1c0788d7", "interface": "public", "region_id": "regionOne", "url": "http://192.168.24.3:9292", "region": "regionOne"}, {"id": "f8785c9b892a4115bc9568332a44107d", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:9292", "region": "regionOne"}], "id": "12ae11712c8049a7920f2852c9c11c42", "type": "image", "name": "glance"}, {"endpoints": [{"id": "19d0f8bf5dfa4755a743f6308103a7ff", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8080/v1/AUTH_0003d3c51fd54cc78dfe4d2d9fe88a33", "region": "regionOne"}, {"id": "9eb7473bd00f459aa1e565e05d9d31da", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:8080", "region": "regionOne"}, {"id": "eed97f3bab7445a5a0bf6bad029647b8", "interface": "public", "region_id": "regionOne", "url": "http://192.168.24.3:8080/v1/AUTH_0003d3c51fd54cc78dfe4d2d9fe88a33", "region": "regionOne"}], "id": "2a2e0a6b529a42019485d483b6f5e3e8", "type": "object-store", "name": "swift"}, {"endpoints": [{"id": "4809d3ef7591415f99e51de8d4d331f9", "interface": "public", "region_id": "regionOne", "url": "http://192.168.24.3:8776/v3/0003d3c51fd54cc78dfe4d2d9fe88a33", "region": "regionOne"}, {"id": "7b622e8f9f0b46b895608029b2b93579", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8776/v3/0003d3c51fd54cc78dfe4d2d9fe88a33", "region": "regionOne"}, {"id": "816e5ef3dcd74fc1baf1b5139bd70672", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:8776/v3/0003d3c51fd54cc78dfe4d2d9fe88a33", "region": "regionOne"}], "id": "47a522fa741842cf82cf515a86a25c6b", "type": "volumev3", "name": "cinderv3"}, {"endpoints": [{"id": "4228380747eb4fa9b2638887e946d434", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8774/v2.1", "region": "regionOne"}, {"id": "4e4b586e3c4b450e8e9f352fd1fcf3d2", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:8774/v2.1", "region": "regionOne"}, {"id": "b4238d6f40054259ab3081bedccfe6a0", "interface": "public", "region_id": "regionOne", "url": "http://192.168.24.3:8774/v2.1", "region": "regionOne"}], "id": "57b36f175e10404b82d61a9b9147d368", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "6974cc5aaed44b20abfdcb3cb604db9b", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:35357", "region": "regionOne"}, {"id": "78c2539e96b2435eb7492fd72dd7eec6", "interface": "public", "region_id": "regionOne", "url": "http://192.168.24.3:5000", "region": "regionOne"}, {"id": "a8fce8cf33b7465c96bc12
2022-03-06 18:34:31,387 342047 INFO [tempest.lib.common.rest_client] Request (QuotasAdminNegativeTestJSON:test_create_floatingip_when_quotas_is_full): 201 POST http://192.168.24.3:35357/v3/projects 0.494s
2022-03-06 18:34:31,387 342047 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
Body: {"project": {"description": "tempest-desc_-1742572824", "name": "tempest-test_project_-1303511082"}}
Response - Headers: {'date': 'Sun, 06 Mar 2022 18:34:30 GMT', 'server': 'Apache', 'content-length': '348', 'vary': 'X-Auth-Token', 'x-openstack-request-id': 'req-a0f44f78-0651-4e2d-afca-047b9323155b', 'connection': 'close', 'content-type': 'application/json', 'status': '201', 'content-location': 'http://192.168.24.3:35357/v3/projects'}
Body: b'{"project": {"id": "faad9129b56948d486863de337d89709", "name": "tempest-test_project_-1303511082", "domain_id": "default", "description": "tempest-desc_-1742572824", "enabled": true, "parent_id": "default", "is_domain": false, "tags": [], "options": {}, "links": {"self": "http://192.168.24.3:35357/v3/projects/faad9129b56948d486863de337d89709"}}}\n'
2022-03-06 18:34:31,390 342047 INFO [tempest.lib.common.rest_client] Request (QuotasAdminNegativeTestJSON:test_create_floatingip_when_quotas_is_full): 503 GET http://192.168.24.3:9696/v2.0/security-groups?tenant_id=faad9129b56948d486863de337d89709 0.002s
2022-03-06 18:34:31,390 342047 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
Body: None
Response - Headers: {'cache-control': 'no-cache', 'connection': 'close', 'content-type': 'text/html', 'status': '503', 'content-location': 'http://192.168.24.3:9696/v2.0/security-groups?tenant_id=faad9129b56948d486863de337d89709'}
Body: b'<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n'
}}}
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
return func(*func_args, **func_kwargs)
File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/admin/test_quotas_negative.py", line 151, in test_create_floatingip_when_quotas_is_full
tenant_id = self.create_project()['id']
File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/base.py", line 845, in create_project
tenant_id=project['id'])['security_groups']
File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 929, in list_security_groups
resp, body = self.get(uri)
File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 314, in get
return self.request('GET', url, extra_headers, headers)
File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 703, in request
self._error_checker(resp, resp_body)
File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 884, in _error_checker
resp=resp)
tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received
Details: 503
For more details please refer the following links below:-
1. https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby/5aa2d3d/logs/undercloud/var/log/tempest/stestr_results.html.gz
2. https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby/5f880b7/logs/undercloud/var/log/tempest/stestr_results.html.gz
The failures here seems because of cloud instability. We are checking the job health for the good run. Hence, for now, closing this bug