Description:-
Master Scenario002 is failing on Barbican related tempest test - tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received , Details: 503
Issue started on 13th May:-
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario002-standalone-master
log snippet:-
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario002-standalone-master/070da93/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
failing tests:-
~~~
barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningSnapshotTest.test_signed_image_upload_boot_snapshot[compute,id-f0603dfd-8b2c-44e2-8b0f-d65c87aab257,image]
barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningTest.test_signed_image_upload_and_boot[compute,id-4343df3c-5553-40ea-8705-0cce73b297a9,image]
~~~
traceback:-
~~~
==============================
Failed 2 tests - output below:
==============================
barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningSnapshotTest.test_signed_image_upload_boot_snapshot[compute,id-f0603dfd-8b2c-44e2-8b0f-d65c87aab257,image]
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/barbican_tempest_plugin/tests/scenario/barbican_manager.py", line 78, in setUp
self.signing_cert_uuid = self._store_cert(
File "/usr/lib/python3.9/site-packages/barbican_tempest_plugin/tests/scenario/barbican_manager.py", line 167, in _store_cert
result = self.secret_client.create_secret(
File "/usr/lib/python3.9/site-packages/barbican_tempest_plugin/services/key_manager/json/secret_client.py", line 44, in create_secret
resp, body = self.post("v1/secrets", body)
File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 299, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 703, in request
self._error_checker(resp, resp_body)
File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 883, in _error_checker
raise exceptions.UnexpectedResponseCode(str(resp.status),
tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received
Details: 503
Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
2022-05-16 01:42:07,963 214690 WARNING [tempest.api.compute.api_microversion_fixture] APIMicroversionFixture class is deprecated and moved to tempest.lib.common.api_microversion_fixture.APIMicroversionFixture. It will be removed in Z cycle.
2022-05-16 01:42:08,597 214690 DEBUG [barbican_tempest_plugin.tests.scenario.barbican_manager] Uploading certificate to barbican
2022-05-16 01:42:08,602 214690 INFO [tempest.lib.common.rest_client] Request (ImageSigningSnapshotTest:setUp): 503 POST http://192.168.24.3:9311/v1/secrets 0.004s
2022-05-16 01:42:08,602 214690 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
Body: {"expiration": "2022-05-21T01:42:08.597707", "algorithm": "rsa", "secret_type": "certificate", "payload_content_type": "application/octet-stream", "payload_content_encoding": "base64", "payload": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVkVENDQXQyZ0F3SUJBZ0lVV3hKQjRzVFZVaVU5V2h3T1ZDUU9kbUpPMUJvd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lqRUxNQWtHQTFVRUJoTUNWVk14Q3pBSkJnTlZCQWdNQWtOQk1SWXdGQVlEVlFRSERBMVRZVzRnUm5KaApibU5wYzJOdk1STXdFUVlEVlFRS0RBcE5lU0JEYjIxd1lXNTVNUmt3RndZRFZRUUREQkJVWlhOMElFTmxjblJwClptbGpZWFJsTUI0WERUSXlNRFV4TmpBeE5ESXdPRm9YRFRJeU1EVXlOakF4TkRJd09Gb3dZakVMTUFrR0ExVUUKQmhNQ1ZWTXhDekFKQmdOVkJBZ01Ba05CTVJZd0ZBWURWUVFIREExVFlXNGdSbkpoYm1OcGMyTnZNUk13RVFZRApWUVFLREFwTmVTQkRiMjF3WVc1NU1Sa3dGd1lEVlFRRERCQlVaWE4wSUVObGNuUnBabWxqWVhSbE1JSUJvakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVk4QU1JSUJpZ0tDQVlFQXkvdWxnT3RpSFZ6SkViQVRjVVUzNW00WHpRNlkKdDRFY2ZtRS9YOWUxcW1Va1R2cTBpTjQvSmlVUXJXUCtIbjRFNFRLbU52TytJS05qSlRsWEpsVWlnSFhPdGdGKwovNWNXUkM0dDhXR3c1TWNyM3BNM2xnMmNmS0lNd0FkWkFsbThDMWd2emhKSG9CSnQwdXM3V1hwL2kxOG0zQ2tFCkMvU05QUFBCWHJxUVE1UmM4RkcvZTFGWWkrR3lCM3E3UWdwamVValBkcWpjWDRqdTl3eDBVdWZwM2o5aXozd0sKUVN4VFlqMVRpTXVxYkJ5c2prYUdCRE42Nmp1WFQ2VlR0cXpxWEdJTlhuZzVzT1BvdzREZkpoVXlVdE1nUktaKwo5SHNEMGdPbFRSVzE2UWtBdmFNT0ttTk52cUpQK0dJOVIzb1k1WUNWK3ZSczAxaTY2YVprY2U0RlFkVmo4eUVrCmplcHNaRnhnbUI1dmZ5TlE0OUd5NDBGbm16VzF3b0VCaEhNUWtJU2hLZGNOZ2huRTBVZ0NudTRKRmdrVklYR1IKVXU0SUM4V21ZS3EreXhnUmVPa2IxY3hRRWRXd3A3dCs5eVVoNHcrQi82dUhMdzhLZjNDYzY2SkJYRE5IRnlKMwpZMzVPaG1XVGYrOTVTZ21JaTZiQVFsaUpWUVlzK3hJeXNvTnJBZ01CQUFHakl6QWhNQklHQTFVZEV3RUIvd1FJCk1BWUJBZjhDQVFFd0N3WURWUjBQQkFRREFnTEVNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJnUUJGWnNmLzJHbGQKeXYrdW9OeXZqVlFhMkwyb2JUUkVubWVOMkFCb1F0VFpPaTJHSXVaenYwSWRzN3AxMDlPU25vTmlrS1psVTNFMwp6cGs1L3dvOU92SldTdDlpTHFLbmR4bXM5RmNtTXQ0K1VMZS94SnlLRmtuNXQ2UTFOOHJLNmZ5ZDY0eW8ycWQyCi9zLzRHdnZYNkE1bW1XU1hkTFBzeWdTeEhtcy9kOUw1ZGdVTVlwc1cvTnRGVDRkMEhSWVRsYldtWXV3UTNtU3EKaDVqSFpzWlNLOU5nSzA5dXhxL2RxSEV5MWtqcS9pM2J2OWFpVVo1b1d0amQ5V1hZQTNZR3V2VkdnZFRLZWRQTgpwRk16NHI3czA2YjNJelREMFYwdVlQT0xiTzVKaHdRaXI5OFpUTGtXWDg1YlJxMDhWNGtGWk5LK0hGNHExRmlBClZNaUJnOW9ndUc3MlR3Q09aemNhQkxqcFZUTXpmQ3Zvb2ZwRUR0M2RKc3pzZGw4VGVFRFJOREhEdGlHdWZIY1oKTXZXZ0lEaEtjTHpxSllabS9yUmpZQXJteWNqaU5uTW00bTZyTTdENVA5YVNWWXF3SWEvR0VySVpRWXhvMHdIZwpLWWR3UC92S1RZNXBuUm1vWXh4Z0hzaURKSVZyZjNjVHZ2MnRjZ1RMN0pFb2VwYkdsR00vK3dZPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==", "name": "tempest-tempest-sec-2075813135"}
Response - Headers: {'content-length': '107', 'cache-control': 'no-cache', 'content-type': 'text/html', 'connection': 'close', 'status': '503', 'content-location': 'http://192.168.24.3:9311/v1/secrets'}
Body: b'<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n'
barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningTest.test_signed_image_upload_and_boot[compute,id-4343df3c-5553-40ea-8705-0cce73b297a9,image]
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/barbican_tempest_plugin/tests/scenario/barbican_manager.py", line 78, in setUp
self.signing_cert_uuid = self._store_cert(
File "/usr/lib/python3.9/site-packages/barbican_tempest_plugin/tests/scenario/barbican_manager.py", line 167, in _store_cert
result = self.secret_client.create_secret(
File "/usr/lib/python3.9/site-packages/barbican_tempest_plugin/services/key_manager/json/secret_client.py", line 44, in create_secret
resp, body = self.post("v1/secrets", body)
File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 299, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 703, in request
self._error_checker(resp, resp_body)
File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 883, in _error_checker
raise exceptions.UnexpectedResponseCode(str(resp.status),
tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received
Details: 503
Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
2022-05-16 01:42:28,246 214689 WARNING [tempest.api.compute.api_microversion_fixture] APIMicroversionFixture class is deprecated and moved to tempest.lib.common.api_microversion_fixture.APIMicroversionFixture. It will be removed in Z cycle.
2022-05-16 01:42:29,414 214689 DEBUG [barbican_tempest_plugin.tests.scenario.barbican_manager] Uploading certificate to barbican
2022-05-16 01:42:29,417 214689 INFO [tempest.lib.common.rest_client] Request (ImageSigningTest:setUp): 503 POST http://192.168.24.3:9311/v1/secrets 0.002s
2022-05-16 01:42:29,417 214689 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
Body: {"expiration": "2022-05-21T01:42:29.414392", "algorithm": "rsa", "secret_type": "certificate", "payload_content_type": "application/octet-stream", "payload_content_encoding": "base64", "payload": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVkVENDQXQyZ0F3SUJBZ0lVVEplSnhjNkRhU1pTMjlFTWpMczBoaDgvdC84d0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lqRUxNQWtHQTFVRUJoTUNWVk14Q3pBSkJnTlZCQWdNQWtOQk1SWXdGQVlEVlFRSERBMVRZVzRnUm5KaApibU5wYzJOdk1STXdFUVlEVlFRS0RBcE5lU0JEYjIxd1lXNTVNUmt3RndZRFZRUUREQkJVWlhOMElFTmxjblJwClptbGpZWFJsTUI0WERUSXlNRFV4TmpBeE5ESXlPVm9YRFRJeU1EVXlOakF4TkRJeU9Wb3dZakVMTUFrR0ExVUUKQmhNQ1ZWTXhDekFKQmdOVkJBZ01Ba05CTVJZd0ZBWURWUVFIREExVFlXNGdSbkpoYm1OcGMyTnZNUk13RVFZRApWUVFLREFwTmVTQkRiMjF3WVc1NU1Sa3dGd1lEVlFRRERCQlVaWE4wSUVObGNuUnBabWxqWVhSbE1JSUJvakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVk4QU1JSUJpZ0tDQVlFQXF1NTFBZVFZZU9udTdnSHBoT2pnankwblFJZ0MKRzlsWERQWVd2WXRCL0hRN21xdkV2ZU1KUU1FWEZUTVZ6VFN0SVVkTWpFNzR3czlXSkRlZUc3MmRNekxqNFBGawpRSWtTeEpLa3dVM3JzZDUvMTFqN2pVeXRSVk5BdnZuc0QwdWp5YkE3MklDNnkzYWZQK21QVHlqQitud3V3eDloCmZuN0F5UDVsOWUyMVpxa1J5a3VvZjl3WldxeStvYzUyZExZU3NlWjlHT0FEamEzNG0xWnJMdVJ1YzZib3h0c1gKNkZGRlpWaVU3dlZ1dXZoRXlpQnhJNy9LaE1EN1RKaEMzSmdrTU5wTWpUN2UrQkNhWGxaMmhyNUpiLy94aXBsNwpJWGJWcGNFT2RyQ2lMREZFWkJwMTBiSk1aQndpN1gvelVJeG4vejRPQzJ6QjdUUlQvZ0xkV2NMNzI3MGZFUzdZCi9rbmFBMHlKUG1MK1pyU3ljUGRFdFQ4aURQanNjcWZRaXJOa1pkZ0dCR1FWQWJCNE94QjhqWmN3MEFuQTZyRGoKTWkyYlR3dUN4T1M4SXhCWVhZMVhUOWI2SGE5UWx4ODB4WFZtNGhoV0JueW1kRlR6TFVjZnQ3bndoTzM4ZUp6agovRTBRZ0FxUTNTVDJ3eVdpcWY4S0pZYkJqR09UZUxOZ0IzUERBZ01CQUFHakl6QWhNQklHQTFVZEV3RUIvd1FJCk1BWUJBZjhDQVFFd0N3WURWUjBQQkFRREFnTEVNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJnUUFESFJ4cTFSbmwKK000dUxWT3ppcU00WTZkOTNyMzVLckVNSWdZbStzejdsT0k5QnBNcEpQWUJMR3pTOFhOUnd0TkRlNktETjE0ZApzWXQvWEk2ckczSkdHeFI4YURFZWltM0FOSXZ4RXNtYXNieUlpSm9xbU9OVy9JelNQRUtodWFxaDF0UHIvbGtrCk11QWw4a25lbXRockF0ZzFnYjJDOGZhR2pjdXc5OW9Rbk1yWnlNcVJRVmlaQnJzRldKZjFmeDYza3NWV2hrdDQKTHVoQTJ4TUlJTmVkdEZDN3dkZmxnald4TVZGeXh0eThXejc2NFB2Q2tPcko3cmV5TGlqejVMSko1TGVRY0g5eApiZHMvQ2ZIWHgxVm5udFZqQnpWZkxQc3pQaDhDTjlxSTh2dHBia2c5dkRwRU1XRDBRQXlMa1R1aWRPcjVrMXRWCmM1QzRIZ1RyY3BpeUJISHNYOU04cHZlWUVORDJuU3MrRnFsalBIUEl0eUwvVU5wYVlwM0JlNUJ5TVhWaGpENnMKVkZxZ3BDamdIU1ZJQnBqdFB3K2E4bitDdElsZXVSUXUvU2FrSFVxclZpdlRSVXVIcHFGbTFhNnJRWFFnRXlhLwp4czhMVDA4c2JuaUcyT0Fxa3h1VWF5NVk2Q1FGekdhemxPV3p4VmFzTG1mU1hYa2pqZlNrSlFZPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==", "name": "tempest-tempest-sec-1023862517"}
Response - Headers: {'content-length': '107', 'cache-control': 'no-cache', 'content-type': 'text/html', 'connection': 'close', 'status': '503', 'content-location': 'http://192.168.24.3:9311/v1/secrets'}
Body: b'<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n'
~~~
More failures:
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario002-standalone-master/7b0c1d2/logs/
Openstack rpm diff between last current-tripleo and tripleo-
https:/
barbican package is updated as well:-
~~~
openstack-
openstack-
~~~