CS9 - OVB FS039 and FS064 master jobs are failing on overcloud deploy. Failed task: 'Check Keystone internal endpoint status'

[Douglas Viroel]

The following periodic jobs are failing on overcloud deploy, task "Check Keystone project status"
periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master
https://logserver.rdoproject.org/76/36976/31/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/14195e2/logs
periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master
https://logserver.rdoproject.org/76/36976/31/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/e05fac4/logs

with the following error:
 Check Keystone project status | undercloud | item=service | error={"ansible_job_id": "825873963093.243789", "ansible_loop_var": "tripleo_keystone_resources_project_async_result_item", "attempts": 1, "changed": false, "extra_data": {"data": null, "details": "None", "response": "None"}, "finished": 1, "msg": "HttpException: 500: Server Error for url: https://overcloud1.ooo.test:13000/v3/projects?domain_id=default, An unexpected error prevented the server from fulfilling your request.", "results_file": "/root/.ansible_async/825873963093.243789", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": [], "tripleo_keystone_resources_project_async_result_item": {"ansible_job_id": "825873963093.243789", "ansible_loop_var": "tripleo_keystone_resources_project", "changed": true, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/825873963093.243789", "started": 1, "tripleo_keystone_resources_project": "service"}}

keystone authentication middleware is showing some errors on controller1 for fs064 run:
2022-04-25 18:07:47.296 17 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-ff292079-b095-43e3-8986-59cf098028ca - - - - -] Authenticating user token process_request /usr/lib/python3.9/site-packages/keystonemiddleware/auth_token/__init__.py:407
2022-04-25 18:07:47.298 17 ERROR keystone.server.flask.request_processing.middleware.auth_context [req-ff292079-b095-43e3-8986-59cf098028ca - - - - -] [Errno 110] Connection timed out: TimeoutError: [Errno 110] Connection timed out

See more on:
https://logserver.rdoproject.org/76/36976/31/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/e05fac4/logs/overcloud1-controller-1/var/log/extra/errors.txt.gz
https://logserver.rdoproject.org/76/36976/31/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/e05fac4/logs/overcloud1-controller-1/var/log/containers/keystone/keystone.log.txt.gz

[Rabi Mishra (rabi)]

I can see latest green runs of these jobs and the error seem like some transient connectivity issue.

[Douglas Viroel (dviroel)]

Moving to Invalid since we don't see this behavior anymore.