periodic ovb fs39 component security fails "Assign admin role to admin project - Internal Server Error"

[Marios Andreou]

At [1][2][3] the

periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master

(security component) is failing deployment with trace that looks like:

        * 2022-05-03 14:15:09 | 2022-05-03 14:15:09.885873 | fa163e57-9ab4-fb92-17cb-00000000843f | FATAL | Assign admin role to admin project for admin user | undercloud | error={"changed": false, "extra_data": {"data": null, "details": "An unexpected error prevented the server from fulfilling your request.", "response": "{\"error\": {\"code\": 500, \"title\": \"Internal Server Error\", \"message\": \"An unexpected error prevented the server from fulfilling your request.\"}}"}, "msg": "HttpException: 500: Server Error for url: https://overcloud.ooo.test:13000/v3/users?name=admin, An unexpected error prevented the server from fulfilling your request."}

Note the trace at [3] is slightly different but looks like it is related imo.

Job history is at [4] but we only currently have these 3 known examples of the bug (other/earlier failures before 01 May are different).

[1] https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/1c3794b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[2] https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/05935d1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[3] https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/510c5d2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[4] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master

[Marios Andreou (marios-b)]

we had a green run with testproject there https://review.rdoproject.org/r/c/testproject/+/41128/16#message-4f3285394e477af2f5f23de635f679105e5c1f80

either this is a transient issue that hit us 2/3 times but now resolved, or it is inconsistent and we'll see more examples.

Should just check over the next few days keep an eye on https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master

[Sandeep Yadav (sandeepyadav93)]

Closing for now as we are seeing green runs[1] and [2]:-

[1] https://review.rdoproject.org/zuul/build/efcb989a6d3641608e0eef1ffffe5920
[2] https://review.rdoproject.org/zuul/build/253d26bfc0e3470694d4eca7ab268afc

We can reopen if we hit this issue again.