periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation fails with validation errors on keystone token request

[Jiri Podivin]

Job fails during execution of the validations catalogue. Three of the validations are consistently failing during keystone token request: overcloud-service-status, openstack-endpoints and opeshift-nw-requirements.

Logs do not contain more information, as the output of the task isn't logged for security reasons.
Testproject run has been initialized to obtain more info.

https://review.rdoproject.org/r/c/testproject/+/46892

Trace:
------

       {
                    "hosts": {
                        "undercloud": {
                            "action": "uri",
                            "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result",
                            "changed": false,
                            "failed": true
                        }
                    },
                    "task": {
                        "duration": {
                            "end": "2023-02-06T00:24:48.319411Z",
                            "start": "2023-02-06T00:24:47.460330Z"
                        },
                        "id": "fa163ecf-160b-3b8d-da21-00000000000f",
                        "name": "openstack_endpoints : Check Keystone"
                    }
                }

Logs:
-----
https://logserver.rdoproject.org/openstack-component-validation/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation/80e7578/logs/undercloud/home/zuul/validations/
https://logserver.rdoproject.org/openstack-component-validation/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation/80e7578/logs/undercloud/home/zuul/validations/5d40b471-acdd-4140-8a60-5b8dbe5310cd_openstack-endpoints_2023-02-06T00:24:46.327363Z.json.txt.gz
https://logserver.rdoproject.org/openstack-component-validation/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation/80e7578/

[Jiri Podivin (jpodivin)]

Logs were collected from the more recent execution with debug enabled.

It appears that the request isn't authenticated, resulting in error 401. Previously this was not an issue, which may point towards a change in keystone token management.

Error message:
--------------
                            "json": {
                                "error": {
                                    "code": 401,
                                    "message": "The request you have made requires authentication.",
                                    "title": "Unauthorized"
                                }
                            }

Logs:
-----

https://logserver.rdoproject.org/92/46892/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation/d676640/logs/undercloud/home/zuul/validations/3d89de37-5d7c-409d-8214-f1c13e8acbaf_overcloud-service-status_2023-02-06T16:14:17.494500Z.json.txt.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/validations-common/+/873355

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Jiri Podivin <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/validations-common/+/873355
Reason: Issue was fixed by https://review.rdoproject.org/r/c/rdo-jobs/+/46994

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-common/+/877447

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/877457

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/tripleo-common/+/878154

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-common/+/878589

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Takashi Kajinami <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-common/+/877447
Reason: master is being abandoned

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/878154
Committed: https://opendev.org/openstack/tripleo-common/commit/010c9aeb56e245694baff302fe4119df32bf0c57
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 010c9aeb56e245694baff302fe4119df32bf0c57
Author: Takashi Kajinami <email address hidden>
Date: Wed Mar 15 15:07:21 2023 +0900

    Read stack output to get AdminPassword

    We are currently using environment API to obtain AdminPassword but
    the API no longer shows the secret informations.
    This updates the logic to use the stack output instead.

    Closes-Bug: #2006125
    Change-Id: I788f7b52cad8008e1a1fbd1329d51606cee262fe

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/877457
Committed: https://opendev.org/openstack/tripleo-common/commit/6f3b270a318f62a6e8826c636b813a9bc8781145
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 6f3b270a318f62a6e8826c636b813a9bc8781145
Author: Takashi Kajinami <email address hidden>
Date: Wed Mar 15 15:07:21 2023 +0900

    Read stack output to get AdminPassword

    We are currently using environment API to obtain AdminPassword but
    the API no longer shows the secret informations.
    This updates the logic to use the stack output instead.

    Conflicts:
            tripleo_common/tests/test_inventory.py

    Closes-Bug: #2006125
    Change-Id: I788f7b52cad8008e1a1fbd1329d51606cee262fe
    (cherry picked from commit 010c9aeb56e245694baff302fe4119df32bf0c57)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/878589
Committed: https://opendev.org/openstack/tripleo-common/commit/8d2c05f395bacd62f4fbaea362eb6d224aee9fe3
Submitter: "Zuul (22348)"
Branch: stable/train

commit 8d2c05f395bacd62f4fbaea362eb6d224aee9fe3
Author: Takashi Kajinami <email address hidden>
Date: Wed Mar 15 15:07:21 2023 +0900

    Read stack output to get AdminPassword

    We are currently using environment API to obtain AdminPassword but
    the API no longer shows the secret informations.
    This updates the logic to use the stack output instead.

    Conflicts:
            tripleo_common/inventory.py
            tripleo_common/tests/test_inventory.py

    Closes-Bug: #2006125
    Change-Id: I788f7b52cad8008e1a1fbd1329d51606cee262fe
    (cherry picked from commit 010c9aeb56e245694baff302fe4119df32bf0c57)
    (cherry picked from commit 6f3b270a318f62a6e8826c636b813a9bc8781145)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common train-eol release.