stx-openstack: Keystone release upgrade fails during application update

[Luan Nunes Utimura]

Brief Description
-----------------
During an attempt to update stx-openstack, it was observed that the keystone release upgrade failed due to pods stuck in a crash loop.

Severity
--------
Major: The stx-openstack update fails and leaves the application in "apply-failed" state.

Steps to Reproduce
------------------
* Upload and apply stx-openstack;
* Update application to a different version.

Expected Behavior
------------------
Update finishes successfully and application is now applied with a different version.

Actual Behavior
----------------
The keystone release upgrade fails and the application update is interrupted.

Reproducibility
---------------
Reproducible.

System Configuration
--------------------
AIO-SX.

Branch/Pull Time/Commit
-----------------------
master:
* starlingx/master/debian/monolithic/latest_build

Last Pass
---------
N/A.

Timestamp/Logs
--------------

Inspecting sysinv logs shows:

sysinv 2022-12-03 00:24:54.649 1302413 ERROR sysinv.conductor.kube_app [-] Application stx-openstack: release keystone: Failed during apply :Helm upgrade failed: post-upgrade hooks failed: job failed: BackoffLimitExceededLast Helm logs:keystone-credential-setup: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Add/Modify event for keystone-credential-setup: MODIFIED
keystone-credential-setup: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Add/Modify event for keystone-credential-setup: MODIFIED
warning: Upgrade "osh-openstack-keystone" failed: post-upgrade hooks failed: job failed: BackoffLimitExceeded: AttributeError: 'ConductorManager' object has no attribute 'RPC_API_NAMESPACE'
2022-12-03 00:24:54.649 1302413 ERROR sysinv.conductor.kube_app Traceback (most recent call last):
2022-12-03 00:24:54.649 1302413 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 151, in dispatch
2022-12-03 00:24:54.649 1302413 ERROR sysinv.conductor.kube_app cb_namespace = proxyobj.RPC_API_NAMESPACE
2022-12-03 00:24:54.649 1302413 ERROR sysinv.conductor.kube_app AttributeError: 'ConductorManager' object has no attribute 'RPC_API_NAMESPACE'
2022-12-03 00:24:54.649 1302413 ERROR sysinv.conductor.kube_app

Describing `keystone-api` pod shows:

Warning FailedMount 15s (x7 over 47s) kubelet MountVolume.SetUp failed for volume "keystone-fernet-keys" : secret "keystone-fernet-keys" not found
Warning FailedMount 15s (x7 over 47s) kubelet MountVolume.SetUp failed for volume "keystone-credential-keys" : secret "keystone-credential-keys" not found

Describing `keystone-credential-setup` pod shows:

Warning BackOff 23s kubelet Back-off restarting failed container

Inspecting `keystone-credential-setup` pod logs shows:

2022-12-03 00:20:33.409 - ERROR - Cannot get secret keystone-credential-keys.
2022-12-03 00:20:33.409 - ERROR - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"keystone-credential-keys\" not found","reason":"NotFound","details":{"name":"keystone-credential-keys","kind":"secrets"},"code":404}

Test Activity
-------------
Developer Testing.

Workaround
----------
N/A.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/openstack-armada-app/+/868234

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/openstack-armada-app/+/868234
Committed: https://opendev.org/starlingx/openstack-armada-app/commit/8e45c65c7ddf15bef43ff613b6484dc91cb011da
Submitter: "Zuul (22348)"
Branch: master

commit 8e45c65c7ddf15bef43ff613b6484dc91cb011da
Author: Luan Nunes Utimura <email address hidden>
Date: Tue Dec 20 14:15:10 2022 -0300

    OSH-Patch: Fixing keystone helm release hooks

    This patch adds a helm hook/hook-weight to two Keystone secrets that
    weren't being correctly replaced during the stx-openstack application
    update.

    As a result, other jobs/pods that depended on these secrets failed, and
    ultimately the Keystone helm release wasn't upgraded by Helm.

    Test plan:
    PASS - Build openstack-helm package
    PASS - Build stx-openstack-helm-fluxcd package
    PASS - Build stx-openstack helm charts
    PASS - Upload/apply/remove stx-openstack

    Closes-Bug: 2000172

    Signed-off-by: Luan Nunes Utimura <email address hidden>
    Change-Id: I4cd39bc0326b2af2c5d1a9c2c2a68198035bd300