migrate_platform_certificates_to_certmanager creates incorrect secret-observer override for oidc-auth-app

[Karla Felix]

System Config
Distributed Coud (Subcloud)

HW Make / Model

Description of failure
The ca.crt is missing inside the system-local-ca-oidc-secret secret after the oidc overrides are updated and user_overrides are added to secret observer.

Issue intermittent (Frequency of occurrence) or 100% Reproducible?

Impact of Failure
Minor

Impact on Customer
Minor.

Steps to reproduce

1. Before migrating to cert-manager using the migrate_platform_certificates_to_certmanager.yml playbook there is no user_overrides in the secret-observer override of oidc-auth-apps.

[sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-show oidc-auth-apps secret-observer kube-system
+--------------------+-----------------+
| Property | Value |
+--------------------+-----------------+
| attributes | enabled: true |
| | |
| combined_overrides | None |
| name | secret-observer |
| namespace | kube-system |
| system_overrides | None |
| user_overrides | None |
+--------------------+-----------------+

2. The migration playbook creates cert-manager managed certificates and updates the oidc overrides accordingly. part of that is adding user_overrides to the secret observer

[sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-show oidc-auth-apps secret-observer kube-system
+--------------------+-------------------------------------------+
| Property | Value |
+--------------------+-------------------------------------------+
| attributes | enabled: true |
| | |
| combined_overrides | observedSecrets: |
| | - deploymentToRestart: stx-oidc-client |
| | filename: ca.crt |
| | secretName: system-local-ca-oidc-secret |
| | - deploymentToRestart: stx-oidc-client |
| | filename: tls.crt |
| | secretName: oidc-auth-apps-certificate |
| | - deploymentToRestart: oidc-dex |
| | filename: tls.crt |
| | secretName: oidc-auth-apps-certificate |
| | |
| name | secret-observer |
| namespace | kube-system |
| system_overrides | None |
| user_overrides | observedSecrets: |
| | - deploymentToRestart: stx-oidc-client |
| | filename: ca.crt |
| | secretName: system-local-ca-oidc-secret |
| | - deploymentToRestart: stx-oidc-client |
| | filename: tls.crt |
| | secretName: oidc-auth-apps-certificate |
| | - deploymentToRestart: oidc-dex |
| | filename: tls.crt |
| | secretName: oidc-auth-apps-certificate |
| | |
+--------------------+-------------------------------------------+

3. There is no ca.crt inside the system-local-ca-oidc-secret secret, so that portion of the override seems incorrect:

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n kube-system get secret system-local-ca-oidc-secret
NAME TYPE DATA AGE
system-local-ca-oidc-secret Opaque 1 13m

[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n kube-system describe secret system-local-ca-oidc-secret
Name: system-local-ca-oidc-secret
Namespace: kube-system
Labels: <none>
Annotations: <none>Type: OpaqueData
====
dex-ca.pem: 1870 bytes
[sysadmin@controller-0 ~(keystone_admin)]$

4. Creating a temporary test job out of the daily-scheduled secret observer cronjob we can see that this is indeed an issue (see No such file or directory error).{}

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n kube-system get cronjob secret-observer-cron-job
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
secret-observer-cron-job 0 1 * * * False 0 13h 5d15h

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl create job --from=cronjob/secret-observer-cron-job testjob -n kube-system
job.batch/testjob created[sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n kube-system get job testjob
NAME COMPLETIONS DURATION AGE
testjob 1/1 7s 24s[sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n kube-system logs testjob-n8vpd | head -n 15
+ cat /var/run/secrets/kubernetes.io/serviceaccount/token
+ KUBE_TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6ImJ3V2tqVkhURXBFaW9xZUwtMWxiNThHQVFxemNmQ2RLY0NUUUY4TDlFTmcifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzQ2NzE0OTA1LCJpYXQiOjE3MTUxNzg5MDUsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsInBvZCI6eyJuYW1lIjoidGVzdGpvYi1uOHZwZCIsInVpZCI6ImM0NzY2NmZiLWI2N2YtNDgxNC05MzY4LTNmZjVkZWUzNDZlMSJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoic2VjcmV0LW9ic2VydmVyIiwidWlkIjoiMWMzMmQ2N2ItZWI0YS00OGM1LThlZDAtZmIxMjBiYzYzOWY4In0sIndhcm5hZnRlciI6MTcxNTE4MjUxMn0sIm5iZiI6MTcxNTE3ODkwNSwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOnNlY3JldC1vYnNlcnZlciJ9.tYom2BJJYW0z6yVMlbw4tDycn5KRbAx_eX0RqcFaya6dppZ4273mecqIifLwxyCFEKKbzVUfzKIX4VO_lHx8QnCvzayoV_4y4wjxgrcGKeRJhWkU7EqDf-deX7Zmp87gr1r9-ag0h4P9vzkKSDDKDpkS01o4AiWWBwTdpai4G88YGPceKEu7RY2P6j0uKZQr3VJb5r97liZhulT1JlAnT80-RIWc13XtAqvEThQyNHN5EHImWviNbivzmc6vSCTMSJjlP7QRy3W5Vvt_LmNysqWSzeIcxEPTbUIYcCeF3U71fq88d3G7IHFannNNZ-z-iW8B0EazvTKfV0VVrRQxig
+ sha256sum /home/system-local-ca-oidc-secret-stx-oidc-client/ca.crt
+ awk '{print $1}'
sha256sum: can't open '/home/system-local-ca-oidc-secret-stx-oidc-client/ca.crt': No such file or directory <=====
+ SECRET_SHA=
+ KUBERNETES_SERVICE_HOST_WITH_BRACKETS=fd00:4888:0:2::1
+ echo fd00:4888:0:2::1
+ grep :
fd00:4888:0:2::1
+ KUBERNETES_SERVICE_HOST_WITH_BRACKETS='[fd00:4888:0:2::1]'
+ curl -sSg -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImJ3V2tqVkhURXBFaW9xZUwtMWxiNThHQVFxemNmQ2RLY0NUUUY4TDlFTmcifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzQ2NzE0OTA1LCJpYXQiOjE3MTUxNzg5MDUsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsInBvZCI6eyJuYW1lIjoidGVzdGpvYi1uOHZwZCIsInVpZCI6ImM0NzY2NmZiLWI2N2YtNDgxNC05MzY4LTNmZjVkZWUzNDZlMSJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoic2VjcmV0LW9ic2VydmVyIiwidWlkIjoiMWMzMmQ2N2ItZWI0YS00OGM1LThlZDAtZmIxMjBiYzYzOWY4In0sIndhcm5hZnRlciI6MTcxNTE4MjUxMn0sIm5iZiI6MTcxNTE3ODkwNSwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOnNlY3JldC1vYnNlcnZlciJ9.tYom2BJJYW0z6yVMlbw4tDycn5KRbAx_eX0RqcFaya6dppZ4273mecqIifLwxyCFEKKbzVUfzKIX4VO_lHx8QnCvzayoV_4y4wjxgrcGKeRJhWkU7EqDf-deX7Zmp87gr1r9-ag0h4P9vzkKSDDKDpkS01o4AiWWBwTdpai4G88YGPceKEu7RY2P6j0uKZQr3VJb5r97liZhulT1JlAnT80-RIWc13XtAqvEThQyNHN5EHImWviNbivzmc6vSCTMSJjlP7QRy3W5Vvt_LmNysqWSzeIcxEPTbUIYcCeF3U71fq88d3G7IHFannNNZ-z-iW8B0EazvTKfV0VVrRQxig' --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt --request PATCH -H 'Accept: application/json' -H 'Content-Type: application/strategic-merge-patch+json' 'https://[fd00:4888:0:2::1]:443/apis/apps/v1/namespaces/kube-system/deployments/stx-oidc-client' --data '{"spec":{"template":{"metadata":{"annotations":{"system-local-ca-oidc-secretsha": ""}}}}}'
{
"kind": "Deployment",
"apiVersion": "apps/v1",
[sysadmin@controller-0 ~(keystone_admin)]$

Log/File location

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/921792
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/70436a8dd9b68683ebe59167bee56b20679555cf
Submitter: "Zuul (22348)"
Branch: master

commit 70436a8dd9b68683ebe59167bee56b20679555cf
Author: Karla Felix <email address hidden>
Date: Tue Jun 11 15:25:04 2024 -0300

    Change oidc-auth-apps-certificate secret observer

    This review will be changing the secret observer of oidc-auth-apps
    to look for the ca.crt value inside of oidc-auth-apps-certificate,
    removing then the step to create a new secret that stores this
    value.

    Test Plan:
    PASS: Generate a new .iso with the changes.
    PASS: Run a full deploy of an AIO-SX and DC system with the
          generated iso.
    PASS: Use oidc to switch to another user context and execute
          kubectl commands in AIO-SX and DC system.
    PASS: Run update-platform-certificate playbook in AIO-SX and
          DC system.
    PASS: Create a test job out of the daily-scheduled secret
          observer and verify the logs to see if finished
          correctly.
    PASS: Delete the secret and verify if the observer will restart
          deployment in the correct timeset.

    Closes-Bug: 2069323

    Change-Id: Ic590fb492caf7e5bbc15851db0a9fffead7cd69b
    Signed-off-by: Karla Felix <email address hidden>