Trident's registry credentials not updated when credentials change

[João Victor Portal]

Brief Description
-----------------
Trident's registry credentials are not updated when credentials change. The pods reports the following status after update:

trident                       trident-csi-5wxp4                                    0/2     ImagePullBackOff        70         77d
trident                       trident-csi-666dd658f6-bdpfv                         0/5     ImagePullBackOff        0          152m
trident                       trident-csi-6kr6r                                    0/2     ImagePullBackOff        74         77d
trident                       trident-csi-7n6zj                                    0/2     ImagePullBackOff        66         77d

Severity
--------
Major

Steps to Reproduce
------------------
Change the registry credentials and check the secret 'trident-local-registry-secret' in namespace 'trident'.

Expected Behavior
------------------
The trident secret should be updated.

Actual Behavior
----------------
The trident secret value doesn't change.

Reproducibility
---------------
100% reproducible.

System Configuration
--------------------
All

Branch/Pull Time/Commit
-----------------------
N/A.

Last Pass
---------
N/A.

Timestamp/Logs
--------------
  Warning Failed 174m (x3 over 176m) kubelet, controller-0 Error: ErrImagePull
  Warning Failed 174m (x3 over 176m) kubelet, controller-0 Failed to pull image "registry.local:9001/docker.io/netapp/trident:20.04.0": rpc error: code = Unknown desc = failed to pull and unpack image "registry.local:9001/docker.io/netapp/trident:20.04.0": failed to resolve reference "registry.local:9001/docker.io/netapp/trident:20.04.0": failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized
  Warning Failed 174m (x3 over 176m) kubelet, controller-0 Error: ErrImagePull
  Warning Failed 174m (x2 over 176m) kubelet, controller-0 Error: ImagePullBackOff
  Warning Failed 174m (x2 over 176m) kubelet, controller-0 Error: ImagePullBackOff

Test Activity
-------------
N/A.

Workaround
----------
Use the following command to get the current base64 encoded credentials:

kubectl get secret trident-local-registry-secret -n trident -o yaml

Decode the data value .dockerconfigjson, replace the password with the new one and replace the base64 encoded username:password as well. Then re-encode the entire string. Store the new data with:

kubectl edit secret trident-local-registry-secret -n trident

Wait for all pods to come back up. You can delete them to speed this process up.

[João Victor Portal (jvictorp)]

The changes are ready to be reviewed at https://review.opendev.org/c/starlingx/config/+/801401 . The tests performed were the following: in an AIOSX VM deploy created with an ISO created from dev branch plus the changes, the following secrets (the complete list of secrets affected by the audit method) were edited manually and after it was verified if the secret value was corrected with the right value (the secret 'trident-local-registry-secret' was created manually).

name registry-local-secret, namespace kube-system
name trident-local-registry-secret, namespace trident
name default-registry-key, namespace armada
name default-registry-key, namespace cert-manager
name default-registry-key, namespace deployment
name default-registry-key, namespace kube-system

Values of secret used:

  Wrong base64 encoded JSON credentials: ".dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS5sb2NhbDo5MDAxIjp7InVzZXJuYW1lIjoienp6IiwicGFzc3dvcmQiOiJ6enoiLCJhdXRoIjoienp6In19fQ=="
  Empty base64 encoded JSON: ".dockerconfigjson: e30="

The pooling interval of audit is 1 min.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "João Victor Portal <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/config/+/801401
Reason: This review will be abandoned and a new one will be opened specifically for a target version.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/805700

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/805700
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/b4ed5c5e42da425b4e1f599b79942b8bea8c1993
Submitter: "Zuul (22348)"
Branch: master

commit b4ed5c5e42da425b4e1f599b79942b8bea8c1993
Author: Joao Victor Portal <email address hidden>
Date: Mon Aug 23 17:30:59 2021 -0300

    Changed Trident secret and registry credentials

    Changed Trident secret name from "trident-local-registry-secret" to
    "default-registry-key" and got registry credentials from sysinv.

    Closes-Bug: 1937301
    Signed-off-by: Joao Victor Portal <email address hidden>
    Change-Id: I8bc236f267b878081de2a54b701a32d60175288f