Clean up static and system overrides as helm releases are deleted

[David Barbosa Bastos]

Brief Description
-----------------
After performing application updates on apps there need helm releases to be deleted, static and system overrides secrets are reported remaining on system.

Severity
--------
Minor

Steps to Reproduce
------------------
Perform 'system application-update' on any application that psp-rolebinding helm release was removed on a recent version (cert-manager, portieris, etc.) and observe static and system overrides secrets on system.

Expected Behavior
------------------
Write down what was expected after taking the steps written above

Actual Behavior
----------------
Static and system overrides should be removed as part of the helm release cleanup process.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
AIO-SX

Branch/Pull Time/Commit
-----------------------
SW_VERSION="23.09"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="2023-09-30_18-00-11"
SRC_BUILD_ID="1357"

Last Pass
---------
n/a

Timestamp/Logs
--------------
After update platform-integ-apps secrets wasn't recreated:
 - ceph-csi-cephfs
 - ceph-csi-rbd
 - ceph-pools-audit

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get secrets -n kube-system
NAME TYPE DATA AGE
bootstrap-token-pgxqts bootstrap.kubernetes.io/token 7 3h40m
ceph-admin Opaque 2 6d2h
ceph-pool-kube-cephfs-data kubernetes.io/cephfs 4 6d2h
ceph-pool-kube-rbd kubernetes.io/rbd 2 6d2h
ceph-pools-audit-static-overrides Opaque 1 6d2h
ceph-pools-audit-system-overrides Opaque 1 6d2h
cephfs-provisioner-static-overrides Opaque 1 6d2h
cephfs-provisioner-system-overrides Opaque 1 6d2h
coredump-secret-token kubernetes.io/service-account-token 3 6d3h
default-registry-key kubernetes.io/dockerconfigjson 1 6d3h
dex-static-overrides Opaque 1 3h2m
dex-system-overrides Opaque 1 3h2m
ic-nginx-ingress-ingress-nginx-admission Opaque 3 6d3h
ingress-nginx-static-overrides Opaque 1 6d3h
ingress-nginx-system-overrides Opaque 1 6d3h
kubevirt-static-overrides Opaque 1 6d2h
kubevirt-system-overrides Opaque 1 6d2h
oidc-client-static-overrides Opaque 1 3h2m
oidc-client-system-overrides Opaque 1 3h2m
oidc-dex Opaque 1 3h2m
rbd-provisioner-static-overrides Opaque 1 6d2h
rbd-provisioner-system-overrides Opaque 1 6d2h
registry-local-secret kubernetes.io/dockerconfigjson 1 6d3h
secret-observer-static-overrides Opaque 1 3h2m
secret-observer-system-overrides Opaque 1 3h2m
sh.helm.release.v1.ic-nginx-ingress.v1 helm.sh/release.v1 1 6d3h
sh.helm.release.v1.oidc-auth-secret-observer.v1 helm.sh/release.v1 1 3h2m
sh.helm.release.v1.oidc-dex.v1 helm.sh/release.v1 1 3h2m
sh.helm.release.v1.stx-ceph-pools-audit.v1 helm.sh/release.v1 1 6d2h
sh.helm.release.v1.stx-ceph-pools-audit.v2 helm.sh/release.v1 1 2m29s
sh.helm.release.v1.stx-ceph-pools-audit.v3 helm.sh/release.v1 1 66s
sh.helm.release.v1.stx-cephfs-provisioner.v1 helm.sh/release.v1 1 6d2h
sh.helm.release.v1.stx-cephfs-provisioner.v2 helm.sh/release.v1 1 2m28s
sh.helm.release.v1.stx-cephfs-provisioner.v3 helm.sh/release.v1 1 65s
sh.helm.release.v1.stx-rbd-provisioner.v1 helm.sh/release.v1 1 6d2h
sh.helm.release.v1.stx-rbd-provisioner.v2 helm.sh/release.v1 1 2m28s
sh.helm.release.v1.stx-rbd-provisioner.v3 helm.sh/release.v1 1 65s

Test Activity
-------------
Feature Testing

Workaround
----------
$ kubectl delete -k /opt/platform/fluxcd/22.12/platform-integ-apps/1.0-113/platform-integ-apps-fluxcd-manifests/rbd-provisioner
secret "rbd-provisioner-static-overrides" deleted
secret "rbd-provisioner-system-overrides" deleted
helmrelease.helm.toolkit.fluxcd.io "rbd-provisioner" deleted

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/899167

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/899167
Committed: https://opendev.org/starlingx/config/commit/301b401c9ac8cc8de50c5672f2b1c10cb6e52df9
Submitter: "Zuul (22348)"
Branch: master

commit 301b401c9ac8cc8de50c5672f2b1c10cb6e52df9
Author: David Barbosa Bastos <email address hidden>
Date: Mon Oct 23 18:30:22 2023 -0300

    Delete Kubernetes resources on application updates

    Within the application update process, it is necessary to
    call "kubectl delete -k <manifest_dir>" to delete the
    charts that will not be present in the new version of the
    application. This way we eliminate unnecessary
    remnants of secrets in the N+1 application.

    Test Plan:
    PASS: Upload/Apply/Remove/Delete cert-manager
    PASS: Upload/Apply/Remove/Delete plataform-integ-apps
    PASS: Platform-integ-app update to the new version with
    changed list of the charts
    PASS: Secrets no longer used were deleted
    PASS: If the update fails, it must remove the secrets that
    are in version N+1 and should not be in version N

    Closes-Bug: 2040277

    Change-Id: I1c281491d30b46a7cbf53211890bb4add021dcc8
    Signed-off-by: David Barbosa Bastos <email address hidden>