AppFwk: No deletion order defined when removing a fluxcd application

[Thales Elero Cervi]

Brief Description
-----------------

When trying to create a cleanup Job for stx-openstack it was noticed that there is no order defined for the application helm release removal [1].

It looks like something that could be enhanced. If there is a dependsOn chain for installation, maybe that reverse chain should be respected for removal [2].

[1] https://opendev.org/starlingx/config/src/branch/master/sysinv/sysinv/sysinv/sysinv/conductor/kube_app.py#L3243
[2] https://opendev.org/starlingx/config/src/branch/master/sysinv/sysinv/sysinv/sysinv/conductor/kube_app.py#L1295

Severity
--------
Minor: It will only affect applications when it is needed to enforce a removal order.

Steps to Reproduce
------------------
* Install a system and apply an stx-openstack app with the following cherry pick: https://review.opendev.org/c/starlingx/openstack-armada-app/+/867892

(It is a cleanup Job that relies on resources of other helm releases to perform its actions)

Expected Behavior
------------------
The dependsOn chain should be respected to remove helm releases and the Job would be able to spawn and run its script.
Application remove should work.

Actual Behavior
----------------
The dependsOn chain is not respected when removing helm releases, so the cleanup Job is hangs waiting for other helm releases' Services that are no longer running.
Application remove fails.

Reproducibility
---------------
Reproducible with a custom build of the stx-openstack that includes https://review.opendev.org/c/starlingx/openstack-armada-app/+/867892

System Configuration
--------------------
AIO-SX

Branch/Pull Time/Commit
-----------------------
master:
* starlingx/master/debian/monolithic/20221206T070000Z

+ https://review.opendev.org/c/starlingx/openstack-armada-app/+/867892

Last Pass
---------
N/A - Although this Job used to work with Armada, now we need to adapt it to FluxCD.

Timestamp/Logs
--------------
sysinv 2022-12-19 20:01:29.772 103223 INFO sysinv.conductor.kube_app [-] Application (wr-openstack) remove started.
sysinv 2022-12-19 20:01:29.931 103223 INFO sysinv.conductor.kube_app [-] lifecycle hook for application wr-openstack (22.08-0) started {'lifecycle_type': u'fluxcd-request', 'relative_timing': u'pre', 'operation': u'delete', 'extra': {}}.
sysinv 2022-12-19 20:01:29.931 103223 INFO sysinv.conductor.kube_app [-] Doing FluxCD operation delete with the following manifest: /opt/platform/fluxcd/22.06/wr-openstack/22.08-0/wr-openstack-fluxcd-manifests
sysinv 2022-12-19 20:01:31.370 103223 INFO sysinv.conductor.kube_app [-] lifecycle hook for application wr-openstack (22.08-0) started {'lifecycle_type': u'fluxcd-request', 'relative_timing': u'post', 'operation': u'delete', 'extra': {'rc': False}}.
sysinv 2022-12-19 20:01:31.380 103223 ERROR sysinv.conductor.kube_app [-] Application remove aborted!.

neutron-resources-cleanup:
 Container
 (...)
  State: Waiting
  Reason: PodInitializing
  Ready: False

$ kubectl -n openstack logs -f neutron-resources-cleanup-ntb2v -c init
Entrypoint WARNING: 2022/12/21 16:57:27 entrypoint.go:72: Resolving dependency Service rabbitmq in namespace openstack failed: endpoints "rabbitmq" not found.
Entrypoint WARNING: 2022/12/21 16:57:27 entrypoint.go:72: Resolving dependency Service keystone-api in namespace openstack failed: endpoints "keystone-api" not found.
Entrypoint WARNING: 2022/12/21 16:57:27 entrypoint.go:72: Resolving dependency Service mariadb in namespace openstack failed: endpoints "mariadb" not found.
Entrypoint WARNING: 2022/12/21 16:57:27 entrypoint.go:72: Resolving dependency Service nova-api in namespace openstack failed: endpoints "nova-api" not found.

Test Activity
-------------
Developer Testing

Workaround
----------
None

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/868928

[Thales Elero Cervi (tcervi)]

This is a nice to have improvement, specially for huge apps like the openstack.
It tries to mirror previous Armada's behavior on our new FluxCD based AppFwk.

Nothing that should gate starlingx release 8.0