AppFwk: No deletion order defined when removing a fluxcd application

Bug #2000326 reported by Thales Elero Cervi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Triaged
Medium
Luan Nunes Utimura

Bug Description

Brief Description
-----------------

When trying to create a cleanup Job for stx-openstack it was noticed that there is no order defined for the application helm release removal [1].

It looks like something that could be enhanced. If there is a dependsOn chain for installation, maybe that reverse chain should be respected for removal [2].

[1] https://opendev.org/starlingx/config/src/branch/master/sysinv/sysinv/sysinv/sysinv/conductor/kube_app.py#L3243
[2] https://opendev.org/starlingx/config/src/branch/master/sysinv/sysinv/sysinv/sysinv/conductor/kube_app.py#L1295

Severity
--------
Minor: It will only affect applications when it is needed to enforce a removal order.

Steps to Reproduce
------------------
* Install a system and apply an stx-openstack app with the following cherry pick: https://review.opendev.org/c/starlingx/openstack-armada-app/+/867892

(It is a cleanup Job that relies on resources of other helm releases to perform its actions)

Expected Behavior
------------------
The dependsOn chain should be respected to remove helm releases and the Job would be able to spawn and run its script.
Application remove should work.

Actual Behavior
----------------
The dependsOn chain is not respected when removing helm releases, so the cleanup Job is hangs waiting for other helm releases' Services that are no longer running.
Application remove fails.

Reproducibility
---------------
Reproducible with a custom build of the stx-openstack that includes https://review.opendev.org/c/starlingx/openstack-armada-app/+/867892

System Configuration
--------------------
AIO-SX

Branch/Pull Time/Commit
-----------------------
master:
* starlingx/master/debian/monolithic/20221206T070000Z

+ https://review.opendev.org/c/starlingx/openstack-armada-app/+/867892

Last Pass
---------
N/A - Although this Job used to work with Armada, now we need to adapt it to FluxCD.

Timestamp/Logs
--------------
sysinv 2022-12-19 20:01:29.772 103223 INFO sysinv.conductor.kube_app [-] Application (wr-openstack) remove started.
sysinv 2022-12-19 20:01:29.931 103223 INFO sysinv.conductor.kube_app [-] lifecycle hook for application wr-openstack (22.08-0) started {'lifecycle_type': u'fluxcd-request', 'relative_timing': u'pre', 'operation': u'delete', 'extra': {}}.
sysinv 2022-12-19 20:01:29.931 103223 INFO sysinv.conductor.kube_app [-] Doing FluxCD operation delete with the following manifest: /opt/platform/fluxcd/22.06/wr-openstack/22.08-0/wr-openstack-fluxcd-manifests
sysinv 2022-12-19 20:01:31.370 103223 INFO sysinv.conductor.kube_app [-] lifecycle hook for application wr-openstack (22.08-0) started {'lifecycle_type': u'fluxcd-request', 'relative_timing': u'post', 'operation': u'delete', 'extra': {'rc': False}}.
sysinv 2022-12-19 20:01:31.380 103223 ERROR sysinv.conductor.kube_app [-] Application remove aborted!.

neutron-resources-cleanup:
 Container
 (...)
  State: Waiting
  Reason: PodInitializing
  Ready: False

$ kubectl -n openstack logs -f neutron-resources-cleanup-ntb2v -c init
Entrypoint WARNING: 2022/12/21 16:57:27 entrypoint.go:72: Resolving dependency Service rabbitmq in namespace openstack failed: endpoints "rabbitmq" not found.
Entrypoint WARNING: 2022/12/21 16:57:27 entrypoint.go:72: Resolving dependency Service keystone-api in namespace openstack failed: endpoints "keystone-api" not found.
Entrypoint WARNING: 2022/12/21 16:57:27 entrypoint.go:72: Resolving dependency Service mariadb in namespace openstack failed: endpoints "mariadb" not found.
Entrypoint WARNING: 2022/12/21 16:57:27 entrypoint.go:72: Resolving dependency Service nova-api in namespace openstack failed: endpoints "nova-api" not found.

Test Activity
-------------
Developer Testing

Workaround
----------
None

tags: added: stx.8
tags: added: stx.apps stx.distro.openstack
Changed in starlingx:
assignee: nobody → Luan Nunes Utimura (lutimura)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/868928

Changed in starlingx:
status: New → In Progress
Ghada Khalil (gkhalil)
tags: added: stx.8.0
removed: stx.8
Revision history for this message
Thales Elero Cervi (tcervi) wrote :

This is a nice to have improvement, specially for huge apps like the openstack.
It tries to mirror previous Armada's behavior on our new FluxCD based AppFwk.

Nothing that should gate starlingx release 8.0

tags: removed: stx.8.0 stx.apps
tags: added: stx.apps
Changed in starlingx:
status: In Progress → Triaged
importance: Undecided → Low
importance: Low → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.