Application rook-ceph-apps status upload-failed post platform upgrade

[Gabriel de Araújo Cabral]

Brief Description
-----------------
Post upgrade from stx 6 to stx 8 in a standard lab, application "rook-ceph-apps" found to have "upload-failed" status.

Severity
--------
Major

Steps to Reproduce
------------------
Execute upgrade with the stated environmental variables in a standard lab environment.

Expected Behavior
------------------
Application "rook-ceph-apps" should have successful upload post platform SW upgrade.

Actual Behavior
----------------
Application "rook-ceph-apps" has failed upload post platform SW upgrade.

Reproducibility
---------------
100% reproducible.

System Configuration
--------------------
Standard lab. 2 controllers 2 computes.

Last Pass
---------
None

Timestamp/Logs
--------------
./sysinv.log:sysinv 2023-01-26 19:58:38.225 1145576 ERROR sysinv.conductor.manager [-] Failed to find an application tarball for rook-ceph-apps.

./fm-event.log:2023-01-26T20:42:21.318 controller-1 fmManager: info { "event_log_id" : "750.001", "reason_text" : "Application Upload Failure", "entity_instance_id" : "region=RegionOne.system=yow-cgcs-wildcat-63-66.k8s_application=rook-ceph-apps", "severity" : "warning", "state" : "set", "timestamp" : "2023-01-26 20:42:21.203438" }

Test Activity
-------------
Upgrade Testing.

Workaround
----------
none

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/876080

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/876080
Committed: https://opendev.org/starlingx/config/commit/b1f44a16d0124f7eed9e2ff74155100779041568
Submitter: "Zuul (22348)"
Branch: master

commit b1f44a16d0124f7eed9e2ff74155100779041568
Author: Gabriel de Araújo Cabral <email address hidden>
Date: Thu Mar 2 06:58:17 2023 -0500

    Remove metadata from apps_metadata dict when deleting app

    When some application is deleted, it's information remains
    cached in all dicts and lists from apps_metadata, with this
    structure:

        apps_metadata =
            {apps: {},
            platform_managed_apps_list: {},
            desired_states: {},
            ordered_apps: []}

    The mentioned behavior was causing a bug after an upgrade from
    stx 6 to stx 8, because an app that has no support in stx 8 is
    deleted but continue cached in apps_metadata, and when
    k8s_application_audit() is run after the upgrade is complete,
    the system uses the cached list to try to upload this app until
    the upload-failed state appears in the alarms list.

    Now the created method is called right after deleting an
    application, this way the app will be completely out of the
    system.

    Test-Plan:
      PASS: Ugrade from stx 6.0 to 8.0 in a standard config lab.
      PASS: During the upgrade, an app from the previous version
      which has no support in the next version is automatically
    deleted.
      PASS: Validate that the deleted app is not in any
      collection from apps_metadata.
      PASS: Validate that the upload try of the deleted app doesn't
    happen anymore.

    Closes-Bug: 2009025

    Signed-off-by: Gabriel de Araújo Cabral <email address hidden>
    Change-Id: I6a54218b398493acc931c5eca34b800383b16cc0