STX-Openstack | Fail to apply - Helm release info for [chart] is not available

[Gabriel Calixto de Paula]

Brief Description
-----------------
STX-Openstack is failing to apply on STX master, it gets stuck on 86% and then fails after reaching the timeout

Severity
--------
Major

Steps to Reproduce
------------------
Apply STX-Openstack

Expected Behavior
------------------
STX-Openstack should apply

Actual Behavior
----------------
STX-Openstack apply reaches timeout and doesn't apply

Reproducibility
---------------
Reproducible

System Configuration
--------------------
DX

Branch/Pull Time/Commit
-----------------------
STX + STX-O 2023-08-13 06:00:00 +0000

Last Pass
---------
Aug-4-2023, STX master 2023-08-04_20:01:38
STX-O master 2023-08-08 15:47:08

Timestamp/Logs
--------------
Sysinv:
sysinv 2023-08-15 02:29:59.065 87035 ERROR sysinv.conductor.kube_app [-] Kube Application execution progress monitor timed out.: sysinv.common.exception.KubeAppProgressMonitorTimeout: Kube Application execution progress monitor timed out.
2023-08-15 02:29:59.065 87035 ERROR sysinv.conductor.kube_app Traceback (most recent call last):
2023-08-15 02:29:59.065 87035 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/conductor/kube_app.py", line 1699, in _make_fluxcd_operation_with_monitor
2023-08-15 02:29:59.065 87035 ERROR sysinv.conductor.kube_app rc = _check_progress()
2023-08-15 02:29:59.065 87035 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/conductor/kube_app.py", line 1677, in _check_progress
2023-08-15 02:29:59.065 87035 ERROR sysinv.conductor.kube_app time.sleep(1)
2023-08-15 02:29:59.065 87035 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/eventlet/greenthread.py", line 36, in sleep
2023-08-15 02:29:59.065 87035 ERROR sysinv.conductor.kube_app hub.switch()
2023-08-15 02:29:59.065 87035 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/eventlet/hubs/hub.py", line 298, in switch
2023-08-15 02:29:59.065 87035 ERROR sysinv.conductor.kube_app return self.greenlet.switch()
2023-08-15 02:29:59.065 87035 ERROR sysinv.conductor.kube_app sysinv.common.exception.KubeAppProgressMonitorTimeout: Kube Application execution progress monitor timed out.
2023-08-15 02:29:59.065 87035 ERROR sysinv.conductor.kube_app
sysinv 2023-08-15 02:29:59.067 87035 INFO sysinv.conductor.kube_app [-] lifecycle hook for application stx-openstack (1.0-1.stx.66-debian-stable-versioned) started {'lifecycle_type': 'fluxcd-request', 'relative_timing': 'post', 'operation': 'apply', 'extra': {'rc': False}}.
sysinv 2023-08-15 02:29:59.068 87035 INFO sysinv.conductor.kube_app [-] lifecycle hook for application stx-openstack (1.0-1.stx.66-debian-stable-versioned) started {'mode': 'manual', 'lifecycle_type': 'rbd', 'relative_timing': 'post', 'operation': 'apply', 'extra': {'was_applied': False}}.
sysinv 2023-08-15 02:29:59.069 87035 INFO sysinv.conductor.kube_app [-] lifecycle hook for application stx-openstack (1.0-1.stx.66-debian-stable-versioned) started {'mode': 'manual', 'lifecycle_type': 'resource', 'relative_timing': 'post', 'operation': 'apply', 'extra': {'was_applied': False}}.
sysinv 2023-08-15 02:29:59.315 87035 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.

helm list:
sysadmin@controller-0 ~(keystone_admin)]$ helm -n openstack list
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/admin.conf
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
osh-openstack-cinder openstack 1 2023-08-15 01:41:03.271899803 +0000 UTC deployed cinder-0.3.11 v1.0.0
osh-openstack-clients openstack 1 2023-08-15 01:49:59.52665254 +0000 UTC deployed clients-0.1.0 1.0
osh-openstack-fm-rest-api openstack 1 2023-08-15 01:49:59.493484735 +0000 UTC deployed fm-rest-api-0.1.0 1.0
osh-openstack-glance openstack 1 2023-08-15 01:39:03.110552241 +0000 UTC deployed glance-0.4.7 v1.0.0
osh-openstack-heat openstack 1 2023-08-15 01:45:35.456180804 +0000 UTC deployed heat-0.3.4 v1.0.0
osh-openstack-horizon openstack 1 2023-08-15 01:50:59.594332585 +0000 UTC deployed horizon-0.3.9 v1.0.0
osh-openstack-ingress openstack 1 2023-08-15 01:30:02.900970814 +0000 UTC deployed ingress-0.2.15 v0.42.0
osh-openstack-keystone openstack 1 2023-08-15 01:36:03.300620213 +0000 UTC deployed keystone-0.3.2 v1.0.0
osh-openstack-libvirt openstack 1 2023-08-15 01:45:33.336824459 +0000 UTC deployed libvirt-0.1.20 v1.0.0
osh-openstack-mariadb openstack 1 2023-08-15 01:30:33.264967246 +0000 UTC deployed mariadb-0.2.31 v10.6.7
osh-openstack-memcached openstack 1 2023-08-15 01:34:03.025155341 +0000 UTC deployed memcached-0.1.13 v1.5.5
osh-openstack-neutron openstack 1 2023-08-15 01:45:37.243162742 +0000 UTC deployed neutron-0.3.12 v1.0.0
osh-openstack-nginx-ports-control openstack 1 2023-08-15 01:30:02.876177332 +0000 UTC deployed nginx-ports-control-0.1.0
osh-openstack-nova openstack 1 2023-08-15 01:45:33.451301246 +0000 UTC deployed nova-0.3.13 v1.0.0
osh-openstack-nova-api-proxy openstack 1 2023-08-15 01:45:33.353447588 +0000 UTC deployed nova-api-proxy-0.1.0 1.0
osh-openstack-openvswitch openstack 1 2023-08-15 01:45:33.756418406 +0000 UTC deployed openvswitch-0.1.14 v1.0.0
osh-openstack-pci-irq-affinity-agent openstack 1 2023-08-15 01:53:59.70535787 +0000 UTC deployed pci-irq-affinity-agent-0.1.0
osh-openstack-placement openstack 1 2023-08-15 01:44:03.54592114 +0000 UTC deployed placement-0.3.5 v1.0.0
osh-openstack-rabbitmq openstack 1 2023-08-15 01:34:33.350649643 +0000 UTC deployed rabbitmq-0.1.27 v3.9.0

Test Activity
-------------
Sanity

Workaround
----------
N/A

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/openstack-armada-app/+/891512

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/antelope
Review: https://review.opendev.org/c/starlingx/openstack-armada-app/+/891448

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Lucas de Ataides Barreto <email address hidden>" on branch: f/antelope
Review: https://review.opendev.org/c/starlingx/openstack-armada-app/+/891448
Reason: Abandoning as 890570 will be reverted

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Lucas de Ataides Barreto <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/openstack-armada-app/+/891512
Reason: Abandoning as 890570 will be reverted

[Lucas de Ataides Barreto (ldeataid)]

This issue was introduced on https://opendev.org/starlingx/config/commit/612e70f4e20498a2f6ea528a6aa1072840287477.
Based on the comments on https://review.opendev.org/c/starlingx/openstack-armada-app/+/891448, that change will be reverted so a fix can be worked on.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/891449

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/891449
Committed: https://opendev.org/starlingx/config/commit/ac59a9abc5517bff264fc08a4a9195ac5ff28090
Submitter: "Zuul (22348)"
Branch: master

commit ac59a9abc5517bff264fc08a4a9195ac5ff28090
Author: Joshua Reed <email address hidden>
Date: Tue Aug 15 18:30:21 2023 +0000

    Revert "Better control over chart list generation."

    This reverts commit 612e70f4e20498a2f6ea528a6aa1072840287477.

    Reason for revert: Breaks dynamically enable/disabled charts orchestrated from the helm plugins.

    Change-Id: I26b7a737de217b87b6152f95364d434616c26265
    Closes-Bug: 2031401
    Signed-off-by: Joshua Reed <email address hidden>