platform-integ-apps does not apply, it remains in uploaded state

[Alexandru Dimofte]

Brief Description
-----------------
The platform-integ-apps was stuck in uploading state. It doesn't apply. This happened on all configurations(simplex, duplex, standard, standard-ext) during the daily sanity execution on 20th of January 2021.

Severity
--------
Critical: System did not recover

Steps to Reproduce
------------------
Installation of any configuration(simplex, duplex, standard, standard-ext)

Expected Behavior
------------------
platform-integ-apps automatically uploads and applies

Actual Behavior
----------------
Stuck in uploading state

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
One node system, Two node system, Multi-node system, Dedicated storage

Branch/Pull Time/Commit
-----------------------
master

Last Pass
---------
19th January 2021

Timestamp/Logs
--------------
Will be attached the collect output.

Test Activity
-------------
Daily Sanity

Workaround
----------
-

[Alexandru Dimofte (adimofte)]

I attached the collected logs.

[Alexandru Dimofte (adimofte)]

I also observed this on /var/log/sysinv.log:

': u'check', 'relative_timing': u'pre', 'mode': u'auto', 'operation': u'apply', 'extra': {}}.
sysinv 2021-01-20 13:16:11.192 106782 INFO sysinv.conductor.manager [-] Auto-apply failed prerequisites for platform-integ-apps: Automatic apply is disabled for platform-integ-apps.
sysinv 2021-01-20 13:16:11.196 106782 INFO sysinv.conductor.kube_app [-] lifecycle hook for application oidc-auth-apps (1.0-29) started {'lifecycle_type': u'check', 'relative_timing': u'pre', 'mode': u'auto', 'operation': u'apply', 'extra': {}}.

[Ghada Khalil (gkhalil)]

Assigning to Dan Voiculeasa to investigate; this stx email thread is pointing to his commit as a potential cause:
http://lists.starlingx.io/pipermail/starlingx-discuss/2021-January/010596.html

[Ghada Khalil (gkhalil)]

stx.5.0 / critical - issue causes red sanity

[Dan Voiculeasa (dvoicule)]

Build issue, stx-platform-helm didn't detect the changes in python-k8sapp-platform, thus python-k8sapp-platform was not rebuilt, thus an older version of the app was used. A build was triggered again which produced the newer version of the python-k8sapp-platform package.

We have a commit to avoid the same in the future. platform-armada-apps: https://review.opendev.org/c/starlingx/platform-armada-app/+/771876

Probably all the apps need to be changed in a same manner.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/platform-armada-app/+/792234

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/platform-armada-app/+/792234
Committed: https://opendev.org/starlingx/platform-armada-app/commit/f3bb236173d8bb336513a868d36f601a23ab87dc
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 66fa48f04d926848a37f5fbb7689cf4b114cb3ba
Author: Pedro Henrique Linhares <email address hidden>
Date: Tue Apr 6 21:45:28 2021 -0300

    Update helm charts config maps after sx-dx migration with new CEPH monitors

    This commit adds annotations that allows config maps to be recreated
    after ceph monitor IP changes due to DX migration so that existing
    StorageClasses can get a reference to the correct monitor. StorageClasses
    and provisioners are recreated during platform-integ-apps auto re-apply.

    Story: 2008587
    Task: 42242

    Signed-off-by: Pedro Linhares <email address hidden>
    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/783727
    Change-Id: I9cedc70326e92796f03520deed7f0857e119257f

commit 9d45149d29b8c05808bcd7e5a129f135b3386931
Author: Isac Souza <email address hidden>
Date: Mon Apr 5 11:07:21 2021 -0300

    Use new method for setting num of replicas

    Use the new _num_replicas_for_platform_app method from the
    helm base class to set the number of replicas in the chart.
    The new method will return the number of provisioned
    controllers with a minimum of 1.

    Tested by building an ISO and installing the armada apps.

    Partial-Bug: 1922278
    Signed-off-by: Isac Souza <email address hidden>
    Change-Id: Idb3c93274a1cb5c410d885d459784382525427a0

commit 1021d50142af6422c9a3f0853f4b7c525e724ab8
Author: Daniel Safta <email address hidden>
Date: Wed Mar 24 12:56:29 2021 +0000

    Removed extra serviceAccount from cephfs-provisioner

    cephfs-provisioner may need to create new resources
    in the kubernetes cluster. It was granted access to
    some of the resources including namespaces but when
    https://review.opendev.org/c/starlingx/platform-armada-app/+/778746
    got merged the serviceAccount was changed.

    I have updated the serviceAccount with access to
    creating new namespaces and secrets.

    The serviceAccount that was initially used to create
    namespaces and secrets is not needed anymore, so I
    have removed it.

    Closes-bug: 1921197
    Change-Id: I3c683776f3ecaf9c78d1a6b5b1108e9582497dde
    Signed-off-by: Daniel Safta <email address hidden>

commit 45fd0a6b2cd2dbaaf7ff21ef989824377c5b17dc
Author: Robert Church <email address hidden>
Date: Thu Mar 18 11:43:21 2021 -0400

    Build: Isolate platform plugins to an app specific directory

    When building the stx-platform-helm RPM for platform-integ-apps the helm
    plugins are installed in a location that could be populated with other
    app plugins if their spec files are not properly set up.

    Adjust the spec to provide an app specific location for the plugins to
    ensure that no other app plugins are included in the application tarball

    Closes-Bug: #1920066
    Change-Id: Id24227cd100a3c29809f1dd01f61ea717...