platform-integ-apps was stuck in uploading state

[Brent Rowsell]

Brief Description
-----------------
The platform-integ-apps was stuck in uploading state.
I was able to recover by going into the DB and deleting the app. It uploaded and applied on the next audit cycle.

Severity
--------
Critical
System did not recover

Steps to Reproduce
------------------
Installation of standard system

Expected Behavior
------------------
platform-integ-apps automatically uploads and applies

Actual Behavior
----------------
Stuck in uploading state

Reproducibility
---------------
Not sure

System Configuration
--------------------
Standard system

Branch/Pull Time/Commit
-----------------------
BUILD_DATE="2019-05-18 06:38:29 -0400"

Last Pass
---------
Don't know

Timestamp/Logs
--------------
2019-05-20 14:56:32.959 102733 INFO sysinv.conductor.manager [req-4c39a974-66d7-48a0-8419-9ac02725a594 None None] SYS_I Clear system config alarm: controller-0 target config b1bc2657-5852-4aa9-947a-ccdb4c0166dc
2019-05-20 14:56:32.978 94511 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host: {'config_applied': 'b1bc2657-5852-4aa9-947a-ccdb4c0166dc', 'first_report': True, 'availability': 'available', 'iscsi_initiator_name': 'iqn.1994-05.com.redhat:1ecda3b822f6'}
2019-05-20 14:56:36.531 102733 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Creating...
2019-05-20 14:56:36.674 102733 INFO sysinv.common.utils [-] Checksum file is included and validated.
2019-05-20 14:56:36.676 102733 INFO sysinv.api.controllers.v1.kube_app [-] No patch required for application platform-integ-apps (1.0-5).
2019-05-20 14:56:37.241 102733 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://192.168.204.2:5491/v1/query/ hdr:None payload:None
2019-05-20 14:56:38.097 102733 WARNING sysinv.api.controllers.v1.rest_api [-] HTTP Error e.code=503 e=HTTP Error 503: Service Unavailable
2019-05-20 14:56:38.097 102733 ERROR sysinv.api.controllers.v1.kube_app [-] No response from patch api: 'NoneType' object has no attribute '__getitem__'
2019-05-20 14:56:38.097 102733 ERROR sysinv.conductor.manager [-] 'NoneType' object has no attribute '__getitem__'. Communication Error with patching subsystem. Preventing managed application

Test Activity
-------------
Other

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/660248

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/660248
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=8a20895fab85bfa57244545a3ebd65a939ff841e
Submitter: Zuul
Branch: master

commit 8a20895fab85bfa57244545a3ebd65a939ff841e
Author: Robert Church <email address hidden>
Date: Mon May 20 23:17:17 2019 -0400

    Check patching api prior to uploading managed app

    The patching operation check is happening too late when uploading a
    platform managed application. If the patching API is not responding, the
    managed application will be created in the database and end up stuck in
    the uploading state as the upload will not proceed.

    Move the check to the proper location to avoid prematurely creating the
    application if it can't be loaded until patch dependencies are verified.

    Also, prior to spawning the greenthread for application apply, save the
    state to 'applying'. This will prevent multiple audit cycles from
    starting greenthreads for the same action.

    Change-Id: Iada51635a6965acb404be05cfd4d9ef167aebe9f
    Closes-Bug: #1829791
    Signed-off-by: Robert Church <email address hidden>