AIO - platform-integ-apps not applied

[Cristopher Lemus]

Brief Description
-----------------
On All-In-One simplex initial configuration, platform-integ-apps remain on "uploaded" status.

Severity
--------
Critical.

Steps to Reproduce
------------------
Follow up docs to install and configure a simplex configuration.

Expected Behavior
------------------
After certain conditions are met, platform-integ-apps is automatically applied.

Actual Behavior
----------------
Uncertain missing conditions are not met, platform-integ-apps remains on "uploaded" status.

Reproducibility
---------------
100%

System Configuration
--------------------
AIO Simplex

Branch/Pull Time/Commit
-----------------------
BUILD_ID="20191105T000000Z"

Last Pass
---------
Passed with yesterday build, 20191104T000000Z.

Timestamp/Logs
--------------
Full collect attached, some details on: http://paste.openstack.org/show/785822/

Test Activity
-------------
Sanity.

[Cristopher Lemus (cjlemusc)]

With the following build, platform-integ-apps was applied automatically:

BUILD_ID="20191106T023000Z"

[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+---------+-----------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+---------+-----------+
| platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-19-centos-stable-versioned | armada-manifest | stx-openstack.yaml | applied | completed |
+---------------------+--------------------------------+-------------------------------+--------------------+---------+-----------+

I'll update if it appears again. Do we have a list of required "conditions"? That will help for future troubleshooting.

[Cristopher Lemus (cjlemusc)]

Second build, BUILD_ID="20191107T023000Z", where this issue has not been replicated.

[Ovidiu Poncea (ovidiuponcea)]

One precondition is most likely missing:

2019-11-05 06:43:14.069 109677 WARNING sysinv.common.ceph [req-327dde66-b7fb-48a6-9de3-3eb12a9938bb None None] Failed to create flag file: /opt/platform/sysinv/19.09/.crushmap_applied. Reason: [Errno 2] No such file or directory: '/opt/platform/sysinv/19.09/.crushmap_applied'

Problem is sysinv/19.09 does not exist in /opt/platform

This is caused by https://review.opendev.org/#/c/692439 and is intermittent as sometimes this subfolder is created in the right order.

Solution is to partially revert the commit or make sure that we always create the necessary structure when creating files in this subfolder. The 2nd solution is tricky as there are other parts in the code that assumes this folder exists.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/694774

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694774
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=dcdeea0254149233f1c0e5a6536a561cf7453cec
Submitter: Zuul
Branch: master

commit dcdeea0254149233f1c0e5a6536a561cf7453cec
Author: Ovidiu Poncea <email address hidden>
Date: Mon Nov 18 15:48:23 2019 +0200

    Fix missing content in /opt/platform/sysinv/19.09

    Due to changes in https://review.opendev.org/#/c/692439 and
    https://review.opendev.org/#/c/691714, sysinv/19.09 subfolder
    of /opt/platform is no longer created, nor its content.
    This breaks the initial assumptions that:
    1. This folder exists => is ok to just create files there
    2. The content of this folder is recreated on each unlock

    To return #1 assumption we now create the folder in advance
    and for #2 we back-up and restore its content.

    Change-Id: I8dd686a66fcc62bbb05b72fda56e86c353d25fee
    Closes-Bug: 1851424
    Closes-Bug: 1852065
    Closes-Bug: 1852127
    Signed-off-by: Ovidiu Poncea <email address hidden>