system application-upload fails with No module named systemconfig.helm_plugins.stx_openstack

[Maria Guadalupe Perez Ibara]

Brief Description
-----------------
system application-upload fails with No module named systemconfig.helm_plugins.stx_openstack

Severity
--------
Critical

Steps to Reproduce
------------------
Follow up documentation: https://wiki.openstack.org/wiki/StarlingX/Containers/Installation#Stage_application_for_deployment

During the step
source /etc/platform/openrc
system application-upload stx-openstack-1.0-11.tgz
system application-list

Expected Behavior
------------------
Application loaded

Actual Behavior
------------------
Application remains on uploading status

Reproducibility
---------------
The issue is 100% reproducible in simplex

System Configuration
--------------------
simplex BM and virtual

Branch/Pull Time/Commit
-----------------------
OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190505T233000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="92"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-05-05 23:30:00 +0000"

Timestamp/Logs
--------------
Application remains on uploading status:

[wrsroot@controller-0 ~(keystone_admin)]$ system application-list
+---------------+-----------------------------+-----------------+---------------+-----------+----------+
| application | version | manifest name | manifest file | status | progress |
+---------------+-----------------------------+-----------------+---------------+-----------+----------+
| stx-openstack | 1.0-11-centos-stable-latest | armada-manifest | manifest.yaml | uploading | None |
+---------------+-----------------------------+-----------------+---------------+-----------+----------+

On `/var/log/sysinv.log` the following error is logged:

2019-05-06 15:06:28.767 73153 ERROR stevedore.extension [-] Could not load 'stx-openstack': No module named systemconfig.helm_plugins.stx_openstack

Test Activity
-------------
Sanity

[Al Bailey (albailey1974)]

The error flagged in this launchpad is benign, although it should be fixed. That error log always occurs and adds excess noise in the logs.

The failure here is that the sysinv process is terminated while the application is being uploaded, and the status in the DB is never updated, so there is no feedback to the user that the process will ever complete.

Here are the logs related to that termination of the process

2019-05-06 15:12:38.790 75366 INFO oslo_service.service [-] Caught SIGTERM, stopping children
2019-05-06 15:12:38.791 75366 INFO oslo.service.wsgi [-] Stopping WSGI server.
2019-05-06 15:12:38.791 75366 INFO oslo_service.service [-] Waiting on 2 children to exit
2019-05-06 15:12:38.792 75739 INFO oslo.service.wsgi [-] Stopping WSGI server.
2019-05-06 15:12:38.792 75738 INFO oslo.service.wsgi [-] Stopping WSGI server.
2019-05-06 15:12:38.804 75366 INFO oslo_service.service [-] Child 75739 exited with status 0
2019-05-06 15:12:38.805 75366 INFO oslo_service.service [-] Child 75738 exited with status 0

[Ghada Khalil (gkhalil)]

Marking as release gating; needs further investigation.
The reporter says the issue is 100% reproducible (although this was not reported in the May 9th sanity), so marking as high priority

As Al noted, someone needs to investigate by the sysinv process was terminated during the application upload.

[Lin Shuicheng (shuicheng)]

I could find the error log in my deployment also. And it is benign as Al mentioned.
Will have a check with it.

[Cindy Xie (xxie1)]

please check the sysinv.log from https://bugs.launchpad.net/starlingx/+bug/1826912, wondering if you're seeing the similar issue?
and can we reproduce it in latest ISO? we've fixed some application-apply issue due to OOM (1826308, 1826445) and Ceph Rest API issue (1827521)?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/660893

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/660893
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=34c627eadc8ee589f28e7961e8648adbd1d15f80
Submitter: Zuul
Branch: master

commit 34c627eadc8ee589f28e7961e8648adbd1d15f80
Author: Shuicheng Lin <email address hidden>
Date: Thu May 23 14:14:37 2019 +0800

    suppress stevedore error message when load helm plugin

    There is error message "ERROR stevedore.extension [-] Could not load 'stx-
    openstack': No module named systemconfig.helm_plugins.stx_openstack".
    The error message is expected, since the package/module name
    (systemconfig.helm_plugins.stx_openstack/platform_integ_apps) defined
    in setup.cfg of sysinv doesn't have corresponding the same name module.
    They are implemented in sysinv.
    Suppress the error message to avoid cause confusion.

    Closes-Bug: 1827952
    Change-Id: I75920e193ff8f6b28dc68ed3f414ac774346e339
    Signed-off-by: Shuicheng Lin <email address hidden>