Brief Description
-----------------
After stx-openstack is successfully applied, the application unexpectedly becomes "not applied" after reapply the application failed.
dbmon service is continuously running after stx-openstack reapply failed, however it couldn't reach mariadb cluster as stx-openstack is no long applied/running.
This issue is split from https://bugs.launchpad.net/starlingx/+bug/1836075
Severity
--------
Major
Steps to Reproduce
------------------
Ref to https://bugs.launchpad.net/starlingx/+bug/1836075
for the steps to reproduce
Expected Behavior
-----------------
When stx-openstack is successfully applied, after a subsequent reapply failed, the system is expected to:
1. stx-openstack will remain as "active" and run with latest successful configuration.
1.1 mariadb cluster will be running and dbmon service will be able to access and monitor the state of mariadb cluster
2. subsequent reapply attempt shall not be rejected
Actual Behavior
---------------
1. stx-openstack application is still active but not running.
1.1 dbmon repeatedly report it could not access mariadb.
2. subsequent reapply was rejected by sysinv: "stx-openstack system app is present but not applied, skipping re-apply"
Reproducibility
---------------
Seen once
System Configuration
--------------------
Two node system
Lab-name: IP_5-6
Branch/Pull Time/Commit
-----------------------
stx master as of 20190708T233000Z
Last Pass
---------
Unknown
Timestamp/Logs
--------------
2019-07-10 08:10:05.824 110608 INFO sysinv.conductor.kube_app [-] Application stx-openstack (1.0-17-centos-stable-versioned) apply completed.
...
2019-07-10 11:01:21.802 110608 INFO sysinv.conductor.kube_app [-] Application stx-openstack (1.0-17-centos-stable-versioned) apply started.
2019-07-10 11:08:37.690 110608 ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/stx-openstack/1.0-17-centos-stable-versioned/stx-openstack-stx-openstack.yaml. See /var/log/armada stx-openstack-apply.log for details.
2019-07-10 11:08:37.696 110608 INFO sysinv.conductor.kube_app [-] Exiting progress monitoring thread for app stx-openstack
2019-07-10 12:17:52.178 228896 INFO sysinv.api.controllers.v1.host [-] stx-openstack system app is present but not applied, skipping re-apply
| 2019-07-10T08:11:06.772 | 290 | service-scn | dbmon | unknown | enabled-active | audit success
| 2019-07-10T11:05:46.432 | 295 | service-scn | dbmon | enabled-active | disabling | audit failed
The description of this Launchpad is confusing, the application did not unexpectedly become "not applied" after the reapply failed. There is no such status as "not applied".
Currently, the stx-openstack application automatic reapply is triggered when:
a) any host in the cluster is unlocked
b) any host in the cluster is deleted
in order to pick up chart override updates and/or redistribute pods.
If stx-openstack app does not exist or the app does not have an "applied" status, the reapply is skipped.
The expected behaviors are incorrect:
1. stx-openstack can not be reapplied on both controllers. It can only be reapplied on the active controller
2. after a reapply failed, the stx-openstack status would be set to "apply-failed" it would not remain as "applied" and the service(s) which belong to the failed chart(s) would not be functional and so would services that depend on these failed services.
3. mariadb cluster would not be running if osh-openstack-
The log "snippet" in this LP has a July 7th, 2019 timestamp whereas the logs of LP1829289 have the timestamp of May 15th, 2019. Furthermore, the 2 LPs show 2 different system configurations. I can't make a connection between these 2 LPs based on given info. It appeared to me that the log snippet in this LP was grabbed from a duplex lab used for sanity with a known failed test case.
Please provide meaningful info that aids the investigation.