Missing user overrides during/after application-update

[Kevin Smith]

Brief Description
-----------------
After issuing an application-update with a new application version tarball, user overrides set on the previous app version are not present on the new version. There also appears to be no facility to set user overrides on the new application version.

Severity
--------
Major: application-update of stx-monitor (or any application) is not possible for any configuration requiring user overrides (distributed cloud, off-box monitoring) requiring removal of old application and application of new one.

Steps to Reproduce
------------------
#application stx-monitor version 1.0-1 installed with below user overrides.

[sysadmin@controller-1 ~(keystone_admin)]$ system helm-override-show stx-monitor elasticsearch-master monitor
+--------------------+--------------------------------------------------------------------------+
| Property | Value |
+--------------------+--------------------------------------------------------------------------+
| attributes | enabled: true |
| | |
| combined_overrides | esJavaOpts: -Djava.net.preferIPv6Addresses=true -Xmx256m -Xms256m |
| | masterService: mon-elasticsearch-data-headless, mon-elasticsearch-master |
| | minimumMasterNodes: 1 |
| | nodeGroup: master |
| | nodeSelector: |
| | elastic-master: enabled |
| | replicas: 1 |
| | resources: |
| | limits: |
| | cpu: "1" |
| | requests: |
| | cpu: 25m |
| | memory: 512Mi |
| | volumeClaimTemplate: |
| | accessModes: |
| | - ReadWriteOnce |
| | resources: |
| | requests: |
| | storage: 4Gi |
| | storageClass: general |
| | |
| name | elasticsearch-master |
| namespace | monitor |
| system_overrides | esJavaOpts: -Djava.net.preferIPv6Addresses=true -Xmx256m -Xms256m |
| | minimumMasterNodes: 1 |
| | nodeGroup: master |
| | nodeSelector: {elastic-master: enabled} |
| | replicas: 1 |
| | resources: |
| | limits: {cpu: '1'} |
| | requests: {cpu: 25m, memory: 512Mi} |
| | volumeClaimTemplate: |
| | accessModes: [ReadWriteOnce] |
| | resources: |
| | requests: {storage: 4Gi} |
| | storageClass: general |
| | |
| user_overrides | masterService: mon-elasticsearch-data-headless, mon-elasticsearch-master |
| | |
+--------------------+--------------------------------------------------------------------------+

# Application is updated:

system application-update ./stx-monitor-1.0-2.tgz -n stx-monitor -v 1.0-2

# While application is updating pods, no user overrides are present:

[sysadmin@controller-1 ~(keystone_admin)]$ system helm-override-show stx-monitor elasticsearch-master monitor
+--------------------+-------------------------------------------------------------------+
| Property | Value |
+--------------------+-------------------------------------------------------------------+
| attributes | enabled: true |
| | |
| combined_overrides | esJavaOpts: -Djava.net.preferIPv6Addresses=true -Xmx256m -Xms256m |
| | minimumMasterNodes: 1 |
| | nodeGroup: master |
| | nodeSelector: |
| | elastic-master: enabled |
| | replicas: 1 |
| | resources: |
| | limits: |
| | cpu: "1" |
| | requests: |
| | cpu: 25m |
| | memory: 512Mi |
| | volumeClaimTemplate: |
| | accessModes: |
| | - ReadWriteOnce |
| | resources: |
| | requests: |
| | storage: 4Gi |
| | storageClass: general |
| | |
| name | elasticsearch-master |
| namespace | monitor |
| system_overrides | esJavaOpts: -Djava.net.preferIPv6Addresses=true -Xmx256m -Xms256m |
| | minimumMasterNodes: 1 |
| | nodeGroup: master |
| | nodeSelector: {elastic-master: enabled} |
| | replicas: 1 |
| | resources: |
| | limits: {cpu: '1'} |
| | requests: {cpu: 25m, memory: 512Mi} |
| | volumeClaimTemplate: |
| | accessModes: [ReadWriteOnce] |
| | resources: |
| | requests: {storage: 4Gi} |
| | storageClass: general |
| | |
| user_overrides | None |
+--------------------+-------------------------------------------------------------------+

# application-update fails due to missing overrides and is rolled back.

# during the update, duplicated empty entries for the application charts appear in the helm_overrides table for the application:

sysinv=# select * from helm_overrides;
         created_at | updated_at | deleted_at | id | name | namespace | user_overrides | app_id
| system_overrides
----------------------------+----------------------------+------------+-----+-----------------------+-------------+--------------------------------------------------------------------------+--------
+-------------------
 2019-10-15 22:06:11.614174 | 2019-10-15 22:06:11.626425 | | 1 | ceph-pools-audit | kube-system | | 1
| {"enabled": true}
 2019-10-15 22:06:11.638136 | 2019-10-15 22:06:11.647411 | | 2 | rbd-provisioner | kube-system | | 1
| {"enabled": true}
 2019-10-22 20:31:11.281329 | 2019-10-22 21:41:51.207121 | | 87 | elasticsearch-client | monitor | masterService: mon-elasticsearch-data-headless, mon-elasticsearch-master+| 13
| {"enabled": true}
                            | | | | | | |
|
 2019-10-22 20:31:11.337741 | 2019-10-22 21:41:51.243799 | | 88 | metricbeat | monitor | | 13
| {"enabled": true}
 2019-10-22 20:31:11.425812 | 2019-10-22 21:41:51.279535 | | 89 | elasticsearch-master | monitor | masterService: mon-elasticsearch-data-headless, mon-elasticsearch-master+| 13
| {"enabled": true}
                            | | | | | | |
|
 2019-10-22 20:31:11.469335 | 2019-10-22 21:41:51.311842 | | 90 | nginx-ingress | monitor | | 13
| {"enabled": true}
 2019-10-22 20:31:11.552011 | 2019-10-22 21:41:51.342785 | | 91 | elasticsearch-curator | monitor | | 13
| {"enabled": true}
 2019-10-22 20:31:11.60172 | 2019-10-22 21:41:51.361469 | | 92 | logstash | monitor | | 13
| {"enabled": true}
 2019-10-23 12:41:11.491803 | 2019-10-23 12:41:11.514338 | | 113 | filebeat | monitor | | 16
| {"enabled": true}
 2019-10-23 12:41:11.535739 | 2019-10-23 12:41:11.549117 | | 114 | elasticsearch-data | monitor | | 16
| {"enabled": true}
 2019-10-23 12:41:11.574275 | 2019-10-23 12:41:11.597312 | | 115 | kube-state-metrics | monitor | | 16
| {"enabled": true}
 2019-10-23 12:41:11.618386 | 2019-10-23 12:41:11.629681 | | 116 | kibana | monitor | | 16
| {"enabled": true}
 2019-10-23 12:41:11.6566 | 2019-10-23 12:41:11.705906 | | 117 | elasticsearch-client | monitor | | 16
| {"enabled": true}
 2019-10-23 12:41:11.728297 | 2019-10-23 12:41:11.740433 | | 118 | metricbeat | monitor | | 16
| {"enabled": true}
 2019-10-23 12:41:11.758631 | 2019-10-23 12:41:11.769476 | | 119 | elasticsearch-master | monitor | | 16
| {"enabled": true}
 2019-10-23 12:41:11.783039 | 2019-10-23 12:41:11.792855 | | 120 | nginx-ingress | monitor | | 16
| {"enabled": true}
 2019-10-23 12:41:11.810651 | 2019-10-23 12:41:11.822849 | | 121 | elasticsearch-curator | monitor | | 16
| {"enabled": true}
 2019-10-23 12:41:11.846112 | 2019-10-23 12:41:11.856278 | | 122 | logstash | monitor | | 16
| {"enabled": true}
 2019-10-22 20:31:11.128912 | 2019-10-22 21:41:51.069156 | | 83 | filebeat | monitor | | 13
| {"enabled": true}
 2019-10-22 20:31:11.1513 | 2019-10-22 21:41:51.093696 | | 84 | elasticsearch-data | monitor | masterService: mon-elasticsearch-data-headless, mon-elasticsearch-master+| 13
| {"enabled": true}
                            | | | | | | |
|
 2019-10-22 20:31:11.18936 | 2019-10-22 21:41:51.151361 | | 85 | kube-state-metrics | monitor | | 13
| {"enabled": true}
 2019-10-22 20:31:11.233296 | 2019-10-22 21:41:51.18361 | | 86 | kibana | monitor | | 13
| {"enabled": true}
(22 rows)

Expected Behavior
------------------
Expectation is for the user overrides set on the old app version to be present on the new updated version

Actual Behavior
----------------
User overrides are not present on new updated app version.

Reproducibility
---------------
100%

System Configuration
--------------------
Any

Branch/Pull Time/Commit
-----------------------
Master Oct 22 2019.

Last Pass
---------
N/A

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Developer Testing

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/691419

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/691420

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/691420
Committed: https://git.openstack.org/cgit/starlingx/monitor-armada-app/commit/?id=b572232c9630a67ea7f8bde4c315b7b8dc198fe6
Submitter: Zuul
Branch: master

commit b572232c9630a67ea7f8bde4c315b7b8dc198fe6
Author: Angie Wang <email address hidden>
Date: Fri Oct 25 14:30:44 2019 -0400

    Add metadata for stx-monitor

    The metadata "maintain_user_overrides" is used to determine whether
    the user overrides will be reused or not when updating the application
    to a new version.

    Set it true for stx-monitor.

    Change-Id: Icfb5963f6396d099a61ca9d76963dc63531595b5
    Partial-Bug: 1849486
    Signed-off-by: Angie Wang <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/691419
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=9a218ed75f5dc907dc302e69a85acf6743c85a84
Submitter: Zuul
Branch: master

commit 9a218ed75f5dc907dc302e69a85acf6743c85a84
Author: Angie Wang <email address hidden>
Date: Thu Oct 24 17:09:44 2019 -0400

    Support to preserve user overrides when updating app to a new version

    This commit updates to provide the ability to preserve the user
    overrides from the old version when updating app to a new version.

    The key "maintain_user_overrides" in application metadata file
    indicates if the user overrides will be reused or not during
    update, user can specify --reuse-user-overrides <true/false>
    to override the metadata preference specified by the application.

    This commit also updates armada log for each application operation
    to include timestamp in its log name and keep the maximum 10 logs
    for each type of log.

    Change-Id: I1c077017c8a60dc63d27135ff5dd966aa38387f8
    Partial-Bug: 1849486
    Signed-off-by: Angie Wang <email address hidden>