Helm overrides are not present for backup or SX upgrades

[David Sullivan]

Brief Description
-----------------

Helm overrides are not included in backups and are not present for restore or SX upgrades. This causes SX upgrades to stay in a config out of date state.

Severity
--------
Major

Steps to Reproduce
------------------
Perform a SX upgrade.
Apply the upgrade playbook
Unlock controller-0

Expected Behavior
------------------
Config out of date alarms clear

Actual Behavior
----------------
Config out of date alarms do not clear

Reproducibility
---------------
Reproducible

System Configuration
--------------------
AIO-SX for upgrades, likely all config for B&R

Branch/Pull Time/Commit
-----------------------
2020-06-15_20-00-00

Last Pass
---------
Unknown

Timestamp/Logs
--------------
sysinv 2020-06-17 20:34:25.143 88284 ERROR sysinv.conductor.kube_app [-] Missing the following overrides: [u'/opt/platform/helm/20.06/cert-manager/1.0-0/cert-manager-cert-manager.yaml']
sysinv 2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: 'NoneType' object is not iterable: TypeError: 'NoneType' object is not iterable
2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last):
2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks
2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task task(self, context)
2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 5024, in _conductor_audit
2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task self._controller_config_active_apply(context)
2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4753, in _controller_config_active_apply
2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task context, config_uuid, config_dict)
2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 8759, in _config_apply_runtime_manifest
2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task self.evaluate_app_reapply(context, app_name)
2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 11075, in evaluate_app_reapply
2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task app.sync_overrides_dir, app.charts, app.name, None)
2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task TypeError: 'NoneType' object is not iterable
2020-06-17 20:34:25.143 88284 ERROR sysinv.openstack.common.periodic_task

Test Activity
-------------
Feature Testing

Workaround
----------
Manually backup and restore helm overrides

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/737343

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/737344

[Ghada Khalil (gkhalil)]

stx.4.0 / high priority - required for systems upgrade support. overrides are lost of simplex upgrades.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/737343
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=807597b2818830bdd7de1cf841874a3d503286e9
Submitter: Zuul
Branch: master

commit 807597b2818830bdd7de1cf841874a3d503286e9
Author: David Sullivan <email address hidden>
Date: Mon Jun 22 14:01:30 2020 -0400

    Add helm overrides to backup and restore

    Backup and restore the helm overrides directory. Ensure it is included
    in the upgrade data migration.

    Partial-Bug: 1884579
    Change-Id: I73ac2fcbd2019069ff225ed90968a05465acce43
    Signed-off-by: David Sullivan <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/737344
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=7f6ebceb8bd925988e25e29f60a0829d819ebc5e
Submitter: Zuul
Branch: master

commit 7f6ebceb8bd925988e25e29f60a0829d819ebc5e
Author: David Sullivan <email address hidden>
Date: Mon Jun 22 13:53:03 2020 -0400

    Migrate helm overrides in SX upgrades

    Now that the helm overrides are added to the upgrade_data, migrate them
    as part of the SX upgrade.

    Change-Id: I2327568ce0adea5f17854abcf1356948b587fb78
    Depends-On: https://review.opendev.org/737343
    Closes-Bug: 1884579
    Signed-off-by: David Sullivan <email address hidden>