Platform upgrade with stx-openstack installed failed due to system config: KeyError: 'keystone'

[Thiago Paiva Brito]

*+Brief Description+*

Attempted to upgrade StarlingX 5.0 + stx-openstack to StarlingX 6.0 but failed due to data-migration-failed.

*+Severity+*

Major.

*+Steps to Reproduce+*
 # Install StarlingX 5.0 in DX+ with stx-openstack
 # Run Ansible Backup
 # Copy 21.12 license/load/signature to controller-0
 # Confirm the system is healthy (system health-query-upgrade)
 # Start Upgrade on controller-1
 # FAILURE: It will fail due to data-migration-failed

*+Expected Behavior+*

Controller-1 upgraded and data migrated

*+Actual Behavior+*

Contoller-1 was upgraded to 6.0 but upgrade failed due to data-migration-failed.
{code:bash}
[sysadmin@controller-0 ~(keystone_admin)]$ system upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | c6c2af98-0774-45e4-8218-09371d2fd52f |
| state        | data-migration-failed                |
| from_release | 21.05                                |
| to_release   | 21.12                                |
+--------------+--------------------------------------+
{code}
*+Reproducibility+*

100% (2 out of 2).

*+System Configuration+*

DX+7

*+Branch/Pull Time/Commit+*

StarlingX 5.0 to Starlingx 6.0

stx-openstack 2022-01-13

*+Last Pass+*

None.

*+Timestamp/Logs+*
{code:java}
// collect --all
/folk/cgts_logs/CGTS-31524 {code}
{code:java}
// controller-1

2022-01-25T13:35:36.327 localhost controller_config[6832]: info sysinv 2022-01-25 13:35:36.276 38071 ERROR sysinv.puppet.puppet [-] failed to create system config: KeyError: 'keystone'
2022-01-25T13:35:36.345 localhost controller_config[6832]: info 2022-01-25 13:35:36.276 38071 ERROR sysinv.puppet.puppet Traceback (most recent call last):
2022-01-25T13:35:36.360 localhost controller_config[6832]: info 2022-01-25 13:35:36.276 38071 ERROR sysinv.puppet.puppet   File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 117, in update_system_config
2022-01-25T13:35:36.382 localhost controller_config[6832]: info 2022-01-25 13:35:36.276 38071 ERROR sysinv.puppet.puppet     config.update(puppet_plugin.obj.get_system_config())
2022-01-25T13:35:36.400 localhost controller_config[6832]: info 2022-01-25 13:35:36.276 38071 ERROR sysinv.puppet.puppet   File "/usr/lib64/python2.7/site-packages/sysinv/puppet/nfv.py", line 96, in get_system_config
2022-01-25T13:35:36.421 localhost controller_config[6832]: info 2022-01-25 13:35:36.276 38071 ERROR sysinv.puppet.puppet     keystone_auth_data = helm_data.get_keystone_auth_data()
2022-01-25T13:35:36.438 localhost controller_config[6832]: info 2022-01-25 13:35:36.276 38071 ERROR sysinv.puppet.puppet   File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 58, in _wrapper
2022-01-25T13:35:36.458 localhost controller_config[6832]: info 2022-01-25 13:35:36.276 38071 ERROR sysinv.puppet.puppet     return func(self, *args, **kwargs)
2022-01-25T13:35:36.473 localhost controller_config[6832]: info 2022-01-25 13:35:36.276 38071 ERROR sysinv.puppet.puppet   File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 936, in get_keystone_auth_data
2022-01-25T13:35:36.494 localhost controller_config[6832]: info 2022-01-25 13:35:36.276 38071 ERROR sysinv.puppet.puppet     keystone_operator = self.chart_operators[self.HELM_CHART_KEYSTONE]
2022-01-25T13:35:36.512 localhost controller_config[6832]: info 2022-01-25 13:35:36.276 38071 ERROR sysinv.puppet.puppet KeyError: 'keystone'
2022-01-25T13:35:36.526 localhost controller_config[6832]: info 2022-01-25 13:35:36.276 38071 ERROR sysinv.puppet.puppet
2022-01-25T13:35:36.538 localhost controller_config[6832]: info sysinv 2022-01-25 13:35:36.278 38071 CRITICAL sysinv [-] Unhandled error: KeyError: 'keystone'
2022-01-25T13:35:36.553 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv Traceback (most recent call last):
2022-01-25T13:35:36.567 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv   File "/usr/bin/sysinv-puppet", line 10, in <module>
2022-01-25T13:35:36.582 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv     sys.exit(main())
2022-01-25T13:35:36.459 localhost systemd[1]: notice controllerconfig.service: main process exited, code=exited, status=1/FAILURE
2022-01-25T13:35:36.594 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv   File "/usr/lib64/python2.7/site-packages/sysinv/cmd/puppet.py", line 77, in main
2022-01-25T13:35:36.612 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv     CONF.action.func(CONF.action.path)
2022-01-25T13:35:36.626 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv   File "/usr/lib64/python2.7/site-packages/sysinv/cmd/puppet.py", line 33, in create_system_config_action
2022-01-25T13:35:36.646 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv     operator.update_system_config()
2022-01-25T13:35:36.660 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv   File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 32, in _wrapper
2022-01-25T13:35:36.678 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv     func(self, *args, **kwargs)
2022-01-25T13:35:36.692 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv   File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 117, in update_system_config
2022-01-25T13:35:36.712 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv     config.update(puppet_plugin.obj.get_system_config())
2022-01-25T13:35:36.727 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv   File "/usr/lib64/python2.7/site-packages/sysinv/puppet/nfv.py", line 96, in get_system_config
2022-01-25T13:35:36.747 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv     keystone_auth_data = helm_data.get_keystone_auth_data()
2022-01-25T13:35:36.762 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv   File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 58, in _wrapper
2022-01-25T13:35:36.780 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv     return func(self, *args, **kwargs)
2022-01-25T13:35:36.794 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv   File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 936, in get_keystone_auth_data
2022-01-25T13:35:36.814 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv     keystone_operator = self.chart_operators[self.HELM_CHART_KEYSTONE]
2022-01-25T13:35:36.842 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv KeyError: 'keystone'
2022-01-25T13:35:36.854 localhost controller_config[6832]: info 2022-01-25 13:35:36.278 38071 ERROR sysinv
2022-01-25T13:35:36.459 localhost systemd[1]: notice Unit controllerconfig.service entered failed state.
2022-01-25T13:35:36.864 localhost controller_config[6832]: info Failed to update puppet hiera system config
2022-01-25T13:35:36.875 localhost controller_config[6832]: info 2022-01-25 13:35:36.374 7357 ERROR controllerconfig.upgrades.controller [-] Failed to update puppet hiera system config: Exception: Failed to update puppet hiera system config
2022-01-25T13:35:36.898 localhost controller_config[6832]: info 2022-01-25 13:35:36.374 7357 ERROR controllerconfig.upgrades.controller Traceback (most recent call last):
2022-01-25T13:35:36.915 localhost controller_config[6832]: info 2022-01-25 13:35:36.374 7357 ERROR controllerconfig.upgrades.controller   File "/usr/lib64/python2.7/site-packages/controllerconfig/upgrades/controller.py", line 956, in upgrade_controller
2022-01-25T13:35:36.940 localhost controller_config[6832]: info 2022-01-25 13:35:36.374 7357 ERROR controllerconfig.upgrades.controller     cutils.create_system_config()
2022-01-25T13:35:36.957 localhost controller_config[6832]: info 2022-01-25 13:35:36.374 7357 ERROR controllerconfig.upgrades.controller   File "/usr/lib64/python2.7/site-packages/controllerconfig/utils.py", line 156, in create_system_config
2022-01-25T13:35:36.980 localhost controller_config[6832]: info 2022-01-25 13:35:36.374 7357 ERROR controllerconfig.upgrades.controller     raise Exception(msg)
2022-01-25T13:35:36.995 localhost controller_config[6832]: info 2022-01-25 13:35:36.374 7357 ERROR controllerconfig.upgrades.controller Exception: Failed to update puppet hiera system config
2022-01-25T13:35:37.013 localhost controller_config[6832]: info 2022-01-25 13:35:36.374 7357 ERROR controllerconfig.upgrades.controller
2022-01-25T13:35:37.027 localhost controller_config[6832]: info 2022-01-25 13:35:36.376 7357 INFO controllerconfig.upgrades.controller [-] Failed to update hiera configuration
2022-01-25T13:35:37.044 localhost controller_config[6832]: info 2022-01-25 13:35:36.376 7357 ERROR controllerconfig.upgrades.controller [-] Failed to update puppet hiera system config: Exception: Failed to update puppet hiera system config
2022-01-25T13:35:37.067 localhost controller_config[6832]: info 2022-01-25 13:35:36.376 7357 ERROR controllerconfig.upgrades.controller Traceback (most recent call last):
2022-01-25T13:35:37.084 localhost controller_config[6832]: info 2022-01-25 13:35:36.376 7357 ERROR controllerconfig.upgrades.controller   File "/usr/lib64/python2.7/site-packages/controllerconfig/upgrades/controller.py", line 1086, in main
2022-01-25T13:35:37.107 localhost controller_config[6832]: info 2022-01-25 13:35:36.376 7357 ERROR controllerconfig.upgrades.controller     upgrade_controller(from_release, to_release)
2022-01-25T13:35:37.125 localhost controller_config[6832]: info 2022-01-25 13:35:36.376 7357 ERROR controllerconfig.upgrades.controller   File "/usr/lib64/python2.7/site-packages/controllerconfig/upgrades/controller.py", line 956, in upgrade_controller
2022-01-25T13:35:36.459 localhost systemd[1]: warning controllerconfig.service failed.
2022-01-25T13:35:37.150 localhost controller_config[6832]: info 2022-01-25 13:35:36.376 7357 ERROR controllerconfig.upgrades.controller     cutils.create_system_config()
2022-01-25T13:35:37.167 localhost controller_config[6832]: info 2022-01-25 13:35:36.376 7357 ERROR controllerconfig.upgrades.controller   File "/usr/lib64/python2.7/site-packages/controllerconfig/utils.py", line 156, in create_system_config
2022-01-25T13:35:37.190 localhost controller_config[6832]: info 2022-01-25 13:35:36.376 7357 ERROR controllerconfig.upgrades.controller     raise Exception(msg)
2022-01-25T13:35:37.205 localhost controller_config[6832]: info 2022-01-25 13:35:36.376 7357 ERROR controllerconfig.upgrades.controller Exception: Failed to update puppet hiera system config
2022-01-25T13:35:37.223 localhost controller_config[6832]: info 2022-01-25 13:35:36.376 7357 ERROR controllerconfig.upgrades.controller
2022-01-25T13:35:37.238 localhost controller_config[6832]: info Upgrade failed: Failed to update puppet hiera system config {code}
*+Alarms+*

None.

*+Test Activity+*

System Test

*+Workaround+*

None.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/829138
Committed: https://opendev.org/starlingx/config/commit/9ad00e9d4c45b7b493c97eb2e1909688e517cfb2
Submitter: "Zuul (22348)"
Branch: master

commit 9ad00e9d4c45b7b493c97eb2e1909688e517cfb2
Author: Thiago Brito <email address hidden>
Date: Mon Feb 14 19:43:10 2022 -0300

    Copy hieradata on openstack when upgrading

    During the platform upgrade process, we first install the newer version
    of the StarlingX system on the standby controller and after that, we
    swact and install the other controller. When this process happens, the
    puppet plugins are not in place on the newly updated controller since
    the stx-openstack app is installed only on the older version of the
    system. This fix bypasses that behavior by checking for all plugins that
    uses Openstack configuration data, if it is generating hieradata during
    a upgrade process, it will copy those configuration from the older
    version's hieradata folder.

    TEST PLAN
    PASS: Installation from scratch on a new SX server
    PASS: Upgrade from previous version to current development version on SX
    PASS: Upgrade from previous version to current development version on DX

    Closes-Bug: #1961114

    Signed-off-by: Thiago Brito <email address hidden>
    Change-Id: Id1cbf682013820f1430bfe292448caeddba585e7