Failure while upgrading controller-0 of System Controller due to dc-cert

[Jessica Castelino]

Brief Description
-----------------
While upgrading a Standard system controller from stx.4.0 to stx.5.0, a failure is encountered due to dc-cert as given below. After running "system host-upgrade controller-0", I am unable to unlock the controller-0.

[sysadmin@controller-1 ~(keystone_admin)]$ system host-unlock controller-0
Invalid secret dc-cert\dc-adminep-certificate
Traceback (most recent call last):

  File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data
    **args)

  File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
    result = getattr(proxyobj, method)(ctxt, **kwargs)

  File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1785, in configure_ihost
    self._puppet.update_secure_system_config()

  File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 31, in _wrapper
    func(self, *args, **kwargs)

  File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 131, in update_secure_system_config
    config.update(puppet_plugin.obj.get_secure_system_config())

  File "/usr/lib64/python2.7/site-packages/sysinv/puppet/platform.py", line 53, in get_secure_system_config
    config.update(self._get_dc_root_ca_config())

  File "/usr/lib64/python2.7/site-packages/sysinv/puppet/platform.py", line 893, in _get_dc_root_ca_config
    system.distributed_cloud_role)

  File "/usr/lib64/python2.7/site-packages/sysinv/common/utils.py", line 2316, in get_admin_ep_cert
    endpoint_cert_secret_ns, endpoint_cert_secret_name

Exception: Invalid secret dc-cert\dc-adminep-certificate

Severity
--------
Critical

Steps to Reproduce
------------------
Upgrade a standard system controller using the steps given below:
system upgrade-start --force
system upgrade-show (wait for "started")
system host-lock controller-1
system host-list
system host-upgrade controller-1

system upgrade-show (wait for "data-migration-complete")
system host-unlock controller-1 (Wait for controller-1 to become unlocked-enabled. Wait for the DRBD sync 400.001 services-related alarm to be cleared)
system host-swact controller-0
system host-lock controller-0
system host-upgrade controller-0 (wait for "upgrading-hosts")
system host-unlock controller-0

Expected Behavior
------------------
Should be able to unlock controller-0

Actual Behavior
----------------
Error while unlocking controller-0

Reproducibility
---------------
100% Reproducible

System Configuration
--------------------
DC setup with standard system controller and aiosx subcloud

Branch/Pull Time/Commit
-----------------------
22nd January, 2021 at 12:28 am

Last Pass
---------
Yes. The last pass was 15th January, 2021.

Timestamp/Logs
--------------
Collect logs attached

Test Activity
-------------
Developer Testing

Workaround
----------
None

[Ghada Khalil (gkhalil)]

This was already reported in https://bugs.launchpad.net/starlingx/+bug/1913039 so I'll close this as a duplicate.