Duplex: Config out-of-date alarms appeared after swact w/ openstack applied

[Yang Liu]

Brief Description
-----------------
On a freshly installed duplex system with stx-openstack, swact controller - config out-of-date alarm will appear.

Severity
--------
Major

Steps to Reproduce
------------------
- Install and configure
- Apply stx-openstack
- swact

Expected Behavior
------------------
Swact should succeed without unexpected alarms

Actual Behavior
----------------
- config out-of-date alarm appeared against controller-1

Reproducibility
---------------
Intermittent

System Configuration
--------------------
Two node system
Lab-name: ip-5-6

Branch/Pull Time/Commit
-----------------------
stx master as of 20200530T013359Z

Last Pass
---------
Not sure.

Timestamp/Logs
--------------
2020-05-31T23:17:49.000 controller-0 -sh: info HISTORY: PID=2807792 UID=42425 system host-swact controller-0

Test Activity
-------------
Normal use

[Yang Liu (yliu12)]

Logs: https://files.starlingx.kube.cengn.ca/launchpad/1881509

[Ghada Khalil (gkhalil)]

Marking as stx.4.0 - requires further investigation as there could be a config issue apart from the alarm being stale.

This could be the same as https://bugs.launchpad.net/starlingx/+bug/1874136

[Ghada Khalil (gkhalil)]

@John Kung, does the runtime manifest which is run on swacts handle anything related to openstack? Yang Liu says this is only seen w/ openstack applied.

[Kristine Bujold (kbujold)]

There is an issue with the manifest apply for openstack::keystone::endpoint::runtime

sysinv 2020-05-31 23:19:49.499 221876 INFO sysinv.conductor.manager [-] Setting config target of host 'controller-1' to '62042df4-8e89-4d13-b82c-7e2afd09c913'.
sysinv 2020-05-31 23:19:49.540 221876 WARNING sysinv.conductor.manager [-] controller-1: iconfig out of date: target 62042df4-8e89-4d13-b82c-7e2afd09c913, applied 7e257187-e75e-46bf-8165-20066d02242f
sysinv 2020-05-31 23:19:49.542 221876 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-1 config applied: 7e257187-e75e-46bf-8165-20066d02242f vs. target: 62042df4-8e89-4d13-b82c-7e2afd09c913.
sysinv 2020-05-31 23:19:49.598 221876 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=62042df4-8e89-4d13-b82c-7e2afd09c913
sysinv 2020-05-31 23:19:49.599 221876 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=62042df4-8e89-4d13-b82c-7e2afd09c913, classes: ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime']
sysinv 2020-05-31 23:19:49.637 221876 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-1 with config_uuid: 62042df4-8e89-4d13-b82c-7e2afd09c913
sysinv 2020-05-31 23:19:53.748 221876 INFO sysinv.puppet.kubernetes [-] get_kubernetes_join_cmd join_cmd=kubeadm join 192.168.205.1:6443 --token xjtepg.prb4umxjlu2yl9dt --discovery-token-ca-cert-hash sha256:967dc2ea50af4679ed43441556d52bcb9f7fb168dd9188bd00e18611149d38a5 --control-plane --certificate-key b08e693f9a388d46c7021c15fb34025f8f6cef5ff5a43f71d57aae7ca760a5ac --cri-socket /var/run/containerd/containerd.sock
sysinv 2020-05-31 23:19:54.657 222753 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
sysinv 2020-05-31 23:19:55.132 221876 INFO sysinv.helm.helm [-] HelmOperator: Loading available helm and armada plugins.
sysinv 2020-05-31 23:19:55.135 221876 INFO sysinv.helm.helm [-] Plugins for generic : loaded from sysinv - /usr/lib64/python2.7/site-packages.
sysinv 2020-05-31 23:19:55.135 221876 INFO sysinv.helm.helm [-] Plugins for stx-monitor : loaded from sysinv - /usr/lib64/python2.7/site-packages.
sysinv 2020-05-31 23:19:55.136 221876 ERROR sysinv.puppet.puppet [-] failed to create system config: KeyError: 'keystone'
2020-05-31 23:19:55.136 221876 ERROR sysinv.puppet.puppet Traceback (most recent call last):
2020-05-31 23:19:55.136 221876 ERROR sysinv.puppet.puppet File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 116, in update_system_config
2020-05-31 23:19:55.136 221876 ERROR sysinv.puppet.puppet config.update(puppet_plugin.obj.get_system_config())
2020-05-31 23:19:55.136 221876 ERROR sysinv.puppet.puppet File "/usr/lib64/python2.7/site-packages/sysinv/puppet/nfv.py", line 96, in get_system_config
2020-05-31 23:19:55.136 221876 ERROR sysinv.puppet.puppet keystone_auth_data = helm_data.get_keystone_auth_data()
2020-05-31 23:19:55.136 221876 ERROR sysinv.puppet.puppet File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 49, in _wrapper
2020-05-31 23:19:55.136 221876 ERROR sysinv.puppet.puppet return func(self, *args, **k...

[Kristine Bujold (kbujold)]

Could not reproduce in IP5-6 with Cengn load 20200713T230429Z

[Ghada Khalil (gkhalil)]

Closing as the issue is not reproducible.