Brief Description
-----------------
On a duplex system with stx-openstack applied, after lock/unlock controller-1, swact, and lock controller-0 - unlocking controller-0 failed with following error. Did not find workaround.
[sysadmin@controller-1 ~(keystone_admin)]$ system host-unlock controller-0
u'\'keystone\'\nTraceback (most recent call last):\n\n File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data\n **args)\n\n File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n result = getattr(proxyobj, method)(ctxt, **kwargs)\n\n File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1687, in configure_ihost\n self._puppet.update_system_config()\n\n File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 31, in _wrapper\n func(self, *args, **kwargs)\n\n File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 116, in update_system_config\n config.update(puppet_plugin.obj.get_system_config())\n\n File "/usr/lib64/python2.7/site-packages/sysinv/puppet/nfv.py", line 96, in get_system_config\n keystone_auth_data = helm_data.get_keystone_auth_data()\n\n File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 49, in _wrapper\n return func(self, *args, **kwargs)\n\n File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 845, in get_keystone_auth_data\n keystone_operator = self.chart_operators[self.HELM_CHART_KEYSTONE]\n\nKeyError: \'keystone\'\n'
Severity
--------
Critical
Steps to Reproduce
------------------
- Install and configure a duplex system
- Apply stx-openstack
- Swact a couple times
- lock/unlock controller-1
- swact
- lock/unlock controller-0
Expected Behavior
------------------
- Able to unlock controller-0
Actual Behavior
----------------
- controller-0 unlocked rejected with following error:
[sysadmin@controller-1 ~(keystone_admin)]$ system host-unlock controller-0
u'\'keystone\'\nTraceback (most recent call last):\n\n File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data\n **args)\n\n File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n result = getattr(proxyobj, method)(ctxt, **kwargs)\n\n File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1687, in configure_ihost\n self._puppet.update_system_config()\n\n File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 31, in _wrapper\n func(self, *args, **kwargs)\n\n File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 116, in update_system_config\n config.update(puppet_plugin.obj.get_system_config())\n\n File "/usr/lib64/python2.7/site-packages/sysinv/puppet/nfv.py", line 96, in get_system_config\n keystone_auth_data = helm_data.get_keystone_auth_data()\n\n File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 49, in _wrapper\n return func(self, *args, **kwargs)\n\n File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 845, in get_keystone_auth_data\n keystone_operator = self.chart_operators[self.HELM_CHART_KEYSTONE]\n\nKeyError: \'keystone\'\n'
Reproducibility
---------------
Intermittent
System Configuration
--------------------
Two node system
Lab-name: ip5-6
Branch/Pull Time/Commit
-----------------------
stx master as of "20200530T013359Z"
Last Pass
---------
Same load
Timestamp/Logs
--------------
2020-05-31T02:10:51.000 controller-1 -sh: info HISTORY: PID=791930 UID=42425 system host-lock controller-0
2020-05-31T02:11:45.000 controller-1 -sh: info HISTORY: PID=791930 UID=42425 system host-unlock controller-0
Test Activity
-------------
Normal use
Seen again on same system. Both set of logs are here: /files. starlingx. kube.cengn. ca/launchpad/ 1881454
https:/
Time stamp for unlock failure in second occurrence: 31T23:42: 12.000 controller-1 -sh: info HISTORY: PID=437060 UID=42425 system host-unlock controller-0
2020-05-