Runtime manifest apply will invoke the evaluation of app re-apply for the supported installed apps. The approach of the evaluation is regenerate application overrides and compare with the old ones. "helm install" command is used to generate overrides for each chart during application overrides generation.
If helm command fails it can interrupt the runtime manifests apply resulting in stuck 250.001 alarm Configuration is out-of-date.
This would only be an issue when applications have overrides in this case the cert-manager application had an override.
sysinv 2020-07-30 00:01:48.553 94899 INFO sysinv.conductor.manager [-] Setting config target of host 'controller-0' to '5e71161d-9ec1-4d68-b2d7-4f04d936cd15'.
sysinv 2020-07-30 00:01:48.566 94899 WARNING sysinv.conductor.manager [-] controller-0: iconfig out of date: target 5e71161d-9ec1-4d68-b2d7-4f04d936cd15, applied 21d1dbbb-6260-4c92-b97d-29f816385b3c
sysinv 2020-07-30 00:01:48.566 94899 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-0 config applied: 21d1dbbb-6260-4c92-b97d-29f816385b3c vs. target: 5e71161d-9ec1-4d68-b2d7-4f04d936cd15.
sysinv 2020-07-30 00:01:48.578 94899 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=5e71161d-9ec1-4d68-b2d7-4f04d936cd15
sysinv 2020-07-30 00:01:48.579 94899 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=5e71161d-9ec1-4d68-b2d7-4f04d936cd15, classes: ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime']
sysinv 2020-07-30 00:01:48.588 94899 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: 5e71161d-9ec1-4d68-b2d7-4f04d936cd15
sysinv 2020-07-30 00:01:50.616 94899 ERROR sysinv.puppet.kubernetes [-] Failed to get device id for pci device 0000:b6:00.0
sysinv 2020-07-30 00:01:50.879 94899 INFO sysinv.puppet.kubernetes [-] get_kubernetes_join_cmd join_cmd=kubeadm join [fd00:4888:0:1::1]:6443 --token 5jg99s.2ymqkn3ks7qpqfbt --discovery-token-ca-cert-hash sha256:5da52e9246e22e7ac542c376da684930fca5ca2b33f5199139b7f87c29b56572 --control-plane --certificate-key daba0f3d927234991b5d8b41bd22937e3d691804513ea58cd81bdb990070cf3a --apiserver-advertise-address fd00:4888:0:1::2 --cri-socket /var/run/containerd/containerd.sock
sysinv 2020-07-30 00:01:52.479 94899 INFO sysinv.conductor.manager [-] No override change after configuration action, skipping re-apply of platform-integ-apps
sysinv 2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: Command '['helm', 'install', '--dry-run', '--debug', '--values', '/var/run/sysinv_tmp/tmpz5ej3V', '--values', '/var/run/sysinv_tmp/tmpyz7iZo', '/var/run/sysinv_tmp/tmp9dMpVO']' returned non-zero exit status 1: CalledProcessError: Command '['helm', 'install', '--dry-run', '--debug', '--values', '/var/run/sysinv_tmp/tmpz5ej3V', '--values', '/var/run/sysinv_tmp/tmpyz7iZo', '/var/run/sysinv_tmp/tmp9dMpVO']' returned non-zero exit status 1
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last):
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task task(self, context)
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 5052, in _conductor_audit
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task self._controller_config_active_apply(context)
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4771, in _controller_config_active_apply
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task context, config_uuid, config_dict)
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 8876, in _config_apply_runtime_manifest
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task self.evaluate_app_reapply(context, app_name)
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 11277, in evaluate_app_reapply
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task armada_format=True, armada_chart_info=app.charts, combined=True)
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 49, in _wrapper
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task return func(self, *args, **kwargs)
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs)
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 680, in generate_helm_application_overrides
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task file_overrides=file_overrides)
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 551, in merge_overrides
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task output = subprocess.check_output(cmd, env=env)
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task raise CalledProcessError(retcode, cmd, output=output)
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task CalledProcessError: Command '['helm', 'install', '--dry-run', '--debug', '--values', '/var/run/sysinv_tmp/tmpz5ej3V', '--values', '/var/run/sysinv_tmp/tmpyz7iZo', '/var/run/sysinv_tmp/tmp9dMpVO']' returned non-zero exit status 1
2020-07-30 00:01:52.695 94899 ERROR sysinv.openstack.common.periodic_task
Severity
--------
Major
Steps to Reproduce
------------------
Not certain how exactly to reproduce. It was seen in a DC lab. The helm install command had failed because tiller was not running yet on the system and the sysinv conductor was trying to apply runtime manifests.
Brief Description
-----------------
Runtime manifest apply will invoke the evaluation of app re-apply for the supported installed apps. The approach of the evaluation is regenerate application overrides and compare with the old ones. "helm install" command is used to generate overrides for each chart during application overrides generation.
If helm command fails it can interrupt the runtime manifests apply resulting in stuck 250.001 alarm Configuration is out-of-date.
This would only be an issue when applications have overrides in this case the cert-manager application had an override.
sysinv 2020-07-30 00:01:48.553 94899 INFO sysinv. conductor. manager [-] Setting config target of host 'controller-0' to '5e71161d- 9ec1-4d68- b2d7-4f04d936cd 15'. conductor. manager [-] controller-0: iconfig out of date: target 5e71161d- 9ec1-4d68- b2d7-4f04d936cd 15, applied 21d1dbbb- 6260-4c92- b97d-29f816385b 3c conductor. manager [-] SYS_I Raise system config alarm: host controller-0 config applied: 21d1dbbb- 6260-4c92- b97d-29f816385b 3c vs. target: 5e71161d- 9ec1-4d68- b2d7-4f04d936cd 15. conductor. manager [-] _config_ update_ hosts config_ uuid=5e71161d- 9ec1-4d68- b2d7-4f04d936cd 15 conductor. manager [-] applying runtime manifest config_ uuid=5e71161d- 9ec1-4d68- b2d7-4f04d936cd 15, classes: ['openstack: :keystone: :endpoint: :runtime' , 'platform: :firewall: :runtime' ] puppet. puppet [-] Updating hiera for host: controller-0 with config_uuid: 5e71161d- 9ec1-4d68- b2d7-4f04d936cd 15 puppet. kubernetes [-] Failed to get device id for pci device 0000:b6:00.0 puppet. kubernetes [-] get_kubernetes_ join_cmd join_cmd=kubeadm join [fd00:4888: 0:1::1] :6443 --token 5jg99s. 2ymqkn3ks7qpqfb t --discovery- token-ca- cert-hash sha256: 5da52e9246e22e7 ac542c376da6849 30fca5ca2b33f51 99139b7f87c29b5 6572 --control-plane --certificate-key daba0f3d9272349 91b5d8b41bd2293 7e3d691804513ea 58cd81bdb990070 cf3a --apiserver- advertise- address fd00:4888:0:1::2 --cri-socket /var/run/ containerd/ containerd. sock conductor. manager [-] No override change after configuration action, skipping re-apply of platform-integ-apps openstack. common. periodic_ task [-] Error during ConductorManage r._conductor_ audit: Command '['helm', 'install', '--dry-run', '--debug', '--values', '/var/run/ sysinv_ tmp/tmpz5ej3V' , '--values', '/var/run/ sysinv_ tmp/tmpyz7iZo' , '/var/run/ sysinv_ tmp/tmp9dMpVO' ]' returned non-zero exit status 1: CalledProcessError: Command '['helm', 'install', '--dry-run', '--debug', '--values', '/var/run/ sysinv_ tmp/tmpz5ej3V' , '--values', '/var/run/ sysinv_ tmp/tmpyz7iZo' , '/var/run/ sysinv_ tmp/tmp9dMpVO' ]' returned non-zero exit status 1 openstack. common. periodic_ task Traceback (most recent call last): openstack. common. periodic_ task File "/usr/lib64/ python2. 7/site- packages/ sysinv/ openstack/ common/ periodic_ task.py" , line 180, in run_periodic_tasks openstack. common. periodic_ task task(self, context) openstack. common. periodic_ task File "/usr/lib64/ python2. 7/site- packages/ sysinv/ conductor/ manager. py", line 5052, in _conductor_audit openstack. common. periodic_ task self._controlle r_config_ active_ apply(context) openstack. common. periodic_ task File "/usr/lib64/ python2. 7/site- packages/ sysinv/ conductor/ manager. py", line 4771, in _controller_ config_ active_ apply openstack. common. periodic_ task context, config_uuid, config_dict) openstack. common. periodic_ task File "/usr/lib64/ python2. 7/site- packages/ sysinv/ conductor/ manager. py", line 8876, in _config_ apply_runtime_ manifest openstack. common. periodic_ task self.evaluate_ app_reapply( context, app_name) openstack. common. periodic_ task File "/usr/lib64/ python2. 7/site- packages/ sysinv/ conductor/ manager. py", line 11277, in evaluate_ app_reapply openstack. common. periodic_ task armada_format=True, armada_ chart_info= app.charts, combined=True) openstack. common. periodic_ task File "/usr/lib64/ python2. 7/site- packages/ sysinv/ helm/helm. py", line 49, in _wrapper openstack. common. periodic_ task return func(self, *args, **kwargs) openstack. common. periodic_ task File "/usr/lib/ python2. 7/site- packages/ oslo_concurrenc y/lockutils. py", line 328, in inner openstack. common. periodic_ task return f(*args, **kwargs) openstack. common. periodic_ task File "/usr/lib64/ python2. 7/site- packages/ sysinv/ helm/helm. py", line 680, in generate_ helm_applicatio n_overrides openstack. common. periodic_ task file_overrides= file_overrides) openstack. common. periodic_ task File "/usr/lib64/ python2. 7/site- packages/ sysinv/ helm/helm. py", line 551, in merge_overrides openstack. common. periodic_ task output = subprocess. check_output( cmd, env=env) openstack. common. periodic_ task File "/usr/lib64/ python2. 7/subprocess. py", line 575, in check_output openstack. common. periodic_ task raise CalledProcessEr ror(retcode, cmd, output=output) openstack. common. periodic_ task CalledProcessError: Command '['helm', 'install', '--dry-run', '--debug', '--values', '/var/run/ sysinv_ tmp/tmpz5ej3V' , '--values', '/var/run/ sysinv_ tmp/tmpyz7iZo' , '/var/run/ sysinv_ tmp/tmp9dMpVO' ]' returned non-zero exit status 1 openstack. common. periodic_ task
sysinv 2020-07-30 00:01:48.566 94899 WARNING sysinv.
sysinv 2020-07-30 00:01:48.566 94899 WARNING sysinv.
sysinv 2020-07-30 00:01:48.578 94899 INFO sysinv.
sysinv 2020-07-30 00:01:48.579 94899 INFO sysinv.
sysinv 2020-07-30 00:01:48.588 94899 INFO sysinv.
sysinv 2020-07-30 00:01:50.616 94899 ERROR sysinv.
sysinv 2020-07-30 00:01:50.879 94899 INFO sysinv.
sysinv 2020-07-30 00:01:52.479 94899 INFO sysinv.
sysinv 2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
2020-07-30 00:01:52.695 94899 ERROR sysinv.
Severity
--------
Major
Steps to Reproduce
------------------
Not certain how exactly to reproduce. It was seen in a DC lab. The helm install command had failed because tiller was not running yet on the system and the sysinv conductor was trying to apply runtime manifests.
tiller- deploy- 5c8dd9fb56- 88w6g_kube- system_ tiller- c1e2563a9146b9f 525c431904761de deea3c7c19a12c7 8507d0e6f125171 1761.log 30T00:01: 50.994005897Z stderr F [main] 2020/07/30 00:01:50 Starting Tiller v2.13.1 (tls=false)
2020-07-
sysinv.log openstack. common. rpc.amqp [-] Exception during message handling: CalledProcessError: Command '['helm', 'install', '--dry-run', '--debug', '--values', '/var/run/ sysinv_ tmp/tmpNTjKcb' , '--values', '/var/run/ sysinv_ tmp/tmpcwivSo' , '/var/run/ sysinv_ tmp/tmpP05WVg' ]' returned non-zero exit status 1
sysinv 2020-07-30 00:01:42.995 94899 ERROR sysinv.
Expected Behavior
------------------
No 250.001 alarm Configuration is out-of-date raised.
No exception logs in sysinv.log
Command '['helm', 'install', '--dry-run', '--debug', '--values', '/var/run/ sysinv_ tmp/tmpz5ej3V' , '--values', '/var/run/ sysinv_ tmp/tmpyz7iZo' , '/var/run/ sysinv_ tmp/tmp9dMpVO' ]' returned non-zero exit status 1
Actual Behavior
----------------
See above
Reproducibility
---------------
Intermittent
System Configuration ------- ------
-------
DC system
Branch/Pull Time/Commit ------- ------- --
-------
Last Pass
---------
Timestamp/Logs
--------------
Test Activity
-------------
Testing
Workaround
----------