Brief Description
-----------------
"system service-parameter-apply kubernetes" causes an error on puppet.
Severity
--------
Major: service parameters are not applied properly.
Steps to Reproduce
------------------
- Run "system service-parameter-apply kubernetes"
- Checkout latest puppet.log.
Expected Behavior
------------------
The command should trigger a puppet runtime manifest run in the system and finish without errors.
After execution 'out-of-date' alarms should be cleared.
Actual Behavior
----------------
The puppet runtime manifest fails with a command 'umask' not found error and the 'out-of-date' alarm is not cleared.
Reproducibility
---------------
100% reproducible
System Configuration
--------------------
Any
Branch/Pull Time/Commit
-----------------------
STX master latest.
Last Pass
---------
Unknown
Timestamp/Logs
--------------
/var/log/sysinv.log:
sysinv 2022-06-02 04:51:16.074 75427 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 013255f8-5568-4260-92c3-b91acbeeb274 {'personalities': ['controller'], 'classes': ['platform::kubernetes::master::change_apiserver_parameters'], 'force': False} to agent
sysinv 2022-06-02 04:51:16.085 66288 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 013255f8-5568-4260-92c3-b91acbeeb274 {'personalities': ['controller'], 'classes': ['platform::kubernetes::master::change_apiserver_parameters'], 'force': False} controller
sysinv 2022-06-02 04:51:16.086 66288 INFO sysinv.agent.manager [-] controller-active
sysinv 2022-06-02 04:51:16.086 66288 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/22.06/hieradata'
sysinv 2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common [-] Failed to execute runtime manifest for host 192.168.204.2: subprocess.CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/opt/platform/puppet/22.06/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmpx9y_gxal.yaml']' returned non-zero exit status 1.
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common Traceback (most recent call last):
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common File "/usr/lib/python3/dist-packages/sysinv/puppet/common.py", line 87, in puppet_apply_manifest
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common subprocess.check_call(cmd, stdout=fnull, stderr=fnull) # pylint: disable=not-callable
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common raise CalledProcessError(retcode, cmd)
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common subprocess.CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/opt/platform/puppet/22.06/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmpx9y_gxal.yaml']' returned non-zero exit status 1.
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common
sysinv 2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager [-] failed to apply runtime manifest: sysinv.common.exception.SysinvException: Failed to execute runtime manifest for host 192.168.204.2
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager Traceback (most recent call last):
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager File "/usr/lib/python3/dist-packages/sysinv/puppet/common.py", line 87, in puppet_apply_manifest
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager subprocess.check_call(cmd, stdout=fnull, stderr=fnull) # pylint: disable=not-callable
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager raise CalledProcessError(retcode, cmd)
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager subprocess.CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/opt/platform/puppet/22.06/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmpx9y_gxal.yaml']' returned non-zero exit status 1.
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager During handling of the above exception, another exception occurred:
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager Traceback (most recent call last):
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager File "/usr/lib/python3/dist-packages/sysinv/agent/manager.py", line 1835, in _apply_runtime_manifest
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager puppet.puppet_apply_manifest(self._mgmt_ip,
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager File "/usr/lib/python3/dist-packages/sysinv/puppet/common.py", line 92, in puppet_apply_manifest
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager raise exception.SysinvException(_(msg))
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager sysinv.common.exception.SysinvException: Failed to execute runtime manifest for host 192.168.204.2
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager
sysinv 2022-06-02 04:51:52.359 66288 INFO sysinv.agent.manager [-] Caught exception _retry_on_missing_inventory_info. Retrying... Exception: Failed to execute runtime manifest for host 192.168.204.2
sysinv 2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: sysinv.common.exception.SysinvException: Failed to execute runtime manifest for host 192.168.204.2
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/sysinv/puppet/common.py", line 87, in puppet_apply_manifest
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp subprocess.check_call(cmd, stdout=fnull, stderr=fnull) # pylint: disable=not-callable
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp raise CalledProcessError(retcode, cmd)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp subprocess.CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/opt/platform/puppet/22.06/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmpx9y_gxal.yaml']' returned non-zero exit status 1.
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp During handling of the above exception, another exception occurred:
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/sysinv/openstack/common/rpc/amqp.py", line 435, in _process_data
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, namespace,
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp return Retrying(*dargs, **dkw).call(f, *args, **kw)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp return attempt.get(self._wrap_exception)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp six.reraise(self.value[0], self.value[1], self.value[2])
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp raise value
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp return f(*args, **kwargs)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/sysinv/agent/manager.py", line 1772, in config_apply_runtime_manifest
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp self._apply_runtime_manifest(config_dict)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/sysinv/agent/manager.py", line 1835, in _apply_runtime_manifest
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp puppet.puppet_apply_manifest(self._mgmt_ip,
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/sysinv/puppet/common.py", line 92, in puppet_apply_manifest
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp raise exception.SysinvException(_(msg))
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp sysinv.common.exception.SysinvException: Failed to execute runtime manifest for host 192.168.204.2
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp
/var/log/puppet/latest/puppet.log:
2022-06-02T04:51:52.039 /usr/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
2022-06-02T04:51:52.042 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:139:in `run'
2022-06-02T04:51:52.051 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
2022-06-02T04:51:52.066 /usr/bin/puppet:11:in `<main>'ESC[0m
2022-06-02T04:51:52.070 ESC[1;31mError: 2022-06-02 04:51:48 +0000 /Stage[main]/Platform::Kubernetes::Master::Change_apiserver_parameters/Exec[update kube-apiserver params]/returns: change from 'notrun' to ['0'] failed: Could not find command 'umask'ESC[0m
2022-06-02T04:51:52.073 ESC[mNotice: 2022-06-02 04:51:48 +0000 /Stage[main]/Platform::Kubernetes::Master::Change_apiserver_parameters/Exec[wait_for_kube_api_server]: Dependency Exec[update kube-apiserver params] has failures: trueESC[0m
2022-06-02T04:51:52.076 ESC[1;33mWarning: 2022-06-02 04:51:48 +0000 /Stage[main]/Platform::Kubernetes::Master::Change_apiserver_parameters/Exec[wait_for_kube_api_server]: Skipping because of failed dependenciesESC[0m
2022-06-02T04:51:52.081 ESC[0;36mDebug: 2022-06-02 04:51:48 +0000 /Stage[main]/Platform::Kubernetes::Master::Change_apiserver_parameters/Exec[wait_for_kube_api_server]: Resource is being skipped, unscheduling all eventsESC[0m
2022-06-02T04:51:52.092 ESC[0;36mDebug: 2022-06-02 04:51:48 +0000 Class[Platform::Kubernetes::Master::Change_apiserver_parameters]: Resource is being skipped, unscheduling all eventsESC[0m
2022-06-02T04:51:52.094 ESC[0;36mDebug: 2022-06-02 04:51:48 +0000 Stage[main]: Resource is being skipped, unscheduling all eventsESC[0m
Test Activity
-------------
Developer Testing
Workaround
----------
NA
Fix proposed to branch: master /review. opendev. org/c/starlingx /stx-puppet/ +/845816
Review: https:/