StarlingX

Debian: Puppet error on service-parameter-apply kubernetes

Bug #1978718 reported by Andy on 2022-06-14

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	Fix Released	Medium	Andy

Bug Description

Brief Description
-----------------
"system service-parameter-apply kubernetes" causes an error on puppet.

Severity
--------
Major: service parameters are not applied properly.

Steps to Reproduce
------------------
- Run "system service-parameter-apply kubernetes"
- Checkout latest puppet.log.

Expected Behavior
------------------
The command should trigger a puppet runtime manifest run in the system and finish without errors.
After execution 'out-of-date' alarms should be cleared.

Actual Behavior
----------------
The puppet runtime manifest fails with a command 'umask' not found error and the 'out-of-date' alarm is not cleared.

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Any

Branch/Pull Time/Commit
-----------------------
STX master latest.

Last Pass
---------
Unknown

Timestamp/Logs
--------------
/var/log/sysinv.log:

sysinv 2022-06-02 04:51:16.074 75427 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 013255f8-5568-4260-92c3-b91acbeeb274 {'personalities': ['controller'], 'classes': ['platform::kubernetes::master::change_apiserver_parameters'], 'force': False} to agent
sysinv 2022-06-02 04:51:16.085 66288 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 013255f8-5568-4260-92c3-b91acbeeb274 {'personalities': ['controller'], 'classes': ['platform::kubernetes::master::change_apiserver_parameters'], 'force': False} controller
sysinv 2022-06-02 04:51:16.086 66288 INFO sysinv.agent.manager [-] controller-active
sysinv 2022-06-02 04:51:16.086 66288 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/22.06/hieradata'
sysinv 2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common [-] Failed to execute runtime manifest for host 192.168.204.2: subprocess.CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/opt/platform/puppet/22.06/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmpx9y_gxal.yaml']' returned non-zero exit status 1.
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common Traceback (most recent call last):
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common File "/usr/lib/python3/dist-packages/sysinv/puppet/common.py", line 87, in puppet_apply_manifest
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common subprocess.check_call(cmd, stdout=fnull, stderr=fnull) # pylint: disable=not-callable
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common raise CalledProcessError(retcode, cmd)
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common subprocess.CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/opt/platform/puppet/22.06/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmpx9y_gxal.yaml']' returned non-zero exit status 1.
2022-06-02 04:51:52.167 66288 ERROR sysinv.puppet.common
sysinv 2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager [-] failed to apply runtime manifest: sysinv.common.exception.SysinvException: Failed to execute runtime manifest for host 192.168.204.2
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager Traceback (most recent call last):
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager File "/usr/lib/python3/dist-packages/sysinv/puppet/common.py", line 87, in puppet_apply_manifest
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager subprocess.check_call(cmd, stdout=fnull, stderr=fnull) # pylint: disable=not-callable
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager raise CalledProcessError(retcode, cmd)
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager subprocess.CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/opt/platform/puppet/22.06/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmpx9y_gxal.yaml']' returned non-zero exit status 1.
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager During handling of the above exception, another exception occurred:
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager Traceback (most recent call last):
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager File "/usr/lib/python3/dist-packages/sysinv/agent/manager.py", line 1835, in _apply_runtime_manifest
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager puppet.puppet_apply_manifest(self._mgmt_ip,
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager File "/usr/lib/python3/dist-packages/sysinv/puppet/common.py", line 92, in puppet_apply_manifest
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager raise exception.SysinvException(_(msg))
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager sysinv.common.exception.SysinvException: Failed to execute runtime manifest for host 192.168.204.2
2022-06-02 04:51:52.168 66288 ERROR sysinv.agent.manager
sysinv 2022-06-02 04:51:52.359 66288 INFO sysinv.agent.manager [-] Caught exception _retry_on_missing_inventory_info. Retrying... Exception: Failed to execute runtime manifest for host 192.168.204.2
sysinv 2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: sysinv.common.exception.SysinvException: Failed to execute runtime manifest for host 192.168.204.2
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/sysinv/puppet/common.py", line 87, in puppet_apply_manifest
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp subprocess.check_call(cmd, stdout=fnull, stderr=fnull) # pylint: disable=not-callable
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp raise CalledProcessError(retcode, cmd)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp subprocess.CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/opt/platform/puppet/22.06/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmpx9y_gxal.yaml']' returned non-zero exit status 1.
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp During handling of the above exception, another exception occurred:
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/sysinv/openstack/common/rpc/amqp.py", line 435, in _process_data
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, namespace,
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp return Retrying(*dargs, **dkw).call(f, *args, **kw)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp return attempt.get(self._wrap_exception)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp six.reraise(self.value[0], self.value[1], self.value[2])
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp raise value
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp return f(*args, **kwargs)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/sysinv/agent/manager.py", line 1772, in config_apply_runtime_manifest
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp self._apply_runtime_manifest(config_dict)
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/sysinv/agent/manager.py", line 1835, in _apply_runtime_manifest
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp puppet.puppet_apply_manifest(self._mgmt_ip,
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python3/dist-packages/sysinv/puppet/common.py", line 92, in puppet_apply_manifest
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp raise exception.SysinvException(_(msg))
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp sysinv.common.exception.SysinvException: Failed to execute runtime manifest for host 192.168.204.2
2022-06-02 04:51:52.360 66288 ERROR sysinv.openstack.common.rpc.amqp

/var/log/puppet/latest/puppet.log:

2022-06-02T04:51:52.039 /usr/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
2022-06-02T04:51:52.042 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:139:in `run'
2022-06-02T04:51:52.051 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
2022-06-02T04:51:52.066 /usr/bin/puppet:11:in `<main>'ESC[0m
2022-06-02T04:51:52.070 ESC[1;31mError: 2022-06-02 04:51:48 +0000 /Stage[main]/Platform::Kubernetes::Master::Change_apiserver_parameters/Exec[update kube-apiserver params]/returns: change from 'notrun' to ['0'] failed: Could not find command 'umask'ESC[0m
2022-06-02T04:51:52.073 ESC[mNotice: 2022-06-02 04:51:48 +0000 /Stage[main]/Platform::Kubernetes::Master::Change_apiserver_parameters/Exec[wait_for_kube_api_server]: Dependency Exec[update kube-apiserver params] has failures: trueESC[0m
2022-06-02T04:51:52.076 ESC[1;33mWarning: 2022-06-02 04:51:48 +0000 /Stage[main]/Platform::Kubernetes::Master::Change_apiserver_parameters/Exec[wait_for_kube_api_server]: Skipping because of failed dependenciesESC[0m
2022-06-02T04:51:52.081 ESC[0;36mDebug: 2022-06-02 04:51:48 +0000 /Stage[main]/Platform::Kubernetes::Master::Change_apiserver_parameters/Exec[wait_for_kube_api_server]: Resource is being skipped, unscheduling all eventsESC[0m
2022-06-02T04:51:52.092 ESC[0;36mDebug: 2022-06-02 04:51:48 +0000 Class[Platform::Kubernetes::Master::Change_apiserver_parameters]: Resource is being skipped, unscheduling all eventsESC[0m
2022-06-02T04:51:52.094 ESC[0;36mDebug: 2022-06-02 04:51:48 +0000 Stage[main]: Resource is being skipped, unscheduling all eventsESC[0m

Test Activity
-------------
Developer Testing

Workaround
----------
NA

Tags:

Andy (andy.wrs) on 2022-06-14

Changed in starlingx:
assignee:	nobody → Andy (andy.wrs)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-06-14: Fix proposed to stx-puppet (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/845816

Changed in starlingx:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-06-14: Fix merged to stx-puppet (master)

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/845816
Committed: https://opendev.org/starlingx/stx-puppet/commit/6d0a4d472d5bec0ae3f361a2bea58429f87f0cd9
Submitter: "Zuul (22348)"
Branch: master

commit 6d0a4d472d5bec0ae3f361a2bea58429f87f0cd9
Author: Andy Ning <email address hidden>
Date: Tue Jun 14 12:18:45 2022 -0400

Fix umask not found when applying kubernetes.pp

    On Debian, umask is a built-in command so "umask 077" in
    kube-apiserver-change-params.erb will fail the manifest
    since umask could not be found. This change fixed it by
    running umask with shell as provider.

    Test Plan on Debian and CentOS:
    PASS: "system service-parameter-apply kubernetes" complete
          successfully, no error in puppet.log.
    PASS: check /tmp/cluster_configmap.yaml and
          /tmp/kubeadm_config_view.yaml have the right permission:
          -rw------- 1 root root /tmp/cluster_configmap.yaml
          -rw------- 1 root root /tmp/kubeadm_config_view.yaml

    Closes-Bug: 1978718
    Signed-off-by: Andy Ning <email address hidden>
    Change-Id: Ie680cac904ef10bf063c0e6329192edc075e6a1e

Changed in starlingx:
status:	In Progress → Fix Released

Ghada Khalil (gkhalil) on 2022-06-28

Changed in starlingx:
importance:	Undecided → Medium
tags:	added: stx.7.0 stx.config stx.debian

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.