Platform: oidc-auth-apps apply failed due to CallledProcessError puppet-manifest-apply

[Yvonne Ding]

Brief Description
-----------------
oidc-auth-apps apply failed. Failed to execute runtime manifest CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/var/run/platform/puppet/20.12/hieradata', '192.168.204.3', 'worker', 'runtime', '/tmp/tmplE10NQ.yaml']'

Severity
--------
Major

Steps to Reproduce
------------------
platform security regression

Expected Behavior
------------------
oidc-auth-apps apply success

Actual Behavior
----------------
oidc-auth-apps apply failed

Reproducibility
---------------
reproducible

System Configuration
--------------------
AIO-DX

Branch/Pull Time/Commit
-----------------------
BUILD_ID="20210318T063229Z"

Timestamp/Logs
--------------
sysinv 2021-03-20 15:25:20.521 7513 INFO sysinv.agent.manager [-] controller-standby or storage, mount /var/run/platform
sysinv 2021-03-20 15:25:20.548 7513 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/var/run/platform/puppet/20.12/hieradata'
sysinv 2021-03-20 15:25:26.169 7513 ERROR sysinv.puppet.common [-] Failed to execute runtime manifest for host 192.168.204.3: CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/var/run/platform/puppet/20.12/hieradata', '192.168.204.3', 'worker', 'runtime', '/tmp/tmplE10NQ.yaml']' returned non-zero exit status 1
2021-03-20 15:25:26.169 7513 ERROR sysinv.puppet.common Traceback (most recent call last):
2021-03-20 15:25:26.169 7513 ERROR sysinv.puppet.common File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 76, in puppet_apply_manifest
2021-03-20 15:25:26.169 7513 ERROR sysinv.puppet.common subprocess.check_call(cmd, stdout=fnull, stderr=fnull) # pylint: disable=not-callable
2021-03-20 15:25:26.169 7513 ERROR sysinv.puppet.common File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
2021-03-20 15:25:26.169 7513 ERROR sysinv.puppet.common raise CalledProcessError(retcode, cmd)
2021-03-20 15:25:26.169 7513 ERROR sysinv.puppet.common CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/var/run/platform/puppet/20.12/hieradata', '192.168.204.3', 'worker', 'runtime', '/tmp/tmplE10NQ.yaml']' returned non-zero exit status 1
2021-03-20 15:25:26.169 7513 ERROR sysinv.puppet.common
sysinv 2021-03-20 15:25:26.171 7513 ERROR sysinv.agent.manager [-] failed to apply runtime manifest: SysinvException: Failed to execute runtime manifest for host 192.168.204.3
2021-03-20 15:25:26.171 7513 ERROR sysinv.agent.manager Traceback (most recent call last):
2021-03-20 15:25:26.171 7513 ERROR sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1774, in _apply_runtime_manifest
2021-03-20 15:25:26.171 7513 ERROR sysinv.agent.manager hieradata_path=hieradata_path)
2021-03-20 15:25:26.171 7513 ERROR sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 81, in puppet_apply_manifest
2021-03-20 15:25:26.171 7513 ERROR sysinv.agent.manager raise exception.SysinvException(_(msg))
2021-03-20 15:25:26.171 7513 ERROR sysinv.agent.manager SysinvException: Failed to execute runtime manifest for host 192.168.204.3
2021-03-20 15:25:26.171 7513 ERROR sysinv.agent.manager
sysinv 2021-03-20 15:25:26.252 7513 INFO sysinv.agent.manager [-] Caught exception. Retrying... Exception: Failed to execute runtime manifest for host 192.168.204.3
sysinv 2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: SysinvException: Failed to execute runtime manifest for host 192.168.204.3
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp **args)
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp return Retrying(*dargs, **dkw).call(f, *args, **kw)
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/retrying.py", line 223, in call
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp return attempt.get(self._wrap_exception)
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp six.reraise(self.value[0], self.value[1], self.value[2])
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp return f(*args, **kwargs)
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1705, in config_apply_runtime_manifest
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp self._apply_runtime_manifest(config_dict, hieradata_path=hieradata_path)
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1774, in _apply_runtime_manifest
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp hieradata_path=hieradata_path)
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 81, in puppet_apply_manifest
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp raise exception.SysinvException(_(msg))
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp SysinvException: Failed to execute runtime manifest for host 192.168.204.3
2021-03-20 15:25:26.253 7513 ERROR sysinv.openstack.common.rpc.amqp

[Jerry Sun (jerry-sun-u)]

This looks like it is related to https://bugs.launchpad.net/starlingx/+bug/1917308. The first thing done involving helm and armada after a swact from controller-1 to controller-0 was an application apply of oidc-auth-apps. The following is seen in the armada logs:

2021-03-20 17:58:38.014 439 ERROR armada.cli [-] Caught unexpected exception: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "write tcp 172.16.192.80:36086->192.168.204.1:5432: write: broken pipe"
        debug_error_string = "{"created":"@1616263117.042123301","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"write tcp 172.16.192.80:36086->192.168.204.1:5432: write: broken pipe","grpc_status":2}"
>

I have seen the grpc issue before, and it seems to be related to the issue above, where the first command that uses armada after a swact seems to fail sometimes. Running another armada command, like applying oidc-auth-apps again, should apply successfully.

The puppet errors seems to have been logged long before oidc-auth-apps failed to apply, and appears irrelevant:
sysinv 2021-03-20 15:25:26.169 7513 ERROR sysinv.puppet.common [-] Failed to execute runtime manifest for host 192.168.204.3: CalledProcessError

This would be an issue with the application framework and not specific to oidc-auth-apps

[Ghada Khalil (gkhalil)]

Marking as a duplicate of https://bugs.launchpad.net/starlingx/+bug/1917308

[Ghada Khalil (gkhalil)]

Fix for duplicate LP: https://bugs.launchpad.net/starlingx/+bug/1917308 merged in r/stx.5.0 on 2021-04-27