Activity log for bug #1918703

Date Who What changed Old value New value Message
2021-03-11 17:01:29 Yvonne Ding bug added bug
2021-03-11 17:02:13 Yvonne Ding description Brief Description ----------------- stx-openstack apply failed due to CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh' '/var/run/platform/puppet/20.12/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmp_3nV4d.yaml']' Severity -------- Major Expected Behavior ------------------ stx-openstack apply success Actual Behavior ---------------- stx-openstack apply failed Reproducibility --------------- reproducible System Configuration -------------------- Regular standard 2+2 Branch/Pull Time/Commit ----------------------- BUILD_ID="" Timestamp/Logs -------------- sysinv 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common [-] Failed to execute runtime manifest for host 192.168.204.2: CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/var/run/platform/puppet/20.12/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmp_3nV4d.yaml']' returned non-zero exit status 1 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common Traceback (most recent call last): 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 76, in puppet_apply_manifest 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common subprocess.check_call(cmd, stdout=fnull, stderr=fnull) # pylint: disable=not-callable 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common raise CalledProcessError(retcode, cmd) 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/var/run/platform/puppet/20.12/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmp_3nV4d.yaml']' returned non-zero exit status 1 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common sysinv 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager [-] failed to apply runtime manifest: SysinvException: Failed to execute runtime manifest for host 192.168.204.2 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager Traceback (most recent call last): 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1775, in _apply_runtime_manifest 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager hieradata_path=hieradata_path) 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 81, in puppet_apply_manifest 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager raise exception.SysinvException(_(msg)) 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager SysinvException: Failed to execute runtime manifest for host 192.168.204.2 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager sysinv 2021-03-11 03:44:31.172 2688 INFO sysinv.agent.manager [-] Caught exception. Retrying... Exception: Failed to execute runtime manifest for host 192.168.204.2 sysinv 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: SysinvException: Failed to execute runtime manifest for host 192.168.204.2 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last): 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp **args) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp return Retrying(*dargs, **dkw).call(f, *args, **kw) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/retrying.py", line 223, in call 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp return attempt.get(self._wrap_exception) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp six.reraise(self.value[0], self.value[1], self.value[2]) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp return f(*args, **kwargs) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1706, in config_apply_runtime_manifest 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp self._apply_runtime_manifest(config_dict, hieradata_path=hieradata_path) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1775, in _apply_runtime_manifest 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp hieradata_path=hieradata_path) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 81, in puppet_apply_manifest 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp raise exception.SysinvException(_(msg)) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp SysinvException: Failed to execute runtime manifest for host 192.168.204.2 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp sysinv 2021-03-11 03:44:35.204 2688 INFO sysinv.agent.manager [-] Agent config applied 748150cb-d1ac-42e9-895b-1aee23c6ffc1 sysinv 2021-03-11 03:50:19.078 2688 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 3cf332ae-6ff2-4644-ae16-1e3a3b4a2639 {u'classes': [u'platform::mtce::runtime'], u'force': False, u'personalities': [u'controller']} controller sysinv 2021-03-11 03:50:19.079 2688 INFO sysinv.agent.manager [-] controller-standby or storage, mount /var/run/platform sysinv 2021-03-11 03:50:19.109 2688 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/var/run/platform/puppet/20.12/hieradata' sysinv 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common [-] Failed to execute runtime manifest for host 192.168.204.2: CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/var/run/platform/puppet/20.12/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmpEHuxMT.yaml']' returned non-zero exit status 1 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common Traceback (most recent call last): 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 76, in puppet_apply_manifest 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common subprocess.check_call(cmd, stdout=fnull, stderr=fnull) # pylint: disable=not-callable 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common raise CalledProcessError(retcode, cmd) 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/var/run/platform/puppet/20.12/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmpEHuxMT.yaml']' returned non-zero exit status 1 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common sysinv 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager [-] failed to apply runtime manifest: SysinvException: Failed to execute runtime manifest for host 192.168.204.2 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager Traceback (most recent call last): 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1775, in _apply_runtime_manifest 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager hieradata_path=hieradata_path) 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 81, in puppet_apply_manifest 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager raise exception.SysinvException(_(msg)) 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager SysinvException: Failed to execute runtime manifest for host 192.168.204.2 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager Brief Description ----------------- stx-openstack apply failed due to CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh' '/var/run/platform/puppet/20.12/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmp_3nV4d.yaml']' Severity -------- Major Expected Behavior ------------------ stx-openstack apply success Actual Behavior ---------------- stx-openstack apply failed Reproducibility --------------- reproducible System Configuration -------------------- Regular standard 2+2 Branch/Pull Time/Commit ----------------------- BUILD_ID="20210306T023358Z" Timestamp/Logs -------------- sysinv 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common [-] Failed to execute runtime manifest for host 192.168.204.2: CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/var/run/platform/puppet/20.12/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmp_3nV4d.yaml']' returned non-zero exit status 1 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common Traceback (most recent call last): 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 76, in puppet_apply_manifest 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common subprocess.check_call(cmd, stdout=fnull, stderr=fnull) # pylint: disable=not-callable 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common raise CalledProcessError(retcode, cmd) 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/var/run/platform/puppet/20.12/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmp_3nV4d.yaml']' returned non-zero exit status 1 2021-03-11 03:44:31.079 2688 ERROR sysinv.puppet.common sysinv 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager [-] failed to apply runtime manifest: SysinvException: Failed to execute runtime manifest for host 192.168.204.2 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager Traceback (most recent call last): 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1775, in _apply_runtime_manifest 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager hieradata_path=hieradata_path) 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 81, in puppet_apply_manifest 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager raise exception.SysinvException(_(msg)) 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager SysinvException: Failed to execute runtime manifest for host 192.168.204.2 2021-03-11 03:44:31.082 2688 ERROR sysinv.agent.manager sysinv 2021-03-11 03:44:31.172 2688 INFO sysinv.agent.manager [-] Caught exception. Retrying... Exception: Failed to execute runtime manifest for host 192.168.204.2 sysinv 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: SysinvException: Failed to execute runtime manifest for host 192.168.204.2 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last): 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp **args) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp return Retrying(*dargs, **dkw).call(f, *args, **kw) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/retrying.py", line 223, in call 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp return attempt.get(self._wrap_exception) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp six.reraise(self.value[0], self.value[1], self.value[2]) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp return f(*args, **kwargs) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1706, in config_apply_runtime_manifest 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp self._apply_runtime_manifest(config_dict, hieradata_path=hieradata_path) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1775, in _apply_runtime_manifest 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp hieradata_path=hieradata_path) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 81, in puppet_apply_manifest 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp raise exception.SysinvException(_(msg)) 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp SysinvException: Failed to execute runtime manifest for host 192.168.204.2 2021-03-11 03:44:31.173 2688 ERROR sysinv.openstack.common.rpc.amqp sysinv 2021-03-11 03:44:35.204 2688 INFO sysinv.agent.manager [-] Agent config applied 748150cb-d1ac-42e9-895b-1aee23c6ffc1 sysinv 2021-03-11 03:50:19.078 2688 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 3cf332ae-6ff2-4644-ae16-1e3a3b4a2639 {u'classes': [u'platform::mtce::runtime'], u'force': False, u'personalities': [u'controller']} controller sysinv 2021-03-11 03:50:19.079 2688 INFO sysinv.agent.manager [-] controller-standby or storage, mount /var/run/platform sysinv 2021-03-11 03:50:19.109 2688 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/var/run/platform/puppet/20.12/hieradata' sysinv 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common [-] Failed to execute runtime manifest for host 192.168.204.2: CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/var/run/platform/puppet/20.12/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmpEHuxMT.yaml']' returned non-zero exit status 1 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common Traceback (most recent call last): 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 76, in puppet_apply_manifest 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common subprocess.check_call(cmd, stdout=fnull, stderr=fnull) # pylint: disable=not-callable 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common raise CalledProcessError(retcode, cmd) 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/var/run/platform/puppet/20.12/hieradata', '192.168.204.2', 'controller', 'runtime', '/tmp/tmpEHuxMT.yaml']' returned non-zero exit status 1 2021-03-11 03:50:26.878 2688 ERROR sysinv.puppet.common sysinv 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager [-] failed to apply runtime manifest: SysinvException: Failed to execute runtime manifest for host 192.168.204.2 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager Traceback (most recent call last): 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1775, in _apply_runtime_manifest 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager hieradata_path=hieradata_path) 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 81, in puppet_apply_manifest 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager raise exception.SysinvException(_(msg)) 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager SysinvException: Failed to execute runtime manifest for host 192.168.204.2 2021-03-11 03:50:26.879 2688 ERROR sysinv.agent.manager
2021-03-11 17:04:38 Yvonne Ding attachment added ALL_NODES_20210311.035540.tar https://bugs.launchpad.net/starlingx/+bug/1918703/+attachment/5475876/+files/ALL_NODES_20210311.035540.tar
2021-03-12 22:20:57 Ghada Khalil tags stx.distro.openstack
2021-04-22 15:13:07 Ghada Khalil starlingx: assignee Eric MacDonald (rocksolidmtce)
2021-04-30 20:34:33 Ghada Khalil starlingx: assignee Eric MacDonald (rocksolidmtce)
2021-05-25 13:16:09 Austin Sun starlingx: assignee chen haochuan (martin1982)
2021-08-03 13:17:56 Austin Sun starlingx: importance Undecided Low
2024-03-11 13:02:58 Thales Elero Cervi starlingx: status New Invalid