Description:-
tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is failing in check/gate/periodic with [ERROR]: Container(s) which finished with wrong return code: ['container-puppet-clustercheck', 'container-puppet-haproxy', 'container-puppet-mysql', container-puppet-memcached', 'container-puppet-keystone', 'container-puppet-rabbitmq']
Build history:-
https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates
Logs:-
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_826/760994/6/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/826e14e/logs/undercloud/home/zuul/overcloud_update_run-Controller.log
~~~
2020-11-03 19:30:16 | These containers are supposed to terminate with a valid exit code and will be checked: ['container-puppet-clustercheck', 'container-puppet-haproxy', 'container-puppet-mysql', 'container-puppet-memcached', 'container-puppet-keystone', 'container-puppet-rabbitmq']
2020-11-03 19:30:18 | [ERROR]: Container(s) which finished with wrong return code: ['container-
2020-11-03 19:30:18 | puppet-clustercheck', 'container-puppet-haproxy', 'container-puppet-mysql',
2020-11-03 19:30:18 | 'container-puppet-memcached', 'container-puppet-keystone', 'container-puppet-
2020-11-03 19:30:18 | rabbitmq']
2020-11-03 19:30:18 | 2020-11-03 19:30:18.601316 | fa163e00-3606-63d5-9328-000000000c44 | FATAL | Check containers status | centos-8-limestone-regionone-0021442577 | error={"changed": false, "msg": "Failed container(s): ['container-puppet-keystone', 'container-puppet-memcached', 'container-puppet-clustercheck', 'container-puppet-rabbitmq', 'container-puppet-mysql', 'container-puppet-haproxy'], check logs in /var/log/containers/stdouts/"}
2020-11-03 19:30:18 | 2020-11-03 19:30:18.603293 | fa163e00-3606-63d5-9328-000000000c44 | TIMING | tripleo_container_manage : Check containers status | centos-8-limestone-regionone-0021442577 | 0:06:59.663136 | 2.84s
2020-11-03 19:30:18 |
~~~
Another logs:-
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/591edba/logs/undercloud/home/zuul/overcloud_update_run-Controller.log.txt.gz
~~~
020-11-04 02:19:05 | These containers are supposed to terminate with a valid exit code and will be checked: ['container-puppet-keystone', 'container-puppet-clustercheck', 'container-puppet-rabbitmq', 'container-puppet-memcached', 'container-puppet-mysql', 'container-puppet-haproxy']
2020-11-04 02:19:06 | [ERROR]: Container(s) which finished with wrong return code: ['container-
2020-11-04 02:19:06 | puppet-keystone', 'container-puppet-clustercheck', 'container-puppet-rabbitmq',
2020-11-04 02:19:06 | 'container-puppet-memcached', 'container-puppet-mysql', 'container-puppet-
2020-11-04 02:19:06 | haproxy']
2020-11-04 02:19:06 | 2020-11-04 02:19:06.624623 | fa163eda-0efd-57b1-6467-000000000c40 | FATAL | Check containers status | upstream-centos-8-rdo-cloud-tripleo-0000556666 | error={"changed": false, "msg": "Failed container(s): ['container-puppet-rabbitmq', 'container-puppet-haproxy', 'container-puppet-keystone', 'container-puppet-mysql', 'container-puppet-memcached', 'container-puppet-clustercheck'], check logs in /var/log/containers/stdouts/"}
2020-11-04 02:19:06 | 2020-11-04 02:19:06.625669 | fa163eda-0efd-57b1-6467-000000000c40 | TIMING | tripleo_container_manage : Check containers status | upstream-centos-8-rdo-cloud-tripleo-0000556666 | 0:02:55.217602 | 0.99s
~~~
Hi,
We are seeing the following error in logs:-
https:/ /logserver. rdoproject. org/openstack- periodic- integration- main/opendev. org/openstack/ tripleo- ci/master/ periodic- tripleo- ci-centos- 8-scenario000- multinode- oooq-container- updates- master/ 591edba/ logs/subnode- 1/var/log/ extra/podman/ containers/ container- puppet- haproxy/ stdout. log.txt. gz
~~~ init_linux. go:211: exec user process caused "exec format error"
tar: Removing leading `/' from member names
+ rm -rf /tmp/tmp.zY0jxgMeAB /tmp/tmp.2ad0Mw8DWe
standard_
~~~
Similiar issue is reported https:/ /bugs.launchpad .net/tripleo/ +bug/1902809
Architecture is different now.. when comparing current vs old good job logs
https:/ /logserver. rdoproject. org/openstack- periodic- integration- main/opendev. org/openstack/ tripleo- ci/master/ periodic- tripleo- ci-centos- 8-scenario000- multinode- oooq-container- updates- master/ 72f9805/ logs/subnode- 1/var/log/ extra/podman/ containers/ container- puppet- clustercheck/ podman_ info.log. txt.gz
~~~
"architecture": "x86_64",
~~~
https:/ /logserver. rdoproject. org/openstack- periodic- integration- main/opendev. org/openstack/ tripleo- ci/master/ periodic- tripleo- ci-centos- 8-scenario000- multinode- oooq-container- updates- master/ 591edba/ logs/subnode- 1/var/log/ extra/podman/ containers/ container- puppet- haproxy/ podman_ info.log. txt.gz
~~~
"architecture": "ppc64le",
~~~