tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is failing in check/gate/periodic with [ERROR]: Container(s) which finished with wrong return code: ['container-puppet-clustercheck', 'container-puppet-haproxy', 'container-puppet-mysql', container-puppet-memcached', 'container-puppet-keystone', 'container-puppet-rabbitmq']

[Sandeep Yadav]

Description:-

tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is failing in check/gate/periodic with [ERROR]: Container(s) which finished with wrong return code: ['container-puppet-clustercheck', 'container-puppet-haproxy', 'container-puppet-mysql', container-puppet-memcached', 'container-puppet-keystone', 'container-puppet-rabbitmq']

Build history:-

https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates

Logs:-

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_826/760994/6/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/826e14e/logs/undercloud/home/zuul/overcloud_update_run-Controller.log

~~~
2020-11-03 19:30:16 | These containers are supposed to terminate with a valid exit code and will be checked: ['container-puppet-clustercheck', 'container-puppet-haproxy', 'container-puppet-mysql', 'container-puppet-memcached', 'container-puppet-keystone', 'container-puppet-rabbitmq']
2020-11-03 19:30:18 | [ERROR]: Container(s) which finished with wrong return code: ['container-
2020-11-03 19:30:18 | puppet-clustercheck', 'container-puppet-haproxy', 'container-puppet-mysql',
2020-11-03 19:30:18 | 'container-puppet-memcached', 'container-puppet-keystone', 'container-puppet-
2020-11-03 19:30:18 | rabbitmq']
2020-11-03 19:30:18 | 2020-11-03 19:30:18.601316 | fa163e00-3606-63d5-9328-000000000c44 | FATAL | Check containers status | centos-8-limestone-regionone-0021442577 | error={"changed": false, "msg": "Failed container(s): ['container-puppet-keystone', 'container-puppet-memcached', 'container-puppet-clustercheck', 'container-puppet-rabbitmq', 'container-puppet-mysql', 'container-puppet-haproxy'], check logs in /var/log/containers/stdouts/"}
2020-11-03 19:30:18 | 2020-11-03 19:30:18.603293 | fa163e00-3606-63d5-9328-000000000c44 | TIMING | tripleo_container_manage : Check containers status | centos-8-limestone-regionone-0021442577 | 0:06:59.663136 | 2.84s
2020-11-03 19:30:18 |
~~~

Another logs:-

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/591edba/logs/undercloud/home/zuul/overcloud_update_run-Controller.log.txt.gz

~~~
020-11-04 02:19:05 | These containers are supposed to terminate with a valid exit code and will be checked: ['container-puppet-keystone', 'container-puppet-clustercheck', 'container-puppet-rabbitmq', 'container-puppet-memcached', 'container-puppet-mysql', 'container-puppet-haproxy']
2020-11-04 02:19:06 | [ERROR]: Container(s) which finished with wrong return code: ['container-
2020-11-04 02:19:06 | puppet-keystone', 'container-puppet-clustercheck', 'container-puppet-rabbitmq',
2020-11-04 02:19:06 | 'container-puppet-memcached', 'container-puppet-mysql', 'container-puppet-
2020-11-04 02:19:06 | haproxy']
2020-11-04 02:19:06 | 2020-11-04 02:19:06.624623 | fa163eda-0efd-57b1-6467-000000000c40 | FATAL | Check containers status | upstream-centos-8-rdo-cloud-tripleo-0000556666 | error={"changed": false, "msg": "Failed container(s): ['container-puppet-rabbitmq', 'container-puppet-haproxy', 'container-puppet-keystone', 'container-puppet-mysql', 'container-puppet-memcached', 'container-puppet-clustercheck'], check logs in /var/log/containers/stdouts/"}
2020-11-04 02:19:06 | 2020-11-04 02:19:06.625669 | fa163eda-0efd-57b1-6467-000000000c40 | TIMING | tripleo_container_manage : Check containers status | upstream-centos-8-rdo-cloud-tripleo-0000556666 | 0:02:55.217602 | 0.99s
~~~

[Sandeep Yadav (sandeepyadav93)]

Hi,

We are seeing the following error in logs:-

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/591edba/logs/subnode-1/var/log/extra/podman/containers/container-puppet-haproxy/stdout.log.txt.gz

~~~
tar: Removing leading `/' from member names
+ rm -rf /tmp/tmp.zY0jxgMeAB /tmp/tmp.2ad0Mw8DWe
standard_init_linux.go:211: exec user process caused "exec format error"
~~~

Similiar issue is reported https://bugs.launchpad.net/tripleo/+bug/1902809

Architecture is different now.. when comparing current vs old good job logs

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/72f9805/logs/subnode-1/var/log/extra/podman/containers/container-puppet-clustercheck/podman_info.log.txt.gz
~~~
"architecture": "x86_64",
~~~

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/591edba/logs/subnode-1/var/log/extra/podman/containers/container-puppet-haproxy/podman_info.log.txt.gz
~~~
"architecture": "ppc64le",
~~~

[Sandeep Yadav (sandeepyadav93)]

Hello,

On investigation we found there is an another job tripleo-upstream-containers-build-master-ppc64le which is pushing ppc container to rdo registy. Because this job is pushing with same name as non ppc - its overriding non ppc containers.

https://logserver.rdoproject.org/ci.centos.org/tripleo-upstream-containers-build-master-ppc64le/2816/logs/logs/containers-successfully-built.log

~~~
trunk.registry.rdoproject.org/tripleomaster/openstack-haproxy 68f97388ca61ef3b5c419b6480308f6a 41165c2d5263 49 minutes ago 774 MB
~~~

~~~
[root@localhost ~]# skopeo inspect docker://trunk.registry.rdoproject.org/tripleomaster/openstack-haproxy:68f97388ca61ef3b5c419b6480308f6a | grep -i arch
        "architecture": "ppc64le",
    "Architecture": "ppc64le",
~~~

Patch is up to disable container push in tripleo-upstream-containers-build-master-ppc64le job - https://review.rdoproject.org/r/#/c/30761/

[Sandeep Yadav (sandeepyadav93)]

Hello,

We repushed x86 arch containers for current-tripleo hash to clear issue we were hitting in check/gate jobs.

Also, we disabled the job to push container - tripleo-upstream-containers-build-master-ppc64le with https://review.rdoproject.org/r/#/c/30761/

In long term, tripleo-upstream-containers-build-master-ppc64le job needs to be modified to push ppc container with correct suffix so that it will not overwrire non-ppc containers.

[wes hayutin (weshayutin)]

https://review.opendev.org/#/c/761446/