multiple rdo master promotion jobs failing to pull containers

Bug #1830198 reported by Marios Andreou
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned

Bug Description

In this morning's promotion run many of the master promotion periodics failed during either undercloud instal or standalone deploy depending on job - logs at [1-5] below with trace like:

    2019-05-23 04:07:34 | "2019-05-23 04:07:19,965 ERROR: 13834 -- ['/usr/bin/podman', 'start', '-a', u'container-puppet-rabbitmq'] run failed after Error: unable to find container container-puppet-rabbitmq: no container with name or ID container-puppet-rabbitmq found: no such container",
    2019-05-23 04:07:34 | "2019-05-23 04:07:19,966 WARNING: 13834 -- Retrying running container: rabbitmq",
    2019-05-23 04:07:34 | "2019-05-23 04:07:19,966 ERROR: 13834 -- Failed running container for rabbitmq", 2019-05-23 04:07:34 | "2019-05-23 04:07:32,508 ERROR: 13826 -- ERROR configuring swift_ringbuilder",
    2019-05-23 04:07:34 | "2019-05-23 04:07:32,509 ERROR: 13826 -- ERROR configuring swift",
    2019-05-23 04:07:34 | "2019-05-23 04:07:32,509 ERROR: 13826 -- ERROR configuring zaqar",
    2019-05-23 04:07:34 | "2019-05-23 04:07:32,509 ERROR: 13826 -- ERROR configuring rabbitmq"
    2019-05-23 04:07:34 | ]

Apparently related to load 14:53 #oooq: < ykarel> marios_|ruck, yes, see #rhos-ops, seems related to load, jpena and kforde looking

Bug filed as this blocks master promotion which is already very red.

[1] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/fd175a3/logs/undercloud/home/zuul/undercloud_install.log.txt.gz
[2] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario002-standalone-master/076d409/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
[3] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/01c2f34/logs/undercloud/home/zuul/undercloud_install.log.txt.gz
[4] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/ec148c2/logs/undercloud/home/zuul/undercloud_install.log.txt.gz
[5] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/88d0794/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

Revision history for this message
wes hayutin (weshayutin) wrote :

periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing openstack/tripleo-ci master openstack-periodic-master master 123 2019-05-23T12:40:27 SUCCESS
periodic-tripleo-fedora-28-master-containers-build-push openstack/tripleo-ci master openstack-periodic-master master 3992 2019-05-23T12:45:31 SUCCESS
periodic-tripleo-centos-7-master-containers-build-push openstack/tripleo-ci master openstack-periodic-master master 3901 2019-05-23T12:50:07 SUCCESS
periodic-tripleo-ci-centos-7-standalone-upgrade-master openstack/tripleo-ci master openstack-periodic-master master 5662 2019-05-23T13:57:02 SUCCESS
periodic-tripleo-ci-centos-7-scenario004-standalone-master openstack/tripleo-ci master openstack-periodic-master master 3964 2019-05-23T13:56:40 SUCCESS
periodic-tripleo-ci-centos-7-scenario003-standalone-master openstack/tripleo-ci master openstack-periodic-master master 3161 2019-05-23T13:57:04 SUCCESS
periodic-tripleo-ci-centos-7-scenario002-standalone-master openstack/tripleo-ci master openstack-periodic-master master 3823 2019-05-23T13:56:36 SUCCESS
periodic-tripleo-ci-centos-7-scenario001-standalone-master openstack/tripleo-ci master openstack-periodic-master master 3815 2019-05-23T13:56:38 SUCCESS
periodic-tripleo-ci-fedora-28-standalone-master openstack/tripleo-ci master openstack-periodic-master master 3561 2019-05-23T13:52:51 SUCCESS
periodic-tripleo-ci-centos-7-standalone-full-tempest-master openstack/tripleo-ci master openstack-periodic-master master 9513 2019-05-23T13:56:36 SUCCESS
periodic-tripleo-ci-centos-7-standalone-master openstack/tripleo-ci master openstack-periodic-master master 3593 2019-05-23T13:57:21 SUCCESS
periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-master openstack/tripleo-ci master openstack-periodic-master master 2484 2019-05-23T13:56:44 SUCCESS
periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-master openstack/tripleo-ci master openstack-periodic-master master 9721 2019-05-23T13:57:22 SUCCESS
periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master openstack/tripleo-ci master openstack-periodic-master master 4557 2019-05-23T13:57:21 SUCCESS
periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master openstack/tripleo-ci master openstack-periodic-master master 8903 2019-05-23T13:56:53 SUCCESS
periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master openstack/tripleo-ci master openstack-periodic-master master 4639 2019-05-23T13:57:13 SUCCESS
periodic-tripleo-ci-centos-7-singlenode-featureset027-master openstack/tripleo-ci master openstack-periodic-master master 3196 2019-05-23T13:56:37 SUCCESS

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.