multiple rdo master promotion jobs failing to pull containers
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Critical
|
Unassigned |
Bug Description
In this morning's promotion run many of the master promotion periodics failed during either undercloud instal or standalone deploy depending on job - logs at [1-5] below with trace like:
2019-05-23 04:07:34 | "2019-05-23 04:07:19,965 ERROR: 13834 -- ['/usr/bin/podman', 'start', '-a', u'container-
2019-05-23 04:07:34 | "2019-05-23 04:07:19,966 WARNING: 13834 -- Retrying running container: rabbitmq",
2019-05-23 04:07:34 | "2019-05-23 04:07:19,966 ERROR: 13834 -- Failed running container for rabbitmq", 2019-05-23 04:07:34 | "2019-05-23 04:07:32,508 ERROR: 13826 -- ERROR configuring swift_ringbuilder",
2019-05-23 04:07:34 | "2019-05-23 04:07:32,509 ERROR: 13826 -- ERROR configuring swift",
2019-05-23 04:07:34 | "2019-05-23 04:07:32,509 ERROR: 13826 -- ERROR configuring zaqar",
2019-05-23 04:07:34 | "2019-05-23 04:07:32,509 ERROR: 13826 -- ERROR configuring rabbitmq"
2019-05-23 04:07:34 | ]
Apparently related to load 14:53 #oooq: < ykarel> marios_|ruck, yes, see #rhos-ops, seems related to load, jpena and kforde looking
Bug filed as this blocks master promotion which is already very red.
[1] http://
[2] http://
[3] http://
[4] http://
[5] http://
periodic- tripleo- centos- 7-master- promote- consistent- to-tripleo- ci-testing openstack/ tripleo- ci master openstack- periodic- master master 123 2019-05-23T12:40:27 SUCCESS tripleo- fedora- 28-master- containers- build-push openstack/ tripleo- ci master openstack- periodic- master master 3992 2019-05-23T12:45:31 SUCCESS tripleo- centos- 7-master- containers- build-push openstack/ tripleo- ci master openstack- periodic- master master 3901 2019-05-23T12:50:07 SUCCESS tripleo- ci-centos- 7-standalone- upgrade- master openstack/ tripleo- ci master openstack- periodic- master master 5662 2019-05-23T13:57:02 SUCCESS tripleo- ci-centos- 7-scenario004- standalone- master openstack/ tripleo- ci master openstack- periodic- master master 3964 2019-05-23T13:56:40 SUCCESS tripleo- ci-centos- 7-scenario003- standalone- master openstack/ tripleo- ci master openstack- periodic- master master 3161 2019-05-23T13:57:04 SUCCESS tripleo- ci-centos- 7-scenario002- standalone- master openstack/ tripleo- ci master openstack- periodic- master master 3823 2019-05-23T13:56:36 SUCCESS tripleo- ci-centos- 7-scenario001- standalone- master openstack/ tripleo- ci master openstack- periodic- master master 3815 2019-05-23T13:56:38 SUCCESS tripleo- ci-fedora- 28-standalone- master openstack/ tripleo- ci master openstack- periodic- master master 3561 2019-05-23T13:52:51 SUCCESS tripleo- ci-centos- 7-standalone- full-tempest- master openstack/ tripleo- ci master openstack- periodic- master master 9513 2019-05-23T13:56:36 SUCCESS tripleo- ci-centos- 7-standalone- master openstack/ tripleo- ci master openstack- periodic- master master 3593 2019-05-23T13:57:21 SUCCESS tripleo- ci-centos- 7-singlenode- featureset050- upgrades- master openstack/ tripleo- ci master openstack- periodic- master master 2484 2019-05-23T13:56:44 SUCCESS tripleo- ci-centos- 7-ovb-3ctlr_ 1comp_1supp- featureset039- master openstack/ tripleo- ci master openstack- periodic- master master 9721 2019-05-23T13:57:22 SUCCESS tripleo- ci-centos- 7-multinode- 1ctlr-featurese t037-updates- master openstack/ tripleo- ci master openstack- periodic- master master 4557 2019-05-23T13:57:21 SUCCESS tripleo- ci-centos- 7-ovb-3ctlr_ 1comp-featurese t035-master openstack/ tripleo- ci master openstack- periodic- master master 8903 2019-05-23T13:56:53 SUCCESS tripleo- ci-centos- 7-multinode- 1ctlr-featurese t030-master openstack/ tripleo- ci master openstack- periodic- master master 4639 2019-05-23T13:57:13 SUCCESS tripleo- ci-centos- 7-singlenode- featureset027- master openstack/ tripleo- ci master openstack- periodic- master master 3196 2019-05-23T13:56:37 SUCCESS
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-