misconfiguration of the new periodic build containers job preventing periodic pipeline to trigger

Bug #1818646 reported by Gabriele Cerami on 2019-03-05
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Gabriele Cerami

Bug Description

periodic pipeline last triggered around 9 AM GMT on 4th March, and it hasn't triggered since.
First I attributed this to the recent rdocloud shortage, but it's now solved.
Asking zuul admins for anomalies, they found this on zuul logs.

2019-03-05 09:10:01,848 DEBUG zuul.layout: Pipeline variant <Job periodic-tripleo-centos-7-master-containers-build-push branches: None source: config/zuul.d/tripleo.yaml@mas
ter#10> matched <Branch 0x7f2a3a756128 openstack-infra/tripleo-ci refs/heads/master updated None..None>
2019-03-05 09:10:01,849 ERROR zuul.Pipeline.rdoproject.org.openstack-periodic: Error freezing job graph for <QueueItem 0x7f2a43d228d0 for <Branch 0x7f2a3a756128 openstack-in
fra/tripleo-ci refs/heads/master updated None..None> in openstack-periodic>
2019-03-05 09:10:01,850 DEBUG zuul.Pipeline.rdoproject.org.openstack-periodic: Reporting change <Branch 0x7f2a3a756128 openstack-infra/tripleo-ci refs/heads/master updated N
one..None>
2019-03-05 09:10:01,850 DEBUG zuul.layout: Project <ProjectConfig git.openstack.org/openstack-infra/tripleo-ci source: config/zuul.d/tripleo.yaml@master None> matched item <
QueueItem 0x7f2a43d228d0 for <Branch 0x7f2a3a756128 openstack-infra/tripleo-ci refs/heads/master updated None..None> in openstack-periodic>
2019-03-05 09:10:01,850 DEBUG zuul.Pipeline.rdoproject.org.openstack-periodic: Invalid config for change <Branch 0x7f2a3a756128 openstack-infra/tripleo-ci refs/heads/master
updated None..None>
2019-03-05 09:10:01,850 INFO zuul.Pipeline.rdoproject.org.openstack-periodic: Reporting item <QueueItem 0x7f2a43d228d0 for <Branch 0x7f2a3a756128 openstack-infra/tripleo-ci
refs/heads/master updated None..None> in openstack-periodic>, actions: [<zuul.driver.sql.sqlreporter.SQLReporter object at 0x7f2a3b17c4a8>]
2019-03-05 09:10:02,507 DEBUG zuul.Pipeline.rdoproject.org.openstack-periodic: Removing change <Branch 0x7f2a3a756128 openstack-infra/tripleo-ci refs/heads/master updated No
ne..None> from queue
2019-03-05 09:10:02,507 DEBUG zuul.Pipeline.rdoproject.org.openstack-periodic: <QueueItem 0x7f2a43d228d0 for <Branch 0x7f2a3a756128 openstack-infra/tripleo-ci refs/heads/mas
ter updated None..None> in openstack-periodic> is a failing item because ['it has an invalid configuration']
2019-03-05 09:10:02,507 DEBUG zuul.Pipeline.rdoproject.org.openstack-periodic: Finished queue processor: openstack-periodic (changed: True)

some kind of misconfiguration on the new periodic-tripleo-centos-7-master-containers-build-push and its interaction with upstream tripleo-ci, is causing zuul to refuse triggering the periodic pipeline

Gabriele Cerami (gcerami) wrote :

Probable cause is this patch

https://review.rdoproject.org/r/18975

that was merged in a time compatible with the breakage

https://review.rdoproject.org/r/19101

to revert it was proposed and merged

Marios Andreou (marios-b) wrote :

thanks for heads up on irc, this just happened on oooq. Its not 100% clear if that was the root yet based on the conversation but i posted the revert and it already merged lets see. copy/pasta from #oooq just now:

13:42 <+panda|ruck|flu> marios: opened https://bugs.launchpad.net/tripleo/+bug/1818646 to track it
13:42 <@openstack> Launchpad bug 1818646 in tripleo "misconfiguration of the new periodic build containers job preventing periodic pipeline to trigger" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami)
13:42 < marios> panda|ruck|flu: ack
13:42 <+panda|ruck|flu> marios: I'm sorry to to have got this first, it's been a busy monday, and usually when you apply something to config, you should check that everything is working well
13:43 < marios> panda|ruck|flu: sure but how
13:43 < marios> panda|ruck|flu: we can't see things there until they merge
13:43 <+panda|ruck|flu> marios: after merge, ensure that the job is passing your point
13:43 < marios> panda|ruck|flu: k this is what we doing today? we were waiting for it to run so we could see
13:43 <+panda|ruck|flu> marios: don't merge and forget :)
13:43 < marios> panda|ruck|flu: i did not
13:43 < marios> panda|ruck|flu: merge and forget
13:43 < marios> panda|ruck|flu: i object to that
13:44 <+panda|ruck|flu> marios: objection overruled!
13:44 <+panda|ruck|flu> marios: objection overruled!
13:44 < marios> panda|ruck|flu: it merged last night. its now slightly passed noon here. so...
13:44 < marios> panda|ruck|flu: and its periodic, and there was rdo outage.
13:44 <+panda|ruck|flu> it merged last night ?
13:44 <+panda|ruck|flu> mmhh,, maybe ti's not it then ...
13:44 <+panda|ruck|flu> Zuul CI <email address hidden>: Change has been successfully merged by
13:45 <+panda|ruck|flu> Zuul CI (2019-03-04 14:18:39+0000) < Reply >
13:45 < marios> panda|ruck|flu: https://review.rdoproject.org/r/#/c/18975/
13:45 < quiquell> panda|ruck|flu, marios: we don't have a log of the missoncifugration from #rdo ?
13:45 <+panda|ruck|flu> quiquell: I have a crypting log pasted from jpena
13:45 <+panda|ruck|flu> quiquell: marios https://softwarefactory-project.io/paste/show/1460/
13:46 <+panda|ruck|flu> marios: anyway yes, rdocloud outage threw me off too, and no blame in general. let's fix this.
13:47 < quiquell> marios, panda|ruck|flu: We have to test the job at a test review
13:46 <+panda|ruck|flu> marios: anyway yes, rdocloud outage threw me off too, and no blame in general. let's fix this.
13:47 < quiquell> marios, panda|ruck|flu: We have to test the job at a test review
13:48 < marios> panda|ruck|flu: err ok except the bit where you accused me of merge and forget. like 13:43 <+panda|ruck|flu> marios: don't merge and forget :)
13:48 < marios> panda|ruck|flu: sure appology accepted :)

Gabriele Cerami (gcerami) wrote :

Periodic pipeline triggered after the revert.

The patch is anyway really critical, and there seems to be nothing wrong with it after deep scrutiny
We'll need to proceed slowly with smaller increments on it.

Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers