Change already reported error resulting in pipeline processing error loop
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Zuul |
Fix Committed
|
High
|
Unassigned |
Bug Description
This happened:
2014-07-26 18:08:19,961 DEBUG zuul.Scheduler: Run handler sleeping
2014-07-26 18:08:19,961 DEBUG zuul.Scheduler: Run handler awake
2014-07-26 18:08:19,961 DEBUG zuul.Independen
2014-07-26 18:08:19,961 DEBUG zuul.Independen
2014-07-26 18:08:19,963 DEBUG zuul.Independen
2014-07-26 18:08:19,963 DEBUG zuul.DependentP
2014-07-26 18:08:19,963 DEBUG zuul.DependentP
2014-07-26 18:08:19,963 DEBUG zuul.DependentP
2014-07-26 18:08:19,963 ERROR zuul.Scheduler: Exception in run handler:
Traceback (most recent call last):
File "/usr/local/
while pipeline.
File "/usr/local/
item, nnfi, ready_ahead)
File "/usr/local/
self.
File "/usr/local/
raise Exception("Already reported change %s" % item.change)
Exception: Already reported change <Change 0x7f6fd4f53e90 109738,1>
The change in question appeared to have been properly reported on the gerrit side and it was merged. But zuul would not remove it from its gate pipeline, it would error during processing of the gate pipeline, then start again and error again. This resulted in many gigs of logs in a short period of time.
Looking at https:/
This line https:/ /git.openstack. org/cgit/ openstack- infra/zuul/ tree/zuul/ scheduler. py#n1415 through and SSHException when paramiko failed to do a thing.
I think I can fix this by changing our retry logic slightly to allow us to retry on ssh failures gracefully.