Intermittent errors on Gerrit side

[Yuriy Taraday]

sync-upstream-openstack job fails sometimes because Gerrit returns some internal server error. These are taken from job log at http://osci-jenkins.srt.mirantis.net:8080/job/sync-upstream-openstack/34854/consoleFull :

1439996808: #oslo.version! fatal: internal server error
1439996808: #oslo.concurrency! fatal: internal server error
1439996808: #oslo.log! fatal: internal server error
1439996808: #oslo.context! fatal: internal server error
1439996845: #oslo.messaging! fatal: internal server error
1439996845: #neutron-lbaas! fatal: internal server error

Those numbers are UNIX timestamps, so you can search specific time in Gerrit logs.

Please fix them.

[Igor Shishkin (teran)]

@Yury, please provide more details since job logs are not available and couldn't find a problems in last few builds of the job:

- which repos are affected
- which gerrit reports such errors
- particular refs causes such problems
- was that problem once or its persistent?

[Yuriy Taraday (yorik-sar)]

Last failure was registered @ Aug 29, 2015, 10:39 AM, so logs are not available anymore. I'll pin new failure of that job once it appears. Answers to your questions follow.

- which repos are affected

The failure that I cited in bug description was triggered by these repos (line from log forwarded to my mail):

FAILED: oslo.version oslo.concurrency oslo.log oslo.context oslo.policy oslo.messaging neutron-lbaas barbican designate

From the looks of it, it affects different repos every time, so it should be Gerrit's fault, not repo's.

- which gerrit reports such errors

This job works with our downstream Gerrit only, review.fuel-infra.org.

- particular refs causes such problems

Seems changing over time.

- was that problem once or its persistent?

The problem is intermittent, it happens now and then, but still happens some time (last time - 2 days ago).

[Mike Scherbakov (mihgen)]

Is this still an issue folks?

[Yuriy Taraday (yorik-sar)]

I'm getting sync-upstream-openstack failures about once in a week currently, but I didn't catch logs from them yet. I'll look into new errors once they occur and close this issue as invalid if there're no issues related to Gerrit in 2 weeks from now.

[Yuriy Taraday (yorik-sar)]

There were no related failures these weeks, so I'm marking this invalid.