If Jenkins is down at submission, the ticket gets stuck in queued forever
Bug #1289296 reported by
Evan
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu CI Engine |
Fix Released
|
High
|
Andy Doan | ||
Ubuntu CI Services |
Fix Released
|
High
|
Andy Doan |
Bug Description
I ran a deployment and Jenkins failed to start due to a race (as far as we can tell). I hadn't noticed this and tried to feed a ticket through, which is still sitting in the queued state an hour later, long after restarting Jenkins.
Nothing in the rabbit queues.
Nothing queued up in Jenkins.
Related branches
lp:~doanac/ubuntu-ci-services-itself/lander-retry-1289296
- Francis Ginther: Approve
- PS Jenkins bot (community): Approve (continuous-integration)
-
Diff: 54 lines (+4/-19)1 file modifiedlander/lander/run_worker.py (+4/-19)
Changed in ubuntu-ci-services-itself: | |
importance: | Undecided → Critical |
milestone: | none → phase-0 |
assignee: | nobody → Andy Doan (doanac) |
Changed in ubuntu-ci-services-itself: | |
status: | In Progress → Fix Committed |
Changed in ubuntu-ci-services-itself: | |
status: | Fix Committed → Fix Released |
tags: | added: airline |
Changed in uci-engine: | |
assignee: | nobody → Andy Doan (doanac) |
importance: | Undecided → High |
milestone: | none → phase-0 |
status: | New → Fix Released |
To post a comment you must log in.
i wish we had the lander's run_worker log for this, but i think this might be easy to re-create. I suspect this could be our issue:
http:// bazaar. launchpad. net/~canonical- ci-engineering/ ubuntu- ci-services- itself/ trunk/view/ head:/lander/ lander/ run_worker. py#L125
Looks like we try a fixed amount of times to contact jenkins before we give up. I'm thinking we probably want these loops to be "while True" type things.
It would be nice to have this contribute to our "engine health" page in the webui. This service doesn't have a REST api running though, so it might not be possible for phase0