can't really "kill" a job
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Ubuntu CI Engine |
Fix Released
|
High
|
Andy Doan | ||
| Ubuntu CI Services |
Fix Released
|
High
|
Andy Doan | ||
Bug Description
The lander jobs running in Jenkins sometimes need to be cancelled. In the case I'm seeing I submitted a job that's going to run forever (separate bug being filed). The problem is that if I kill the jenkins job, the rabbit-worker doing the task will still keep chugging along.
The best thing we can do for now, is to restart the worker and have it wait for something new. The problem is that the workers currently don't catch SIGINT and will exit without acking the rabbit message. So the next time it starts, its going to go right back to work on the hung request.
For now, we should handle SIGINT and ack the rabbit message. This way when it comes back up it will be ready for new work.
post phase-0: we need to re-think these workers/
Related branches
- Vincent Ladeuil (community): Approve
- PS Jenkins bot (community): Approve (continuous-integration)
-
Diff: 83 lines (+15/-5)4 files modifiedbranch-source-builder/run_worker (+4/-1)
charms/precise/rabbitmq-worker/hooks/hooks.py (+3/-0)
image-builder/run_worker (+4/-1)
test_runner/run_worker (+4/-3)
| Changed in ubuntu-ci-services-itself: | |
| importance: | Undecided → High |
| Changed in ubuntu-ci-services-itself: | |
| status: | New → In Progress |
| Changed in ubuntu-ci-services-itself: | |
| milestone: | none → phase-0 |
| Changed in ubuntu-ci-services-itself: | |
| status: | In Progress → Fix Committed |
| Andy Doan (doanac) wrote | #1 |
| Changed in ubuntu-ci-services-itself: | |
| status: | Fix Committed → Fix Released |
| milestone: | phase-0 → backlog |
| status: | Fix Released → Confirmed |
| assignee: | Andy Doan (doanac) → nobody |
| Andy Doan (doanac) wrote | #2 |
| Changed in ubuntu-ci-services-itself: | |
| status: | Confirmed → Fix Released |
| Changed in ubuntu-ci-services-itself: | |
| assignee: | nobody → Andy Doan (doanac) |
| milestone: | backlog → phase-0 |
| Changed in uci-engine: | |
| assignee: | nobody → Andy Doan (doanac) |
| importance: | Undecided → High |
| milestone: | none → phase-0 |
| status: | New → Fix Released |
I'm actually going to re-open this for post phase0. The solution i've done for now is a bandaid.