If there is an unexpected exception in a celery task, the tests on Jenkins hang
Bug #1221116 reported by
Michele Simionato
This bug affects 1 person
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| OpenQuake Engine |
Fix Released
|
High
|
Michele Simionato | ||
Bug Description
In production the supervisor looks at the celery logs and restarts a worker affected by an untrapped exception. In Jenkins however there is no such mechanism and the tests just hang. Since the celery log is not immediately visible, it is pretty difficult for the user to understand what is happening and after 10 hours the build stops with a Timeout exception.
Possible solutions are to introduce a control mechanism in the tests too or to change the way we are using celery (more radical).
Revision history for this message
| Michele Simionato (michele-simionato) wrote | #1 |
Revision history for this message
| Michele Simionato (michele-simionato) wrote | #2 |
| Changed in oq-engine: | |
| status: | New → In Progress |
| importance: | Undecided → High |
| assignee: | nobody → Michele Simionato (michele-simionato) |
| milestone: | none → 1.0.1 |
| Changed in oq-engine: | |
| status: | In Progress → Fix Committed |
| Changed in oq-engine: | |
| status: | Fix Committed → Fix Released |
To post a comment you must log in.
This will be solved automatically once we remove the old distribution and we replace it with the "parallelize" mechanism. So it does not need to be a separate bug.