Jobs are not stopped by a celery kill
Bug #1292606 reported by
Daniele Viganò
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenQuake Engine |
Fix Released
|
Critical
|
Michele Simionato |
Bug Description
When you kill celery on the workers the job on the masters doesn't exit and keep running waiting for data. This behavior probably depends on the switch to iterator_native() which is not polling the workers anymore (the old iterator did)
Changed in oq-engine: | |
importance: | Medium → Critical |
status: | New → Confirmed |
Changed in oq-engine: | |
status: | Confirmed → In Progress |
Changed in oq-engine: | |
status: | In Progress → Fix Committed |
Changed in oq-engine: | |
status: | Fix Committed → Fix Released |
To post a comment you must log in.
We found two solutions:
- Modify celery to add a callback to drain_events() in backends/amqp.py [iter_native() -> get_many() -> drain_events()]
- Create a supervisor as a forked thread/process which monitor the workers status