celerybeat keeps transactions open forever
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Launchpad itself |
Fix Released
|
High
|
Aaron Bentley |
Bug Description
celery has been keeping transactions open indefinitely on staging since about 12 hours ago. It needs to only keep a transaction open when it's actually accessing the DB, and it needs to use a custom DB user instead of launchpad_main.
They're starting every 10 minutes:
08:43:40 < lifeless> 630199524 | lpmain_staging | 8739 | 4697436 | launchpad_main | | | | -1 | 2012-05-23 22:37:34.912112+00 | 2012-05-23 22:37:34.919692+00 | 2012-05-23 22:37:34.919955+00 | f | <IDLE> in transaction
08:43:44 < lifeless> 630199524 | lpmain_staging | 5964 | 4697436 | launchpad_main | | | | -1 | 2012-05-23 22:27:34.850953+00 | 2012-05-23 22:27:34.858329+00 | 2012-05-23 22:27:34.858573+00 | f | <IDLE> in transaction
08:43:48 < lifeless> 630199524 | lpmain_staging | 27425 | 4697436 | launchpad_main | | | | -1 | 2012-05-23 21:37:34.551134+00 | 2012-05-23 22:17:34.821917+00 | 2012-05-23 22:17:34.822145+00 | f | <IDLE> in transaction
It's touching the job tables:
08:52:01 < lifeless> wgrant: select pg_class.relname from pg_locks, pg_class where pid=8739 and pg_locks.
08:52:04 < lifeless> relname
08:52:04 < lifeless> job__status_
08:52:05 < lifeless> branchjob_
08:52:06 < lifeless> -------
08:52:09 < lifeless> branchjob_job_key
08:52:11 < lifeless> job_pkey
08:52:14 < lifeless> job
The connections went away after WebOps stopped celery:
09:07:43 < wgrant> spm: Which celery processes did you kill?
09:07:58 < spm> all of them
09:08:13 < spm> /etc/init.
09:08:17 < wgrant> Ah
09:08:19 < spm> /etc/init.
09:08:35 < spm> sudo /sbin/stop celerybeat-
Related branches
- j.c.sackett (community): Approve
-
Diff: 98 lines (+27/-14)3 files modifieddatabase/schema/security.cfg (+7/-0)
lib/lp/services/job/celeryjob.py (+12/-12)
lib/lp/services/job/tests/test_celeryjob.py (+8/-2)
Changed in launchpad: | |
importance: | High → Critical |
importance: | Critical → High |
Changed in launchpad: | |
assignee: | nobody → Aaron Bentley (abentley) |
status: | Triaged → In Progress |
summary: |
- celery keeps transactions open forever + celerybeat keeps transactions open forever |
tags: |
added: qa-ok removed: qa-needstesting |
Changed in launchpad: | |
status: | Fix Committed → Fix Released |
Fixed in stable r15310 <http:// bazaar. launchpad. net/~launchpad- pqm/launchpad/ stable/ revision/ 15310>.