celerybeat keeps transactions open forever

[William Grant]

celery has been keeping transactions open indefinitely on staging since about 12 hours ago. It needs to only keep a transaction open when it's actually accessing the DB, and it needs to use a custom DB user instead of launchpad_main.

They're starting every 10 minutes:

08:43:40 < lifeless> 630199524 | lpmain_staging | 8739 | 4697436 | launchpad_main | | | | -1 | 2012-05-23 22:37:34.912112+00 | 2012-05-23 22:37:34.919692+00 | 2012-05-23 22:37:34.919955+00 | f | <IDLE> in transaction
08:43:44 < lifeless> 630199524 | lpmain_staging | 5964 | 4697436 | launchpad_main | | | | -1 | 2012-05-23 22:27:34.850953+00 | 2012-05-23 22:27:34.858329+00 | 2012-05-23 22:27:34.858573+00 | f | <IDLE> in transaction
08:43:48 < lifeless> 630199524 | lpmain_staging | 27425 | 4697436 | launchpad_main | | | | -1 | 2012-05-23 21:37:34.551134+00 | 2012-05-23 22:17:34.821917+00 | 2012-05-23 22:17:34.822145+00 | f | <IDLE> in transaction

It's touching the job tables:

08:52:01 < lifeless> wgrant: select pg_class.relname from pg_locks, pg_class where pid=8739 and pg_locks.relation=pg_class.oid;
08:52:04 < lifeless> relname
08:52:04 < lifeless> job__status__id__key
08:52:05 < lifeless> branchjob__branch__idx
08:52:06 < lifeless> ------------------------
08:52:09 < lifeless> branchjob_job_key
08:52:11 < lifeless> job_pkey
08:52:14 < lifeless> job

The connections went away after WebOps stopped celery:

09:07:43 < wgrant> spm: Which celery processes did you kill?
09:07:58 < spm> all of them
09:08:13 < spm> /etc/init.d/celeryd-job stop
09:08:17 < wgrant> Ah
09:08:19 < spm> /etc/init.d/celeryd-job_slow stop
09:08:35 < spm> sudo /sbin/stop celerybeat-staging-bzrsyncd

[Launchpad QA Bot (lpqabot)]

Fixed in stable r15310 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15310>.