Launchpad itself

Codeimportworker using too many DB connections.

Bug #239569 reported by Herb McNew
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Medium
Michael Hudson-Doyle
Launchpad itself 10.03

Bug Description

It has been noticed on a couple of occassions since we rolled out the new code import system that the codeimportworker has anywhere from 30-50 DB connections open at any given time. Using today as an example there hasn't been less that 30 connections open and has ranged as high as 56. Attached you'll find a debug log with some additional information.

Related branches

lp:~mwhudson/launchpad/no-codeimportworker-db-access
Tim Penhey (community): Approve
Diff: 1534 lines (+585/-360)
15 files modified
database/schema/security.cfg (+0/-22)
lib/canonical/config/schema-lazr.conf (+2/-2)
lib/canonical/launchpad/interfaces/librarian.py (+1/-1)
lib/canonical/launchpad/xmlrpc/faults.py (+10/-0)
lib/lp/code/interfaces/codeimportscheduler.py (+26/-0)
lib/lp/code/model/tests/test_codeimportjob.py (+4/-27)
lib/lp/code/xmlrpc/codeimportscheduler.py (+66/-5)
lib/lp/code/xmlrpc/tests/test_codeimportscheduler.py (+140/-0)
lib/lp/codehosting/codeimport/dispatcher.py (+1/-1)
lib/lp/codehosting/codeimport/tests/test_workermonitor.py (+244/-179)
lib/lp/codehosting/codeimport/workermonitor.py (+58/-111)
lib/lp/codehosting/inmemory.py (+12/-2)
lib/lp/testing/__init__.py (+1/-1)
scripts/code-import-worker-monitor.py (+19/-8)
scripts/code-import-worker.py (+1/-1)
Revision history for this message
Herb McNew (herb) wrote :
Revision history for this message
Diogo Matsubara (matsubara) wrote :

Do we have a limit to the number of open connections? What problem is this causing?

Changed in launchpad:
status: New → Incomplete
Revision history for this message
Herb McNew (herb) wrote :

We had a hard limit of 180 that was recently increased to 250 for 2 addtional app servers that are about to come online. At 50+ open connections the code import system is obviously using 20% of our available connections. Further it's using 3x-5x the number of connections that any of the other software uses. As an example the app servers use 8 connections and the authserver uses 15 connections.

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

This should apparently drop to 20 connections with Storm. I chatted to Stuart about it and he didn't seem to think it was a super high priority problem.

Revision history for this message
Tim Penhey (thumper) wrote :

We'll look to make the import slaves to talk to the database through the web service.

Changed in launchpad-bazaar:
assignee: nobody → mwhudson
importance: Undecided → Medium
status: Incomplete → Confirmed
Michael Hudson-Doyle (mwhudson)
Changed in launchpad-code:
status: Triaged → In Progress
visibility: private → public
Revision history for this message
Ursula Junque (ursinha) wrote : Bug fixed by a commit

Fixed in stable r10510 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/10510>

Changed in launchpad-code:
milestone: none → 10.03
status: In Progress → Fix Committed
tags: added: qa-needstesting
Michael Hudson-Doyle (mwhudson)
tags: added: qa-ok
removed: qa-needstesting
Tim Penhey (thumper)
Changed in launchpad-code:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.