storm processing of result sets can be very very slow
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Launchpad itself |
Triaged
|
Low
|
Unassigned | ||
Storm |
Confirmed
|
Low
|
Unassigned |
Bug Description
We've had timeouts where a couple thousand objects are returned from a fast SQL query. E.g. the sql is reported as taking 300ms, but the page times out; changing those queries to be a count() stopped the timeout;
One theory is that the there is something taking up excessive time in the db layers - dbapi/storm.
Another theory is that the DB time is actually much greater.
Looking at the tracer code that aggregates SQL time, we count as sql time the time to obtain a cursor, not the aggregate time involved in retrieving content from it.
I think that the latter theory is correct and what we need to do is to also add to the sql time the time spent in fetch* calls to the raw cursor, because in large result sets this may be batched?, and we're thus we're blaming python for time spent in the DB layers.
Related to this we may want to start calculating deserialisation
IMO this is a 'high' priority bug, because accurate information about performance is an essential tool for the development process.
Related branches
- Gustavo Niemeyer: Needs Fixing
-
Diff: 65 lines (+22/-8)2 files modifiedstorm/database.py (+17/-4)
storm/tracer.py (+5/-4)
affects: | launchpad-foundations → storm |
summary: |
- OOPS may be underrepresenting storm/sql time + storm processing of result sets can be very very slow |
Triaging as low for now, and asking for Stuart's thoughts. If he confirms your suspicions, I agree that getting this fixed is important and will raise the priority.