handling of Specification result sets with 3000 rows is slow
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Storm |
New
|
Undecided
|
Unassigned |
Bug Description
A query of ours in Launchpad returns 3000 Specification objects.
They are defined like this:
class Specification(
"""See ISpecification."""
implements(
_defaultOrder = ['-priority', 'definition_
# db field names
name = StringCol(
title = StringCol(
summary = StringCol(
definition_
priority = EnumCol(
assignee = ForeignKey(
drafter = ForeignKey(
approver = ForeignKey(
owner = ForeignKey(
datecreated = UtcDateTimeCol(
private = BoolCol(
product = ForeignKey(
productseries = ForeignKey(
distribution = ForeignKey(
distroseries = ForeignKey(
goalstatus = EnumCol(
goal_proposer = ForeignKey(
date_
goal_decider = ForeignKey(
date_
milestone = ForeignKey(
specurl = StringCol(
whiteboard = StringCol(
direction_
man_days = IntCol(
implementat
superseded_by = ForeignKey(
completer = ForeignKey(
date_completed = UtcDateTimeCol(
starter = ForeignKey(
date_started = UtcDateTimeCol(
# useful joins
mentoring_
subscriptions = SQLMultipleJoin
subscribers = SQLRelatedJoin(
feedbackreq
sprint_links = SQLMultipleJoin
sprints = SQLRelatedJoin(
bug_links = SQLMultipleJoin(
bugs = SQLRelatedJoin(
linked_branches = SQLMultipleJoin
spec_
dependencies = SQLRelatedJoin(
blocked_specs = SQLRelatedJoin(
The query takes 500ms to run on the server (ok, thats a little slow) but takes between 10000ms and 12000ms to iterate in Python
kcachgrind says the following (%relative to parent)
calls method time-%-parent
3076 _load_object 87%
3072 get_obj_info 43%
3072 _add_to_alive 4%
3072 _set_values 33%
bug 618367 has the bug report in Launchpad with some details about this (but its about the page performance per se, so I may be closing that bug if I find a workaround / make the page better some other way.
description: | updated |
I've been doing some investigation and I'm going to narrow this bug right down - getting rid of all the joins still gives us a 10-12 second (under lsprof) parse time for a 3000 row result set. I'm attaching the kcachegrind which shows this pretty clearly.
To reproduce, grab lp db-stable/devel - and do a query for Specifications which returns 3000 of them; for some reason thats really quite slow - 4ms each or thereabouts.