Archive.getPublishedBinaries(created_since_date=) is very slow

Bug #1441729 reported by Colin Watson on 2015-04-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Critical
Colin Watson
lazr.restful
High
Colin Watson

Bug Description

To cope with the new world order for ddebs, ddeb-retriever is going to need to use Archive.getPublishedBinaries(status="Published", created_since_date=threshold, ordered=False) to catch up with recent binary publications. Unfortunately, this is very slow right now; I've been trying to use 2015-03-15 as a fairly arbitrary test threshold (not remembering off the top of my head exactly when the qastaging database was last restored), and I can't get that to not time out on qastaging.

Example timeout: OOPS-4a160fb13a4f966d24123318cd4c46eb

I think it would be sensible to index BPPH.datecreated to speed this up.

Related branches

Colin Watson (cjwatson) on 2015-04-08
tags: added: oops performance soyuz-publish timeout
Changed in launchpad:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Colin Watson (cjwatson)
Colin Watson (cjwatson) wrote :

William's suggestion in https://code.launchpad.net/~cjwatson/launchpad/db-index-bpph-datecreated/+merge/255539/comments/635978/+download requires some lazr.restful work to make it possible to slot in StormRangeFactory.

Changed in lazr.restful:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Colin Watson (cjwatson)
Colin Watson (cjwatson) wrote :

12:08 <cjwatson> wgrant: I'm giving up on the StormRangeFactory bit for now. I do actually now have something that appears to work - haven't tried with launchpadlib, but the next_collection_link entries have elaborate-looking memos with full dates in them. The
                 tricky bit is that it breaks if you pass the "wrong" parameters, because e.g. BPN isn't in the resultset so SRF can't cope with a resultset ordered by BPN, and at the moment I'm inserting the ...
12:08 <cjwatson> ... range factory in the interface so it doesn't get to be parameter-sensitive.
12:08 <cjwatson> Maybe this is fixable with a DecoratedResultSet or something, but I've run past the end of my timebox for this.
12:10 <cjwatson> I'll just push up something today that implements order_by_date without that.
12:11 <wgrant> cjwatson: I think launchpadlib is a bit magical, since it supports slicing.
12:12 <wgrant> But yeah, SRF for the default sort order is awkward.
12:12 <wgrant> (and pointless, since it can't be sorted by an index)

Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
Changed in launchpad:
status: In Progress → Fix Committed
Colin Watson (cjwatson) on 2015-04-13
tags: added: qa-ok
removed: qa-needstesting
Colin Watson (cjwatson) on 2015-04-14
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers