Timed out while loading subscriptions for most-common-problems

[Evan]

[12:20:38] <Laney> ev: trouble> "An error occurred while trying to load the most common problems" when I'm logged in ;-)
[12:21:00] <Laney> https://errors.ubuntu.com/?user=laney&period=day that is

[12:23:46] <ev> Laney: ah, I can reproduce that. Looks like we're taking too long to iterate through all your subscriptions: Failed to load resource: the server responded with a status of 504 (Gateway Time-out)
[12:24:02] <tumbleweed> Laney: your auto-subscribe script?
[12:24:05] <Laney> I'm probably subscribed to quite a lot of stuff
[12:24:16] <Laney> tumbleweed: that shouldn't be too many, but haskell-* likely is

https://bugs.launchpad.net/~laney/+packagebugs

[Evan (ev)]

Brian, I'm subscribing this to you for the moment. Any speed improvements we can make to this query?

We can probably also bring the time down by moving to the 24h for the day period column family set up:
https://bugs.launchpad.net/errors/+bug/1046269
https://bugs.launchpad.net/errors/+bug/1008730

[Brian Murray (brian-murray)]

For reference Laney is subscribed to a total of 403 packages.

In [4]: len(laney.getBugSubscriberPackages())
Out[4]: 403

[Brian Murray (brian-murray)]

Getting the source packages that a user is subscribed to does not take too long, it is looking up all the binaries for the source package that takes the most amount of time.

[Brian Murray (brian-murray)]

We work around this for teams that are subscribed to lots of packages, like foundations-bugs and ubuntu-x-swat, by caching their subscriptions in the UserBinaryPackages column family. We could do the same thing for Laney and other users we encounter with lots of subscriptions.

Alternatively, one way to speed this up is to switch from using the getPublishedBinaries Launchpad API request for a source package and just use binaryFileUrls and then do something like this:

bf.split('/')[-1].split('_')[0] for bf in json_request(bfu_url)

while this works it seems kind of nasty.

We could also create a column family with a cache of source packages to binary packages.

[Brian Murray (brian-murray)]

Further testing seems to indicate that using binaryFileUrls isn't any faster although the Launchpad cache may be changing the test results.

[Brian Murray (brian-murray)]

I think the best thing to do here is to add laney to list of cached users.

[Iain Lane (laney)]

Cool.

Maybe you could do this a bit more dynamically by noticing the failure, checking the number of subscribed packages is over some threshold and caching the user. Or just cache for all users who have access and are subscribed to >threshold packages, if that's not too expensive.

Anyway, I just tried the URL in the description and got a list of problems. Kudos.

[Brian Murray (brian-murray)]

That is interesting because the version of daisy with this change has not been rolled out yet.

[Iain Lane (laney)]

On Wed, Jun 26, 2013 at 03:32:04PM -0000, Brian Murray wrote:
> That is interesting because the version of daisy with this change has
> not been rolled out yet.

That /is/ interesting - https://errors.ubuntu.com/?user=laney&period=day
shows me some crashes. Does it for you?

Although they aren't for a very diverse range of packages, so I'm not
sure it's complete.

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]