launchpad performance is unsuited for batch tasks

[Björn Michaelsen]

As discussed here:
 http://www.iloveubuntu.net/libreoffice-developers-are-working-automated-import-crash-testing-60-crashes-detected-while-testing
upstream LibreOffice is crawling bugzillas and launchpad for documents for regression testing.

Upstream reports it took 5 hours to download a mere 500 document even with 9 concurrent connections. Obviously that is abysmally poor compared to plain bugzillas. Anyway to improve this?

[William Grant (wgrant)]

We need specific descriptions of things that are slow, with as much information as possible. What are they trying to grab? How are they doing it? Which particular bit of the process is slow? 500 documents in 5 hours with 9 connections means a document per connection per 6 minutes, which makes no sense -- no Launchpad request takes more than 20s, and most operations are capped at <5s.

[Björn Michaelsen (bjoern-michaelsen)]

Here is the script:
https://gerrit.libreoffice.org/gitweb?p=core.git;a=blob;f=bin/get-bugzilla-attachments-by-mimetype;h=93928843c776156411e3f150e07a5a2d966c10c6;hb=3bcd2d3787c5c6598656b34c00f04612bc99d8d2

Checking back with the users it seems the main problem is that there is no ability to search for attachments by mimetype.

[Markus Mohrhard (moggi)]

Just as a side note I use a little updated version of the above script which only handles 2 file types per script and therefore only goes through all bugs twice per connection.

But this is still very slow compared to bugzilla where I use the same set up and we get 27000 documents from different bugzilla instances in 11h.

In both cases the script is not optimised and performs poorly but in lauchpad the missing feature to search for bugs by mimetype limits us to the searching bugs in the Libreoffice category and requires that we go through all bug reports.