Comment 5 for bug 357907

We've done a lot of work on investigating this, and have at least found the cause.

The problem is that, when a bug is filed with a BLOB attachment, Launchpad automatically tries to parse the attachment line by line. This is done in a big while loop, and the OOPSes are occuring because the BLOB files from certain projects are large enough (~100MB in some cases) for the request timeout to kick in shortly after this while loop has completed but before the request has completely been dealt with by LP. This threw up a couple of red herrings when we were looking for the problem.

We have a few possible solutions to the problem:

 1. Create a script that processes apport data and make it possible for the +filebug process to tell it "Hey, this LibraryFileAlias is mine, please process it and update this bug appropriately" after the bug has been filed.
 2. Make it so that the apport data get processed before the user is pointed at +filebug, so that the requisite data are available to +filebug as via a series of queries instead of locked away in a BLOB.
 3. A variation on option 1, whereby +filebug will only use the asynchronous method for files over a certain size, e.g. 25MB or so).
 4. Stop parsing once we have the salient data (i.e. bug summary, subscribers, tags, etc.) and finish parsing later.
 5. When the user hits +filebug, a page showing a spinner and telling the user "Processing apport data...". This would fire a job to process the data, which we'd poll periodically. For the non-AJAX use case we'd have to do this using refreshes.

Currently I'm favouring option 5, but we haven't made a decision yet.

Whatever we choose, we currently estimate that the work would take up to a week and a half to complete, so we're hoping to figure out the details of the best solution and start working on it in the first week of the 3.1.11 cycle (wb 9th November 2009).