bug search results are not exportable without custom software

[Jonathan Knowles]

Symptoms
========

When users do a bug search, the search results can be browsed in-line in batches, but cannot be exported easily [there is a hidden way to export just the bug ids, but this isn't useful as an 'export'].

Some users like to take things offline (e.g. for reports), and being able to do that without custom software (e.g. Python and Launchpadlib) would be very convenient for them. Canonical OEM has customers/partners that run Windows and IE and have no ability to install custom software - so for them they simply cannot export the results of a bug search today. These users could use a 'standard' interchange format like CSV (comma separated values) quite easily.

Analysis
========

Bug searches can be arbitrarily large (e.g. 500K bugs in one search in Ubuntu). A simple view that renders such a large batch will timeout (it can be reasonably expected to take more than 20 seconds with our current stack). Other than that doing an export of bug search results in a standard flattened interchange format makes a lot of sense.

Some ways to mitigate against the timeout aspect are:
 - only export some N (e.g. 1000) bugs
 - ask the user how many bugs they want exported and refuse values larger than some N (e.g. 1000)
 - use a backend task to do the export and store the result in the librarian
 - only export one page at a time and force users to paginate and export each page

The export probably wants to export the same data we show in the matching search, giving a reasonably powerful ad-hoc reporting system.

Recommendations
===============

 - do some fact checking with Canonical OEM (who are requesting this) to see whether an arbitrary limit would impact them
 - if doing a backend export be sure to link it all in with longpoll so the user doesn't have to browse away from the results
 - a backend export is probably the most sensible thing to do, because it will allow very large exports without rework in the future

[Jonathan Knowles (jsk)]

Intellectronica pointed out that a similar solution exists for sprints. For example:

https://edge.launchpad.net/sprints/ubuntu-it-meeting/+attendees-csv

[Daniel Holbach (dholbach)]

Would Status and Importance be per task? Are all tasks listed? That's particularly important for teams' bug lists.

[Jonathan Knowles (jsk)]

Daniel - that's correct. Status and Importance would be listed per task (as they are in graphical views today).

Interestingly, today there's no way to the +bugs-text view appear for a person or a team, but this is a known issue (see bug 157410).

Basically, our intent is to make the CSV export behave in a similar way to the graphical listings. So there would be one bug(task) per row, and multiple metadata per bug(task). So for team bug listings, you'd get multiple entries if there were multiple tasks assigned to the same team, attached to the same bug.

[Markus Korn (thekorn)]

In a per task view it would be nice to also have an url-column

[Christian Reis (kiko)]

On Fri, Dec 07, 2007 at 05:56:31AM -0000, Markus Korn wrote:
> In a per task view it would be nice to also have an url-column

Can you clarify exactly what you mean here? And is it related to the CSV
view this bug proposes?

[Daniel Holbach (dholbach)]

I guess Markus meant an upstream bug URL?

[Markus Korn (thekorn)]

Well I thought about some use cases for such an CSV view. Let's say I want to import a CSV file into an database, In this case it would be nice to have an unique key for each element in a data-row. As (unfortunately) a task can not be addressed (and identified) by an ID directly the url of a task is the only unique reference to a bugreport-task combination.

So an example implementation should look like:
Bug-ID, task-url, package/project, summary, importance, status
1, "https://bugs.launchpad.net/ubuntu/+bug/1", "ubuntu", "Microsoft has a majority market share", "Critical", "Confirmed"
1, "https://bugs.launchpad.net/openoffice/+bug/11", "openoffice", "Microsoft has a majority market share", "Undecided", "Confirmed"

I also think it might be useful to add as much data-columns as possible, like
* has_patch
* assignee
* mentor_available
* security_related
* private
* milestone

This way the CSV view can be a useful extension of LP

Markus

[Eleanor Berger (intellectronica)]

With the Launchpad web service API providing most (if not all) of this data using a richer interface, this isn't really needed anymore.

[Christian Reis (kiko)]

You guys are getting the use cases wrong. To think that an API is a substitute for a CSV export really suggests you haven't thought about who is using the bugtracker. Few of our users are actually developers comfortable enough to write software to poll an API. And you know what they would do if they could? Generate a CSV file.

We definitely should do it for them.

[Eleanor Berger (intellectronica)]

> You guys are getting the use cases wrong. To think that an API is a substitute for a
> CSV export really suggests you haven't thought about who is using the bugtracker.
> Few of our users are actually developers comfortable enough to write software to poll
> an API. And you know what they would do if they could? Generate a CSV file.

> We definitely should do it for them.

The reason having an API makes this bug not important is not that we expect all users to be hackers and just use the API directly, but that we assume, now that the API is available, that some features, the ones that are not universally important, to be implemented externally, rather than by adding more features to Launchpad itself.

We don't actually know of any one user who currently hopes to use such a feature. That doesn't necessarily means that we shouldn't build it and see if they will come, but it makes designing the feature more difficult, at the very least.

It's not hard to implement this, but I would prefer to talk to a few potential consumers of these exports before we bump the importance up and schedule it.

[Christian Reis (kiko)]

On Sat, Jan 24, 2009 at 02:56:51PM -0000, Tom Berger wrote:
> We don't actually know of any one user who currently hopes to use such a
> feature.

Talk to OEM services.

[Daniel Holbach (dholbach)]

Maybe it's not that important any more, via API we can even get bug lists anonymously easily.

[Gavin Panella (allenap)]

I'll tentatively mark this as Won't Fix then.

[Gavin Panella (allenap)]

That didn't last long. OEM Services do still want this feature.

[Graham Binns (gmb)]

I'm still unconvinced that this couldn't be solved with a script that uses the LP API rather than devoting an LP developer resource to including it in Launchpad proper, which is likely to be quite a while off given the amount of work we've got to do already.

I'd be happy to help anyone who wants to try writing an API script for this.

[Jonathan Lange (jml)]

On Mon, Jul 5, 2010 at 4:00 PM, Graham Binns <email address hidden> wrote:
> I'm still unconvinced that this couldn't be solved with a script that
> uses the LP API rather than devoting an LP developer resource to
> including it in Launchpad proper, which is likely to be quite a while
> off given the amount of work we've got to do already.
>

FWIW, the use case is a bunch of really busy people scurrying to get
reports in to their customers as quickly as possible so they can get
paid. It's the kind of situation where switching from the browser to a
terminal is an unwelcome break.

[Emilien Klein (emilien-klein)]

I'd add a new use case: offline bug processing.

For example today I will be for 2 hours in the train, and I'd like to work on open bugs. In the current situation I need to copy/paste from the website, or make screenshots. And luckily my project has few enough bugs (20) that it all fits on one page, if pagination was involved (like for the Firefox bugs example in the OP) this task would be impossible to perform.

Is this feature such a complex one? Isn't it just duplicating the algorithm used to produce the "+bugs-text" flavor, and add comma's and other info from the database?

I'd be happy to have a look at the code if someone could point me to where the code is generating "+bugs-text".

If we absolutely want to go the API route, I'd have to create an external site that queries LP and provides a "download as .csv" option. Seems a better approach to be able to merge that code back into LP instead of having to build yet another external service from scratch.