Need way to list and observe branch tips

[Gustavo Niemeyer]

For the Ensemble work we've been debating, we need a way to list (location, tip revision id, (official_series, official_series)) for all branches of a distribution (principia, in our case).

Since this list will hopefully get huge over time, we'll also need a way to ask for the changes since a given date, in which case we'd only get results for those branches that are either new or have had changes to their tip revision id since the provided timestamp.

Please note that we don't need/want *all the data* for a branch in that list, since it's going to be quite a lot of unnecessary information being transferred over the wire.

For reference only, in a conversation with Jono over UDS, he's come up with this sketched query which would do something resembling what we'd need:

SELECT unique_name, last_scanned_id FROM Branch JOIN DistroSeries ON
Branch.distroseries = DistroSeries.id JOIN Distribution ON
DistroSeries.distribution = Distribution.id WHERE Distribution.name =
'ubuntu' AND branch.last_scanned > '2011-05-01';

[Robert Collins (lifeless)]

all branches, or all the branches aliased (the 'official branches')?

[Gustavo Niemeyer (niemeyer)]

All branches. People can publish any branches they want as formulas. The official ones are just a selection of those.

[Francis J. Lacoste (flacoste)]

Although adding a method just for this use case would work. I'd rather we export a getBranches() method which returns the full branch information and which takes parameters to query the results. That would useful in more use cases and should still work for you (at the expense of returning a little more information than you stricly require).

[Gustavo Niemeyer (niemeyer)]

Note that it's not just a bit more, but _a lot_ more information. All we need is really just a couple of strings (location, revid). Compare that with the 40+ (!) fields of a branch, including the _whiteboard_. Then, imagine iterating over a potentially long list of these. I'm generally against early optimization, but this is a lot of overhead.

[Robert Collins (lifeless)]

On Sat, Jun 11, 2011 at 12:56 PM, Gustavo Niemeyer <email address hidden> wrote:
> Note that it's not just a bit more, but _a lot_ more information.  All
> we need is really just a couple of strings (location, revid).  Compare
> that with the 40+ (!) fields of a branch, including the _whiteboard_.
> Then, imagine iterating over a potentially long list of these.  I'm
> generally against early optimization, but this is a lot of overhead.

So the current Launchpad API is very much about exporting /objects/.
This is why its so flexible, and why its so slow. AIUI its baked into
lazr.restful at a fairly deep level.

We're expecting that you will want to examine between 1K and 20K
objects; this is probably outside what we can reliably do in 1 second
even for the minimal data - so will need batching etc.

However, we can do a view returning json of the results of your query,
with batching, pretty easily, even if the API framework would be in
our way.

So I'm in favour of us having an *unstable* (that is, devel only, no
long term support guarantee) API that precisely fits your needs.
Should we need to change it in future we'd coordinate that with you :
the lack of long term support guarantee says that we're not supporting
this for shipping in API clients on the next Ubuntu LTS (if you need
that, you should say so now). Generally for webapps and
backend-in-the-datacentre unstable APIs are fine - we don't change
them willy-nilly.

-Rob

[Gustavo Niemeyer (niemeyer)]

Ensemble will ship in the next LTS, but this is not going to be _in_ the LTS. As you say, it's something for the data center, so we should be fine.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r13439 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13439>.

[William Grant (wgrant)]

Works fine with the date restriction. And, impressively, the unrestricted variant renders in under 5s most of the time on qastaging, into a glorious 30MB JSON document containing every Ubuntu branch.