Add a disk resource usage report for distributions

[Francis J. Lacoste]

We need a way of knowing the disk resources used by a distribution (ideally split by series). This is especially important for derived distributions so that we know the amount of resources used.

[Julian Edwards (julian-edwards)]

Are you talking about an internal lpstats graph, or something IN Launchpad?

[Robert Collins (lifeless)]

Either would do the job; in-LP would be easier to maintain I suspect.

[Julian Edwards (julian-edwards)]

There's an asynchronously-loaded JS portlet on the PPA index page that gives repo size. The same page template is used for Copy Archives but the portlet had to be disabled because it consistently times out on distro-sized archives.

Therefore, this will need some very careful optimisation to work.

[Robert Collins (lifeless)]

On Tue, Aug 9, 2011 at 11:09 PM, Julian Edwards
<email address hidden> wrote:
> There's an asynchronously-loaded JS portlet on the PPA index page that
> gives repo size.  The same page template is used for Copy Archives but
> the portlet had to be disabled because it consistently times out on
> distro-sized archives.
>
> Therefore, this will need some very careful optimisation to work.

It times out on PPAs too ;) bug 739070

I suspect we want to cache the quota usage rather than generating from
first principles each time.

[Gavin Panella (allenap)]

I extracted a script (attached) to experiment with. It contains a
function that's based on ArchiveView.repository_usage, but with some
changes around the quota calculations (authorized_size can be None,
which cannot be used for arithmetic in this version of reality).

The LOSAs sprinting this week so Julian ran this on dogfood, but the
performance was *terrible*. Queries were taking multiple minutes to
complete. It looked almost as though they were just hanging, but
Julian reckoned it was thrashing the disk.

This needs further investigation, and probably with a lot of help from
the LOSAs, so I'm putting this back on the pile until next week when
the LOSAs are fully available once again.

[Robert Collins (lifeless)]

Yes, it would be terrible - see the timeout bug I linked too that is
doing precisely this on *one* archive, let alone all.

Further, that script uses the oop layer, so you are doing thousands of
complete table scans on LFA, which will be exacerbating things
something chronic :)

Note that the report can sensibly be in LP itself - it doesn't need to
be a chart showing all distros - just showing the portlet for each
distro would meet our initial needs.