Oops pruner script should compress oops files to save disk space

[Diogo Matsubara]

The OOPS pruner script deletes non-referenced OOPSes after a certain amount of time (45 days).
We want the pruner to compress oops files to save disk space both on production servers and on devpad.

[Diogo Matsubara (matsubara)]

I had a pre-implementation chat with Stuart about this bug. I'm mutating this bug into the correct thing to do.

A summary of our chat:

Currently the oops pruner does its job using only the oops filename. The pruner would need to open each oops file and parse its contents to find out the OOPS type.

An easier solution is to extend the oops pruner to compress the oops files instead.

Care must be taken for mutating the file system while iterating over it. It's better to store a list of files to compress and compress them after all the iteration has been completed.

[Diogo Matsubara (matsubara)]

A fix is ready and reviewed. Need to check with Tom if he'll accept the fix or mark this bug as "Won't fix".
Tom raised the issue that the fix might pose a performance problem. Meanwhile a new harddrive was added to devpad, minimizing disk space the problem.

[Diogo Matsubara (matsubara)]

marking as won't fix because the new hard drive worked better than compress oops to save disk space

[Stuart Bishop (stub)]

We still want this, it just isn't that important.

[Diogo Matsubara (matsubara)]

On Thu, Sep 20, 2007 at 02:53:06AM -0000, Stuart Bishop wrote:
> We still want this, it just isn't that important.

The branch:

 bzr+ssh://devpad.canonical.com/code/matsubara/launchpad/bug-130901

has a fix for it, but TomH was concerned that this might cause a performance
problem in production servers.

--
Diogo M. Matsubara

[Stuart Bishop (stub)]

Diogo Matsubara wrote:
> On Thu, Sep 20, 2007 at 02:53:06AM -0000, Stuart Bishop wrote:
>> We still want this, it just isn't that important.
>
> The branch:
>
> bzr+ssh://devpad.canonical.com/code/matsubara/launchpad/bug-130901
>
> has a fix for it, but TomH was concerned that this might cause a performance
> problem in production servers.

Given that this is a batch process that can easily be run with 'nice' if CPU
and disk access on the app servers really is an issue (doubtful) I think we
should consider landing this.

It will probably *save* CPU cycles and disk seeks, as the compression only
happens once but the rsyncs of the tree that occur every 10 minutes will
only have to deal with much smaller files.

--
Stuart Bishop <email address hidden> http://www.canonical.com/
Canonical Ltd. http://www.ubuntu.com/

[Tom Haddon (mthaddon)]

Spoke with James Troup and he agrees that there are performance concerns here. Using "nice" will help with CPU usage, but not with disk IO. Disk space on the individual servers and on devpad is no longer a premium as we've provisioned extra disk space on devpad and are monitoring it in any case. And rather than doing unnecessary work to compress and then uncompress again to read, we'd rather just not compress.