Oops pruner script should compress oops files to save disk space

Bug #130901 reported by Diogo Matsubara
4
Affects Status Importance Assigned to Milestone
Launchpad itself
Won't Fix
Medium
Unassigned

Bug Description

The OOPS pruner script deletes non-referenced OOPSes after a certain amount of time (45 days).
We want the pruner to compress oops files to save disk space both on production servers and on devpad.

Revision history for this message
Diogo Matsubara (matsubara) wrote :

I had a pre-implementation chat with Stuart about this bug. I'm mutating this bug into the correct thing to do.

A summary of our chat:

Currently the oops pruner does its job using only the oops filename. The pruner would need to open each oops file and parse its contents to find out the OOPS type.

An easier solution is to extend the oops pruner to compress the oops files instead.

Care must be taken for mutating the file system while iterating over it. It's better to store a list of files to compress and compress them after all the iteration has been completed.

description: updated
Changed in launchpad:
importance: Undecided → High
status: New → In Progress
Changed in launchpad:
assignee: nobody → matsubara
Revision history for this message
Diogo Matsubara (matsubara) wrote :

A fix is ready and reviewed. Need to check with Tom if he'll accept the fix or mark this bug as "Won't fix".
Tom raised the issue that the fix might pose a performance problem. Meanwhile a new harddrive was added to devpad, minimizing disk space the problem.

Revision history for this message
Diogo Matsubara (matsubara) wrote :

marking as won't fix because the new hard drive worked better than compress oops to save disk space

Changed in launchpad:
status: In Progress → Won't Fix
Revision history for this message
Stuart Bishop (stub) wrote :

We still want this, it just isn't that important.

Changed in launchpad:
assignee: matsubara → nobody
importance: High → Medium
status: Won't Fix → Confirmed
Revision history for this message
Diogo Matsubara (matsubara) wrote : Re: [Bug 130901] Re: Oops pruner script should compress oops files to save disk space

On Thu, Sep 20, 2007 at 02:53:06AM -0000, Stuart Bishop wrote:
> We still want this, it just isn't that important.

The branch:

 bzr+ssh://devpad.canonical.com/code/matsubara/launchpad/bug-130901

has a fix for it, but TomH was concerned that this might cause a performance
problem in production servers.

--
Diogo M. Matsubara

Revision history for this message
Stuart Bishop (stub) wrote : Re: [Bug 130901] Re: Oops pruner script should compress oops files to save disk space

Diogo Matsubara wrote:
> On Thu, Sep 20, 2007 at 02:53:06AM -0000, Stuart Bishop wrote:
>> We still want this, it just isn't that important.
>
> The branch:
>
> bzr+ssh://devpad.canonical.com/code/matsubara/launchpad/bug-130901
>
> has a fix for it, but TomH was concerned that this might cause a performance
> problem in production servers.

Given that this is a batch process that can easily be run with 'nice' if CPU
and disk access on the app servers really is an issue (doubtful) I think we
should consider landing this.

It will probably *save* CPU cycles and disk seeks, as the compression only
happens once but the rsyncs of the tree that occur every 10 minutes will
only have to deal with much smaller files.

--
Stuart Bishop <email address hidden> http://www.canonical.com/
Canonical Ltd. http://www.ubuntu.com/

Revision history for this message
Tom Haddon (mthaddon) wrote :

Spoke with James Troup and he agrees that there are performance concerns here. Using "nice" will help with CPU usage, but not with disk IO. Disk space on the individual servers and on devpad is no longer a premium as we've provisioned extra disk space on devpad and are monitoring it in any case. And rather than doing unnecessary work to compress and then uncompress again to read, we'd rather just not compress.

Stuart Bishop (stub)
Changed in launchpad:
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.