buildd-manager should be more rsync-friendly with log syncing

[Tom Haddon]

The buildd-manager use twisted's default logging, which rotates every 977K and uses a numbered extentsion. This means all files change contents on every sync, which is very rsync unfriendly. Ideally we'd be able to use some external method of rotating logs so that we could use daily rotation with dateext.

[Julian Edwards (julian-edwards)]

This is how the Landscape team does it:

https://pastebin.canonical.com/33150/

[Robert Collins (lifeless)]

This is a subtle buh - and that fragment is interesting - its basically not safe (no callFromThread) and otherwise the same as the built in twistd setup. (Thanks spiv for analysing this)

twistd -l foo.log
will behave identically except that it will use SIGUSR1 not SIGUSR2 to rotate.

Both twistd and the fragment you pasted use LogFile.fromFullPath, so you get the default rotation behaviour.

The constructor parameter rotateLength when set to None will disable rotation.

See pydoc twisted.python.logfile.LogFile for details.

I think that a small patch to twistd would let us say --rotate=None and disable rotation altogether, which is what we want - that lets us manually rotate at our schedule rather than being forced to stay under 1 MB.

I'm going to request that in the upstream now.

[Robert Collins (lifeless)]

Since I can't find a dialog, I'll just link - http://twistedmatrix.com/trac/ticket/4557 - filed about this.

[Henning Eggers (henninge)]

So, I have a branch here that disables twistd's own logging (every 1000000 bytes) and makes it accept SIGUSR1 to re-open its log file. That should satisfy the needs for proper logging, shouldn't it?

[Henning Eggers (henninge)]

Oh, my patch for twisted was rejected, btw.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r11244 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/11244>.