App Servers: Remove need for restart on logrotation

[Steve McInerney]

Currently to rotate logs on the various app servers, the individual app server instances are restarted post rotate.
Necessitating a small outage.

It would be preferred: "to mv the log, send a signal telling zope to open fresh ones", quoting lifeless.

[Diogo Matsubara (matsubara)]

Francis, is this doable? If yes, is it something that your team would do right? If not, please re-assign to the appropriate team.

Thanks

[Diogo Matsubara (matsubara)]

As per bug 365509 the ability to reload the app server would also be useful to re-read a changed config file.

[Stuart Bishop (stub)]

On Wed, Apr 29, 2009 at 9:53 AM, Diogo Matsubara
<email address hidden> wrote:
> As per bug 365509 the ability to reload the app server would also be
> useful to re-read a changed config file.

Alternatively, make the appservers rotate their own logs. I think Z3
and/or Python already has the feature. We just need to enable it.

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

[Diogo Matsubara (matsubara)]

On Wed, Apr 29, 2009 at 12:48 AM, Stuart Bishop
<email address hidden> wrote:
> On Wed, Apr 29, 2009 at 9:53 AM, Diogo Matsubara
> <email address hidden> wrote:
>> As per bug 365509 the ability to reload the app server would also be
>> useful to re-read a changed config file.
>
> Alternatively, make the appservers rotate their own logs. I think Z3
> and/or Python already has the feature. We just need to enable it.

Could this feature be used to reload the config too?

--
Diogo M. Matsubara

[Tom Haddon (mthaddon)]

Improving the importance on this bug per RT#34781 - have been bitten by this a few times and it ends up being user affecting if the restart fails. The length of time that it's user affecting depends on the availability of sysadmins.

[Tom Haddon (mthaddon)]

This also takes staging down probably about 30% of the time the logrotate runs, as the staging app server is doing so much more (app server, librarian, xmlrpc) that it seems to fail to stop and start a lot more frequently than on production.

[Tom Haddon (mthaddon)]

Possibly related is bug 396209

[Stuart Bishop (stub)]

There is already builtin support for logfile rotation. You just need to add options to the logfile stanzas in launchpad.conf:

<eventlog>
    <logfile>
        path launchpad.log
        old-files 9
        max-size 10000000
    </logfile>
</eventlog>

<accesslog>
    <logfile>
        path launchpad-access.log
        old-files 9
        max-size 10000000
    </logfile>
</accesslog>

Is this good enough for our needs? If so, please update the production configs and mark this bug fixed.

[Stuart Bishop (stub)]

Needs feedback.

[Tom Haddon (mthaddon)]

I think this should be okay - we'll give it a try and let you know if there are any problems (and update the bug either way).

[Tom Haddon (mthaddon)]

We'd like to test this on staging first, but I can't see how to configure this for the staging librarian logfile - this seems to be specified in the librarian_server stanza of staging-lazr.conf rather than in the format as above.

[Stuart Bishop (stub)]

This bug only mentions the appservers, which are Zope3 based applications. Librarian issue should be a separate bug - it is twisted based and will need someone familiar with the Twisted framework to look at it. I'm not opening that bug, as Librarian log rotation already seems to be happening on production (although we don't seem to be pruning old logs - there are nearly 5000 of them).

[Tom Haddon (mthaddon)]

Ok, well I guess we'll try this on staging and see how thing go. It may be okay that the librarian logs don't get rotated, or we may need to figure out some other way of doing that, or they may get rotated themselves. We shall see...

[Gary Poster (gary)]

Hi Tom. Did you perform this experiment? Stuart is right that there is built in support--maybe even some other variations, if I remember correctly. If the built-in support doesn't work now, we should maybe pursue fixing this as the solution: we should not have to restart apps for log rotation.

[Gary Poster (gary)]

Stuart, could you handle this, including shepherding this through the losas?

I'm pretty sure Zope supports precisely what lifeless says in the original bug description. I'd like that to be what we aim for. I'm happy to help dig in if needed.

Thank you

[Tom Haddon (mthaddon)]

I can confirm this is working on staging. However, the logrotation is really rsync-unfriendly - it appends a number to the end of the logs and every time it does this each archived log gets bumped up a number. It also doesn't compress at all.

Are there any options that would include compressing logs, and also not change the filenames on all archived logs each time they are rotated?

[Tom Haddon (mthaddon)]

Actually, I guess changing the filenames each time has the advantage of auto-pruning out the oldest logs, but the compression of rotated logs would be nice.

[Stuart Bishop (stub)]

I'm adding a SIGUSR2 signal handler to reopen the logfiles. We can then use logrotate to maintain the logfiles. This seems a better solution that extending the builtin log rotation.

[Diogo Matsubara (matsubara)]

Fixed in devel r9259.

[Tom Haddon (mthaddon)]

I've added a "reload" target to edge initscripts. Will put this in place for logrotate next week to test (waiting til losas will be around to deal with any fallout).