Apache2 logrotate kills existing connections on server

[Joe Kislo]

Apache2 logrotate kills existing connections on server and momentarily rejects
requests

I havn't fully diagnosed this one, but I had several (5 parallel jobs) running
firing xmlrpc requests at an apache2 server overnight. All 5 jobs failed with
an invalid server response at 6:25am. Either one of two things (or both)
happened: they were in the middle of an API request that was terminated abruptly
prior to a proper response, or they made a request while the server was not
started. I'm not exactly sure which, but based on my apache knowledge, I would
assume both would occour in the ubuntu configuration, as described below.

Looking in /etc/logrotate.d/apache2

it looks like it does an /etc/init.d/apache2 restart

looking in /etc/init.d/apache2, it looks like that turns into an apache -k stop

According to this documentation:
http://httpd.apache.org/docs/2.0/stopping.html

stop will IMMEDIATELY kill off all of the children. Which is obviously bad bad bad.

I would assume the line in the logrotate.d/apache2 file should read
/etc/init.d/apache reload

Which should translate to a graceful restart of apache2, thusly not killing
existing connections, and not momentarily rejecting incoming reuqests.

[Adam Conrad (adconrad)]

/etc/logrotate.d/apache2 is a conffile, meaning your changes won't get
overwritten by us on upgrade, so feel free to change your local version. The
default is to do a hard restart because we've seen many other instances where
graceful would cause the server to die completely when some rogue extension or
other segfaults as the parent reloads, leaving you with NO apache from 6:25
onward. I decided at that time that the current compromise was the lesser of
two evils.

I'm still trying to figure out the best way to serve everyone well here without
the world blowing up. ;)

[Joe Kislo (joe-k12s)]

Hmmm... Well, it's good to hear that it's a config file.. but I think doing a
hard restart is really bad. I do understand your concern that the server going
down at 6:30 would be really bad though. I think with an apachectl restart.. if
they broke their config file, the server would go down aswell. Verses graceful
if there's a broken config file, it WON'T graceful at all (EG: staying up under
the working config)

Maybe it's the "other guys problem" in me that says it should be a graceful
restart, but I have to imagine the majority of users don't have busted/broken
apache plugins in their production environments. So I would assume this apache
crashing problem affects the minority of users. The hard restart at 6:30am,
affects ALL ubuntu users. Since apache is one of the premier services that will
be run on ubuntu servers, I can't imagine it would be acceptable to need to go
down to rotate the logs.

I realize I can change this by modifying that file, and I certainly have... but
there's NO WAY anybody else is going to ever notice this until it's too late.
The only reason why I noticed this is because an API call (Business logic call
over SOAP) was running at 6:30am, and got terminated mid-communiucation. This
left the database of that site in a damaged condition. (These servers are not
in production for us yet.. but it would have broken any of our production
systems if we had ubuntu in production).

One of two things should be done:
* Ask the user when installing/updating the apache, if they want everybody to be
booted at 6:30am, or possibly risk apache crashing if your apache is unstable/broken
* Write a script that checks for apache crashing, and notify the administrator
that their system is broken

The script would work something like this:

if (Check the pid file exists) {
 if (Check the apache process running)) {
  apachectl graceful
  sleep 10
  if (!(Check the pid file exists) || !(Check the apache process is running)) {
   apachectl start
   Email administrator saying apache crashed during log rotate, and the issue
needs to be delt with immediately (EG: SSL services restarted... etc.)
  }
 }
}

I feel like the latter solution is the best... since it should take care of your
concern that apache will be down going forward after the log rotate.

[Abel Cheung (abelcheung)]

Very interesting. With bad extension or bad config file, Apache would die on reload but still keep running on restart? That's new discovery.

[Mathias Gug (mathiaz)]

According to README.Debian :

Since a graceful reload (/etc/init.d/apache2 reload) does not work reliably in
all situations, we use restart during logrotate. Since 2.2.3-5, we use 2.2's new
graceful-stop feature, so that restart should work even if apache is slow to
stop.

2.2.3-5 is in gutsy.

[Aidan Furlan (aidan-epochlabs)]

I have SSL running on Apache, and logrotate causes Apache to die completely once a week because it needs the passphrase to the SSL key to restart. Naturally it took me a couple of weeks to notice the pattern and realise that logrotate was causing this. It seems to me that anyone running SSL with a secure key will have this problem.

I think the script that Joe Kislo wrote above is the best solution for future versions.

[Mathias Gug (mathiaz)]

Debian fixed this bug in apache2-2.2.4-1. Gutsy and Hardy have newer packages.