[gutsy] graceful-stop fails when apache listens on more than one socket

[Jürgen Kreileder]

Binary package hint: apache2

This fix from 2.2.6 should be backported to Gutsy. Currently graceful-stop (which is used by init.d script) doesn't work with apache on gutsy (2.2.4-3build1) whenever apache listens on more than socket. This affects all servers that have SSL enabled (apache listens on ports 80 and 443).

,--
| 2.2.6-2
| Published in hardy-release on 2007-10-23
|
| apache2 (2.2.6-2) unstable; urgency=low
|
| * Avoid calling apr_pollset_poll() and accept_func() when the listening
| sockets have already been closed on graceful stop or reload. This
| hopefully fixes processes not being killed (closes: #445263, #447164)
| and the "Bad file descriptor: apr_socket_accept: (client socket)"
| error message (closes: #400918, #443310)
`--

[Rocco (rocco)]

I totally agree with Juergen.

[Rocco (rocco)]

This is a serious problem and needs to be fixed asap.

[fw (f-wahl)]

Agree. Please fix asap. It's becoming a big problem for us.

[Rocco (rocco)]

Recompiling the Hardy version (2.2.6-3) for Gutsy and installing verifies that this problem is fixed in that version.

Please,

Action, Urgency, Excellence

[gripped (lee-r-harris)]

Just to add that I believe I am suffering from this bug but my apache is only listening on 80.

So It seems this bug can affect all apache users

none static mods are:
apache2 -t -D DUMP_MODULES
Loaded Modules:
 core_module (static)
 log_config_module (static)
 logio_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 actions_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 authn_file_module (shared)
 authz_default_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 cache_module (shared)
 cgi_module (shared)
 dir_module (shared)
 disk_cache_module (shared)
 env_module (shared)
 expires_module (shared)
 headers_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 php5_module (shared)
 rewrite_module (shared)
 ruby_module (shared)
 setenvif_module (shared)
 status_module (shared)
Syntax OK

[Erik Franzén (erik-franzens)]

Have also been hit by this problem in Gutsy. Would be nice with a fix!

[Erik Franzén (erik-franzens)]

A workaround for this bug is to modify /etc/init.d/apache2 with the following code.

Find function apache_stop() and add the following at the end:

echo "Cleaning up apache beacause apache bug #174805 @ bugs.launchpad.net."
ps -opid=,cmd= -Capache2 | grep "/usr/sbin/apache2" | awk '{print $1}' | xargs kill -9
echo "Done cleaning."

Maybee it is not a good solution, but it works

[Jürgen Kreileder (jk)]

The cleaner work-around is to just replace "$APACHE2CTL graceful-stop" with "$APACHE2CTL stop".

Seriously, I don't understand why this bug doesn't get fixed (the fix in hardy doesn't count). It must bite quite a few people running web servers on Ubuntu.

[Daniel Hahler (blueyed)]

Please follow https://wiki.ubuntu.com/StableReleaseUpdates to get this into Gutsy. Are there other branches affected, e.g. Dapper?

[.:. brainsik (brainsik)]

This shows Apache listening on 80 and 443 and that `/etc/init.d/apache2 stop` works just fine:

# netstat -tlnp | grep apache
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1632/apache2
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 1632/apache2
# /etc/init.d/apache2 stop
 * Stopping web server apache2
   ...done.
# netstat -tlnp | grep apache
# /etc/init.d/apache2 start
 * Starting web server apache2
   ...done.
# netstat -tlnp | grep apache
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2454/apache2
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2454/apache2

Here I call graceful-stop directly:

# netstat -tlnp | grep apache
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2454/apache2
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2454/apache2
# apache2ctl graceful-stop
# netstat -tlnp | grep apache
# apache2ctl start
# netstat -tlnp | grep apache
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2506/apache2
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2506/apache2

In both cases the server stops/starts just fine. This is true for the entire web cluster I manage. Something else is going on that's causing problems for other people and it would be nice to know what.

[.:. brainsik (brainsik)]

I did some more research and my guess is that what's happening is you may have long-lived connections to your apache processes that are preventing them from dying in a timely manner. If you are hosting large downloads I think this would happen. Haven't tested. For my sites, I prefer the graceful-stop approach as it allows the current requests to finish before shutting down.

[Jürgen Kreileder (jk)]

No, it's not caused by connections. Listeners are enough.
I can imagine that the problem might be MPM specific, I haven't test it though.

Anyway, there's a fix for the problem which really should have gone into a Gutsy update.

[Guillermo Pérez (bisho)]

As I reported in bug #220220, on my system the problem is worse:

When apache kills some of it's children (too many idle processes, MaxRequestsPerChild reached) it also gets stuck in "Gracefully finishing" state.

This leaves from time to time stuck child processes, that remain there till apache reaches maxclients and STOPS answering to connections! This is a severe bug, worse than just "avoid using reload" on init script. I have to monitor and completely restart apache every 5-7 days on a high-traffic server and this is causing me lot's of problems. Please try to release a fix soon. Thanks!

[Guillermo Pérez (bisho)]

Please, fix this bug soon.

Is not only a problem with reload, but also affects normal operation as apache kills its childs acording to MaxRequestsPerChild and orMaxSpareServers, making Ubuntu completely unusable as a production web server.

I this continue to be unsolved, I will be forced to move away from ubuntu, and I'm very confortable with this distribution till now.

There is already a patch for this in apache bug https://issues.apache.org/bugzilla/show_bug.cgi?id=42829. Why is not being applied to ubuntu's apache?

[Chuck Short (zulcss)]

Has anyone tried this on hardy?

Thanks
chuck

[Guillermo Pérez (bisho)]

On a devel machine (under light load compared to production servers) the reload does not give any problem with hardy.

[Chuck Short (zulcss)]

Fixed for hardy.

[Guillermo Pérez (bisho)]

This means that this fix will not be backported to gutsy?

It's a very big problem that makes apache unusable on production servers!

[Swift (swift-3d4x)]

I totally agree with Guillermo, this is a very important fix and needs to be backported to gutsy asap.
Because of this bug, we have currently severe problems with our servers.

[Miguel Balsevich (mbalsevich)]

I add my voice to the fix requests (or maybe there is one and I have not understood?)
This happens WITHOUT using the init.d scripts: All that is needed is some heavy load.

Currenly I have set:
MinSpareServers 150
MaxSpareServers 150
MaxClients 150
MaxRequestsPerChild 80000

To avoid apache from starting/stopping its child processes frequently.
Still, I need to restart apache every 48 hours.

[Guillermo Pérez (bisho)]

Same here... It's sad it is still not fixed on Gutsy. It gives an ugly impression about Ubuntu's quality. It means it could not be trusted as server distro.

[Guillermo Pérez (bisho)]

Please, reopen this bug. It's still affecting gutsy.

It shouldn't be hard to backport this change. Currently gutsy is unusable as a webserver in production evironments, and upgrade to hardy is usually a hard task it must be handle with a lot of care. This is a must fix!

[tinodj (gjorgjioski)]

how to fix it in gutsy?

[Guillermo Pérez (bisho)]

Sadly there is not fix :(. Avoid apache reload, use stop; sleep 5; start.