apache2 fails to wait on stop/restart

[David Ames]

Apache2 is failing to wait for all its threads to terminate when stopping. This leaves TCP ports still in use when apache2 tires to restart. This has been seen on Trusty and Xenial

This becomes a problems on restarts and stop/starts.

I have seen this running a simple loop with service apache2 restart. However this is inconsistent.

It happens more predictably when wsgi is involved even if stop and then start are used instead of restart:

 * Stopping web server apache2
 *
 * Starting web server apache2
(98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
(98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
AH00015: Unable to open logs
Action 'start' failed.
The Apache error log may have more information.

[Christian Ehrhardt  (paelzer)]

Hi David,
I tried reproducing this while triaging to confirm and set bug status.

In a trivial loop like:
  for i in {1..100}; do echo $i; service apache2 restart; done;
It almost always blocks it by
  Mar 31 11:09:34 xenial-tests systemd[1]: apache2.service: Start request repeated too quickly.
  Mar 31 11:09:34 xenial-tests systemd[1]: Failed to start LSB: Apache2 web server.

I also installed wsgi and some other mods to slow down start/stop, but that didn't help either.
(apt-get install libapache2-mod-wsgi libapache2-mod-nss libapache2-mod-scgi libapache2-mod-webauth libapache2-modsecurity)
Installing these slows it down, so that systemd no more complains about it restarting too quickly, but it doesn't trigger the issue.

Did you something more to better trigger the issue in your tests?

[Christian Ehrhardt  (paelzer)]

I wonder if this bug might be related.
Similar errors also using wsgi to make it more reproducable, ...

https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1428796

Did you set up something similar for your wsgi and if so would it qualify as a dup to the other bug or should we keep it separate?

[David Ames (thedac)]

Christian,

Sorry for taking so long to get back to you.

We see this in the OpenStack charming team with openstack-dashboard (horizon) and keystone (when run on apache2 >= liberty). We have had to implement hacky work-arounds because apache2 is not waiting until it releases the port(s) before exiting.

You can deploy either of these and then test my loop from above.
  juju deploy openstack-dashboard

The 1428796 is similar in that apache2 is not waiting till it releases ports, but we see this on trusty and xenial so it is not systemd specific.

[Christian Ehrhardt  (paelzer)]

Hi David,
I'm cleaning dormant bugs atm and it seems that bug fell everybodies todo list for quite a while.
It is long enough that I have to ask what the status today is about the openstack-dashboard and if you still have the hacky workarounds in place due to this?

In fact I found another old bug 1463635 which complains just vice versa "the server waits too long". Maybe you could leverage something from the configs discussed there or is this just a bug from the past and can be closed today?

[Bryce Harrington (bryce)]

Is anyone still reproducing this issue on bionic or newer?

[Launchpad Janitor (janitor)]

[Expired for apache2 (Ubuntu) because there has been no activity for 60 days.]