"service stop; service start" fails with wsgi.py from freeipa-server

[Timo Aaltonen]

If apache is running, doing 'systemctl stop apache2.service; systemctl start apache2.service' fails with:

● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2)
   Active: failed (Result: exit-code) since to 2015-03-05 20:33:16 EET; 6s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 27891 ExecStop=/etc/init.d/apache2 stop (code=exited, status=0/SUCCESS)
  Process: 27943 ExecStart=/etc/init.d/apache2 start (code=exited, status=1/FAILURE)

maalis 05 20:32:56 ipa02.tyrell apache2[27943]: no listening sockets available, shutting down
maalis 05 20:32:56 ipa02.tyrell apache2[27943]: AH00015: Unable to open logs
maalis 05 20:32:56 ipa02.tyrell apache2[27943]: Action 'start' failed.
maalis 05 20:32:56 ipa02.tyrell apache2[27943]: The Apache error log may have more information.
maalis 05 20:33:16 ipa02.tyrell systemd[1]: apache2.service: control process exited, code=exited status=1
maalis 05 20:33:16 ipa02.tyrell systemd[1]: Failed to start LSB: Apache2 web server.
maalis 05 20:33:16 ipa02.tyrell systemd[1]: Unit apache2.service entered failed state.
maalis 05 20:33:16 ipa02.tyrell systemd[1]: apache2.service failed.
maalis 05 20:33:16 ipa02.tyrell apache2[27943]: *
maalis 05 20:33:16 ipa02.tyrell apache2[27943]: * The apache2 instance did not start within 20 seconds. Please read the log files to discover problems

error.log doesn't have more than this:

[Thu Mar 05 20:32:56.258403 2015] [mpm_event:notice] [pid 27786:tid 139760377153408] AH00491: caught SIGTERM, shutting down

so stopping the service returns too quickly and start gets run too early?

[Timo Aaltonen (tjaalton)]

this works with upstart and IIRC sysv (debian)

[Steve Langasek (vorlon)]

Your output shows that systemd is invoking the init script for apache2, because there is no native systemd unit in the package. So this is not a problem specific to systemd, the init script must be racy when running under sysvinit as well. (And even under upstart, since apache2 has no upstart job.)

[Timo Aaltonen (tjaalton)]

That might be, but the repro rate on systemd is 100%, while I haven't been able to reproduce it with upstart.

[Timo Aaltonen (tjaalton)]

Hm, this seems to be related to mod_nss being enabled with Freeipa. Disabling it fixes apache2 restart, though installing & enabling it doesn't fail restart with upstart..

[Martin Pitt (pitti)]

> maalis 05 20:32:56 ipa02.tyrell apache2[27943]: no listening sockets available, shutting down
> maalis 05 20:32:56 ipa02.tyrell apache2[27943]: AH00015: Unable to open logs

That's certainly a good hint where to start. What socket/log dir is apache trying to create?

[Timo Aaltonen (tjaalton)]

I'm getting a better error on Debian

maalis 06 09:43:20 ipa01 apache2[2223]: Starting web server: apache2(98)Address already in use: AH00072: make_sock: could not bind to address [::]:443
maalis 06 09:43:20 ipa01 apache2[2223]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:443
maalis 06 09:43:20 ipa01 apache2[2223]: no listening sockets available, shutting down
maalis 06 09:43:20 ipa01 apache2[2223]: AH00015: Unable to open logs

not sure if the logs part is significant here, maybe just fallout from failing to bind

[Timo Aaltonen (tjaalton)]

So it's freeipa-server apache config that causes it, not mod_nss. Turns out it's mod_wsgi, this is enough to reproduce the failure:

WSGISocketPrefix /run/apache2/wsgi
WSGIDaemonProcess ipa processes=6 threads=1 maximum-requests=500
WSGIProcessGroup ipa
WSGIApplicationGroup ipa
WSGIImportScript /usr/share/ipa/wsgi.py process-group=ipa application-group=ipa

put it in conf-enabled/mock-ipa.conf and grab wsgi.py from freeipa-server. Processes is normally =2, and that's enough to reproduce the bug when the full ipa server is installed, but for demonstration purposed this config will do (plus a2enmod wsgi of course).

[Timo Aaltonen (tjaalton)]

hrm, wsgi.py depends on python-freeipa, which itself has modules that import stuff from freeipa-server, so it's not that simple to reproduce

[Timo Aaltonen (tjaalton)]

that said, it fails with upstart too