Steps to reproduce/verify on Xenial: === References: - https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html - https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html#delegation-to-daemon-process - https://github.com/GrahamDumpleton/mod_wsgi/commit/13169f2a0610d7451fae92a414e8e20b91e348c9#diff-3e8b16b2885169dcec2dac843521e12cR29 - https://www.howtoforge.com/tutorial/how-to-run-python-scripts-with-apache-and-mod_wsgi-on-ubuntu-18-04/ Create a container with apache2/mod-wsgi in daemon mode, w/ keep-alive timeout long enough, and wsgi hello-world: --- $ lxc launch ubuntu:xenial lp1863232x $ lxc exec lp1863232x -- su - ubuntu $ sudo apt update $ sudo apt install -y apache2 libapache2-mod-wsgi $ sudo sed -i '/^KeepAliveTimeout/ s/ .*/ 15/' /etc/apache2/apache2.conf $ cat <http-request GET /hello-world HTTP/1.1 Host: 127.0.0.1 Connection: keep-alive EOF One connection/One request: $ (cat http-request; sleep 1) | telnet 127.0.0.1 80 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. HTTP/1.1 200 OK Date: Mon, 17 Feb 2020 23:33:19 GMT Server: Apache/2.4.18 (Ubuntu) Content-Length: 13 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/plain Hello World! Connection closed by foreign host. One connection/Two requests (added timestamps to check timeout values) $ (cat http-request; sleep 1; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done 23:34:42 == Trying 127.0.0.1... 23:34:42 == Connected to 127.0.0.1. 23:34:42 == Escape character is '^]'. 23:34:42 == HTTP/1.1 200 OK 23:34:42 == Date: Mon, 17 Feb 2020 23:34:42 GMT 23:34:42 == Server: Apache/2.4.18 (Ubuntu) 23:34:42 == Content-Length: 13 23:34:42 == Keep-Alive: timeout=5, max=100 23:34:42 == Connection: Keep-Alive 23:34:42 == Content-Type: text/plain 23:34:42 == 23:34:42 == Hello World! 23:34:43 == HTTP/1.1 200 OK 23:34:43 == Date: Mon, 17 Feb 2020 23:34:43 GMT 23:34:43 == Server: Apache/2.4.18 (Ubuntu) 23:34:43 == Content-Length: 13 23:34:43 == Keep-Alive: timeout=5, max=99 23:34:43 == Connection: Keep-Alive 23:34:43 == Content-Type: text/plain 23:34:43 == 23:34:43 == Hello World! 23:34:58 == Connection closed by foreign host. Reproduce the problem by placing 'sudo systemctl reload apache2' between the two HTTP requests (second request hits Error 503, depending on apache2 MPM module: mpm_event/mpm_worker/mpm_prefork) --- $ lsb_release -cs xenial $ dpkg -s libapache2-mod-wsgi | grep ^Version Version: 4.3.0-1.1build1 For reference on socket filename: $ sudo systemctl restart apache2 $ ls -1 /var/run/apache2/wsgi.*.sock /var/run/apache2/wsgi.6244.0.1.sock $ (cat http-request; sleep 1; sudo systemctl reload apache2; sleep 5; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done 23:48:29 == Trying 127.0.0.1... 23:48:29 == Connected to 127.0.0.1. 23:48:29 == Escape character is '^]'. 23:48:29 == HTTP/1.1 200 OK 23:48:29 == Date: Mon, 17 Feb 2020 23:48:29 GMT 23:48:29 == Server: Apache/2.4.18 (Ubuntu) 23:48:29 == Content-Length: 13 23:48:29 == Keep-Alive: timeout=15, max=100 23:48:29 == Connection: Keep-Alive 23:48:29 == Content-Type: text/plain 23:48:29 == 23:48:29 == Hello World! 23:48:35 == HTTP/1.1 503 Service Unavailable 23:48:35 == Date: Mon, 17 Feb 2020 23:48:35 GMT 23:48:35 == Server: Apache/2.4.18 (Ubuntu) 23:48:35 == Content-Length: 374 23:48:35 == Connection: close 23:48:35 == Content-Type: text/html; charset=iso-8859-1 23:48:35 == 23:48:35 == 23:48:35 == 23:48:35 == 503 Service Unavailable 23:48:35 == 23:48:35 ==

Service Unavailable

23:48:35 ==

The server is temporarily unable to service your 23:48:35 == request due to maintenance downtime or capacity 23:48:35 == problems. Please try again later.

23:48:35 ==
23:48:35 ==
Apache/2.4.18 (Ubuntu) Server at 127.0.0.1 Port 80
23:48:35 == 23:48:35 == Connection closed by foreign host. Notice the error log mention socket file name not found: $ sudo tail /var/log/apache2/error.log ... [Mon Feb 17 23:47:43.366910 2020] [mpm_event:notice] [pid 5892:tid 140660633114496] AH00493: SIGUSR1 received. Doing graceful restart [Mon Feb 17 23:47:43.422167 2020] [mpm_event:notice] [pid 5892:tid 140660633114496] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 Python/2.7.12 configured -- resuming normal operations [Mon Feb 17 23:47:43.422207 2020] [core:notice] [pid 5892:tid 140660633114496] AH00094: Command line: '/usr/sbin/apache2' [Mon Feb 17 23:47:46.384026 2020] [wsgi:error] [pid 5898:tid 140660447377152] (2)No such file or directory: [client 127.0.0.1:33536] mod_wsgi (pid=5898): Unable to connect to WSGI daemon process '127.0.0.1' on '/var/run/apache2/wsgi.5892.0.1.sock'. The socket file name has changed: $ ls -1 /var/run/apache2/wsgi.*.sock /var/run/apache2/wsgi.6244.1.1.sock With the fix/patch for Xenial: --- $ sudo add-apt-repository ppa:mfo/lp1863232v2 $ sudo apt update $ sudo apt install -y libapache2-mod-wsgi $ sudo systemctl restart apache2 $ dpkg -s libapache2-mod-wsgi | grep ^Version Version: 4.3.0-1.1ubuntu1 Same behavior continues by default: --- $ ls -1 /var/run/apache2/wsgi.*.sock /var/run/apache2/wsgi.4149.0.1.sock $ ps 4149 PID TTY STAT TIME COMMAND 4149 ? Ss 0:00 /usr/sbin/apache2 -k start $ sudo systemctl reload apache2 $ ls -1 /var/run/apache2/wsgi.*.sock /var/run/apache2/wsgi.4149.1.1.sock $ sudo systemctl reload apache2 $ ls -1 /var/run/apache2/wsgi.*.sock /var/run/apache2/wsgi.4149.2.1.sock Manually changing the setting: --- $ echo 'WSGISocketRotation Off' | sudo tee -a /etc/apache2/conf-enabled/wsgi.conf $ sudo systemctl restart apache2 $ ls -1 /var/run/apache2/wsgi.*.sock /var/run/apache2/wsgi.6554.u33.1.sock Notice name pattern change as in patch. (u33 = UID 33 = www-data:) $ id -un 33 www-data And socket file name remains constant: $ sudo systemctl reload apache2 $ ls -1 /var/run/apache2/wsgi.*.sock /var/run/apache2/wsgi.6554.u33.1.sock $ sudo systemctl reload apache2 $ ls -1 /var/run/apache2/wsgi.*.sock /var/run/apache2/wsgi.6554.u33.1.sock Check the problem by placing 'sudo systemctl reload apache2' between the two HTTP requests (second request hits Error 500 now) --- The second HTTP request (after reload) no longer hits Error 503, but now Error 500, and the log file no longer mentions file not found, but a different message. The Error 500 is observed in/consistent with Eoan as well, thus not a missing patch or something -- this is an improvement. $ (cat http-request; sleep 1; sudo systemctl reload apache2; sleep 5; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done 23:51:50 == Trying 127.0.0.1... 23:51:50 == Connected to 127.0.0.1. 23:51:50 == Escape character is '^]'. 23:51:50 == HTTP/1.1 200 OK 23:51:50 == Date: Mon, 17 Feb 2020 23:51:50 GMT 23:51:50 == Server: Apache/2.4.18 (Ubuntu) 23:51:50 == Content-Length: 13 23:51:50 == Keep-Alive: timeout=15, max=100 23:51:50 == Connection: Keep-Alive 23:51:50 == Content-Type: text/plain 23:51:50 == 23:51:50 == Hello World! 23:51:56 == HTTP/1.1 500 Internal Server Error 23:51:56 == Date: Mon, 17 Feb 2020 23:51:56 GMT 23:51:56 == Server: Apache/2.4.18 (Ubuntu) 23:51:56 == Content-Length: 607 23:51:56 == Connection: close 23:51:56 == Content-Type: text/html; charset=iso-8859-1 23:51:56 == 23:51:56 == 23:51:56 == 23:51:56 == 500 Internal Server Error 23:51:56 == 23:51:56 ==

Internal Server Error

23:51:56 ==

The server encountered an internal error or 23:51:56 == misconfiguration and was unable to complete 23:51:56 == your request.

23:51:56 ==

Please contact the server administrator at 23:51:56 == webmaster@localhost to inform them of the time this error occurred, 23:51:56 == and the actions you performed just before this error.

23:51:56 ==

More information about this error may be available 23:51:56 == in the server error log.

23:51:56 ==
23:51:56 ==
Apache/2.4.18 (Ubuntu) Server at 127.0.0.1 Port 80
23:51:56 == 23:51:56 == Connection closed by foreign host. $ sudo tail /var/log/apache2/error.log ... [Mon Feb 17 23:51:51.260775 2020] [mpm_event:notice] [pid 6554:tid 140543840806784] AH00493: SIGUSR1 received. Doing graceful restart [Mon Feb 17 23:51:51.326786 2020] [mpm_event:notice] [pid 6554:tid 140543840806784] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 Python/2.7.12 configured -- resuming normal operations [Mon Feb 17 23:51:51.326873 2020] [core:notice] [pid 6554:tid 140543840806784] AH00094: Command line: '/usr/sbin/apache2' [Mon Feb 17 23:51:56.277943 2020] [wsgi:alert] [pid 7039:tid 140543732401920] mod_wsgi (pid=7039): Request origin could not be validated. [Mon Feb 17 23:51:56.278181 2020] [wsgi:error] [pid 6907:tid 140543690430208] [client 127.0.0.1:33556] Truncated or oversized response headers received from daemon process '127.0.0.1': /var/www/html/hello-world.py