daemon rotates socket on restart

[Dariusz Gadomski]

[Impact]

 * Lack of option for disabling wsgi socket rotation leads to errors on graceful restarts, making them not as graceful.
 * Specifically, when mod-wsgi is running in daemon mode (which uses
   sockets), and a graceful restart ('sudo systemctl reload apache2')
   happens, the socket filename changes, and upcoming HTTP requests
   in a keep-alive connection (i.e., same connection is re-utilized
   for multiple HTTP requests) initiated before the graceful restart
   are failed (HTTP 503 error) because the socket file is not found.

 * This change introduces a new config option WSGISocketRotation that allows to disable the rotation.
 * The option is disabled by default, so the default behavior remains
   consistent with the previous versions (ie, socket rotation occurs.)
 * This is actually desired, and designed that way by upstream,
   because disabling socket rotation requires no _wsgi_ config changes,
   as they can impact/alter the upcoming HTTP requests (see patch link.)

[Test Case]

 * Setup apache2 with mod-wsgi.
 * Make sure there are some wsgi sockets open.
 * Reload apache gracefully.
 * (Detailed steps are provided in comments #9 and #10)

Expected result:
No errors related to sockets in the logs

Actual result:
There are error messages related to sockets in the logs.

[Regression Potential]

 * Since the value is set to On by default any regressions would manifest only after explicitly setting it to Off.
 * After it's set to off WSGI application behavior will change on reloads - connections should be resumed instead of cancelled.

[Other Info]

 * The fix has been introduced in mod-wsgi 4.6.0,
   thus already present in Disco/Eoan/Focal,
   only needed in Xenial/Bionic.

 * Original bug description:
On Apache reloads the WSGI daemon tries to rotate wsgi sockets causing unnecessary log entries, especially in OpenStack context.

This has been addressed in mod-wsgi upstream (4.6.0) and could be backported to Ubuntu.

[Dariusz Gadomski (dgadomski)]

Since it is already present upstream it's fixed for focal and eoan.

[Dariusz Gadomski (dgadomski)]

SRU proposal for Xenial.

[Dariusz Gadomski (dgadomski)]

SRU proposal for bionic.

[Mauricio Faria de Oliveira (mfo)]

Hi Dariusz,

I reviewed the debdiffs, they look good overall.

I just had some minor changes to reflect review
practices/comments that I have been thru myself:

- Nice catch on updating the Maintainers field.

- Changelog order of patch file/description:
  I changed to file first, description later,
  as I've seen as more used/standard practice.

- DEP3 headers are present in the .patch; good.

I updated from 'Origin: upstream' to 'backport'
because the patch has a removal of (unneeded)
release notes file.

Even being uneeded, there were changes to the
upstream patch, so it's no longer clean apply/
cherry pick, thus the change to 'backport',
per Debian DEP-3 spec [1]:

  "backport" (in the case of an upstream patch that had to be modified to apply on the current version)

I added a '[backport]' section to the .patch
file describing that. (and this other change:)

I also noted that on Xenial there are changes
actually needed to src/server/__init__.py, so
arguably indeed a "true" backport this time.

Oh, and there's a digit missing in Bug-Ubuntu
number, just added that/goes to right URL now.

- Xenial fix on .patch file:

Still on Xenial there's an extra '+'/plus sign
in the commented line added to __init__.py, so
I fixed that one:

   +++#WSGISocketRotation Off

Bionic had that right:

   ++#WSGISocketRotation Off

- Version numbers look good/upgrade path is OK:

- - Versions 4.3.0-1.1ubuntu1 and 4.5.17-1ubuntu1
    never existed in package's publishing history [2].

- - Upgrade path is OK in the same release.

$ grep -m2 urgency= x/dput/lp1863232_xenial_mod-wsgi.debdiff
+mod-wsgi (4.3.0-1.1ubuntu1) xenial; urgency=medium
 mod-wsgi (4.3.0-1.1build1) xenial; urgency=medium

$ dpkg --compare-versions 4.3.0-1.1build1 lt 4.3.0-1.1ubuntu1 ; echo $?
0

$ grep -m2 urgency= b/dput/lp1863232_bionic_mod-wsgi.debdiff
+mod-wsgi (4.5.17-1ubuntu1) bionic; urgency=medium
 mod-wsgi (4.5.17-1) unstable; urgency=medium

$ dpkg --compare-versions 4.5.17-1 lt 4.5.17-1ubuntu1; echo $?
0

- - Upgrade path is OK across releases (t/x/b/e)

$ rmadison -a source mod-wsgi | grep -e trusty -e eoan
 mod-wsgi | 3.4-4ubuntu2 | trusty | source
 mod-wsgi | 3.4-4ubuntu2.1.14.04.2 | trusty-security | source
 mod-wsgi | 3.4-4ubuntu2.1.14.04.2 | trusty-updates | source
 mod-wsgi | 4.6.5-1 | eoan

$ dpkg --compare-versions 3.4-4ubuntu2.1.14.04.2 lt 4.3.0-1.1ubuntu1 ; echo $?
0

$ dpkg --compare-versions 4.3.0-1.1ubuntu1 lt 4.5.17-1ubuntu1; echo $?
0

$ dpkg --compare-versions 4.5.17-1ubuntu1 lt 4.6.5-1 ; echo $?
0

That's it!

cheers,
Mauricio

[1] https://dep-team.pages.debian.net/deps/dep3/
[2] https://launchpad.net/ubuntu/+source/mod-wsgi/+publishinghistory

[Mauricio Faria de Oliveira (mfo)]

Attaching the updated debdiffs for reference.

They build successfully on a PPA for all architectures,
and have been used on steps to reproduce/verify (below.)

[Mauricio Faria de Oliveira (mfo)]

Dariusz,

As part of my learning homework to be able to
technically review the patch in this proposal,

I came across three things that may be useful
for the bug description/SRU template, so I'll
just add them, if you don't mind.

1) Understanding the scenario where the bug
happens (mod-wsgi running in daemon mode/not
embedded mode, only the former uses sockets;
and reload/not-restart apache2 between HTTP
requests in the same keep-alive connection)
[1, 2].

2) Detailed steps to reproduce/verify the bug.

3) Confirming that the default behavior with
the patch remains as in the previous version
(of course, as it should/is supposed to, but
it's worth mentioning in bug description :-)

With all those points now understood/verified,
I'll proceed with the upload for SRU to B/X.

Thanks!

cheers,
Mauricio

[1] https://github.com/GrahamDumpleton/mod_wsgi/commit/13169f2a0610d7451fae92a414e8e20b91e348c9#diff-3e8b16b2885169dcec2dac843521e12cR29
[2] https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html#delegation-to-daemon-process

[Mauricio Faria de Oliveira (mfo)]

Steps to reproduce/verify on Xenial:
===

References:
- https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html
- https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html#delegation-to-daemon-process
- https://github.com/GrahamDumpleton/mod_wsgi/commit/13169f2a0610d7451fae92a414e8e20b91e348c9#diff-3e8b16b2885169dcec2dac843521e12cR29
- https://www.howtoforge.com/tutorial/how-to-run-python-scripts-with-apache-and-mod_wsgi-on-ubuntu-18-04/

Create a container with apache2/mod-wsgi in daemon mode,
w/ keep-alive timeout long enough, and wsgi hello-world:
---

$ lxc launch ubuntu:xenial lp1863232x
$ lxc exec lp1863232x -- su - ubuntu

$ sudo apt update
$ sudo apt install -y apache2 libapache2-mod-wsgi

$ sudo sed -i '/^KeepAliveTimeout/ s/ .*/ 15/' /etc/apache2/apache2.conf

$ cat <<EOF | sudo tee /etc/apache2/conf-available/wsgi.conf
WSGIScriptAlias /hello-world /var/www/html/hello-world.py
WSGIDaemonProcess 127.0.0.1 processes=2 threads=16 display-name=%{GROUP}
WSGIProcessGroup 127.0.0.1
EOF

$ sudo a2enconf wsgi
$ sudo systemctl reload apache2

$ cat <<EOF | sudo tee /var/www/html/hello-world.py
def application(environ, start_response):
    status = '200 OK'
    output = b'Hello World!\n'

    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]

    start_response(status, response_headers)
    return [output]
EOF

$ curl 127.0.0.1/hello-world
Hello World!

Check WSGI socket filename changes when
reloading apache2 (aka graceful restart):
---

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.0.1.sock

$ sudo systemctl reload apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.1.1.sock

$ sudo systemctl reload apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.2.1.sock

Create script and shell one-liner
to send two HTTP requests in the
same connection (keep-alive used)
---

$ cat <<EOF >http-request
GET /hello-world HTTP/1.1
Host: 127.0.0.1
Connection: keep-alive

EOF

One connection/One request:

$ (cat http-request; sleep 1) | telnet 127.0.0.1 80
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
HTTP/1.1 200 OK
Date: Mon, 17 Feb 2020 23:33:19 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 13
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/plain

Hello World!
Connection closed by foreign host.

One connection/Two requests (added timestamps to check timeout values)

$ (cat http-request; sleep 1; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done
23:34:42 == Trying 127.0.0.1...
23:34:42 == Connected to 127.0.0.1.
23:34:42 == Escape character is '^]'.
23:34:42 == HTTP/1.1 200 OK
23:34:42 == Date: Mon, 17 Feb 2020 23:34:42 GMT
23:34:42 == Server: Apache/2.4.18 (Ubuntu)
23:34:42 == Content-Length: 13
23:34:42 == Keep-Alive: timeout=5, max=100
23:34:42 == Connection: Keep-Alive
23:34:42 == Content-Type: text/plain
23:34:42 ==
23:34:42 == Hello World!
23:34:43 == HTTP/1.1 200 OK
23:34:43 == Date: Mon, 17 Feb 2020 23:34:43 GMT
23:34:43 == Server: Apache/2.4.18 (Ubuntu)...

[Mauricio Faria de Oliveira (mfo)]

Steps to reproduce/verify on Bionic:
===

Similarly to steps described for Xenial.
Skipping the identical steps.

$ lxc launch ubuntu:bionic lp1863232b
$ lxc exec lp1863232b -- su - ubuntu

One connection/Two requests (added timestamps to check timeout values)
---

$ (cat http-request; sleep 1; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done
00:03:46 == Trying 127.0.0.1...
00:03:46 == Connected to 127.0.0.1.
00:03:46 == Escape character is '^]'.
00:03:46 == HTTP/1.1 200 OK
00:03:46 == Date: Tue, 18 Feb 2020 00:03:46 GMT
00:03:46 == Server: Apache/2.4.29 (Ubuntu)
00:03:46 == Content-Length: 13
00:03:46 == Vary: Accept-Encoding
00:03:46 == Keep-Alive: timeout=15, max=100
00:03:46 == Connection: Keep-Alive
00:03:46 == Content-Type: text/plain
00:03:46 ==
00:03:46 == Hello World!
00:03:47 == HTTP/1.1 200 OK
00:03:47 == Date: Tue, 18 Feb 2020 00:03:47 GMT
00:03:47 == Server: Apache/2.4.29 (Ubuntu)
00:03:47 == Content-Length: 13
00:03:47 == Vary: Accept-Encoding
00:03:47 == Keep-Alive: timeout=15, max=99
00:03:47 == Connection: Keep-Alive
00:03:47 == Content-Type: text/plain
00:03:47 ==
00:03:47 == Hello World!
00:03:56 == Connection closed by foreign host.

Reproduce the problem by placing
'sudo systemctl reload apache2'
between the two HTTP requests
(second request hits Error 503,
depending on apache2 MPM module.
mpm_event just closes connection,
mpm_worker/mpm_prefork hit 503)
---

$ lsb_release -cs
bionic

$ dpkg -s libapache2-mod-wsgi | grep ^Version
Version: 4.5.17-1

$ sudo a2dismod mpm_event
$ sudo a2enmod mpm_worker
$ sudo systemctl restart apache2

For reference on socket filename:

$ sudo systemctl restart apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4982.0.1.sock

$ (cat http-request; sleep 1; sudo systemctl reload apache2; sleep 5; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done
14:46:04 == Trying 127.0.0.1...
14:46:04 == Connected to 127.0.0.1.
14:46:04 == Escape character is '^]'.
14:46:04 == HTTP/1.1 200 OK
14:46:04 == Date: Tue, 18 Feb 2020 14:46:04 GMT
14:46:04 == Server: Apache/2.4.29 (Ubuntu)
14:46:04 == Content-Length: 13
14:46:04 == Vary: Accept-Encoding
14:46:04 == Keep-Alive: timeout=15, max=100
14:46:04 == Connection: Keep-Alive
14:46:04 == Content-Type: text/plain
14:46:04 ==
14:46:04 == Hello World!
14:46:10 == HTTP/1.1 503 Service Unavailable
14:46:10 == Date: Tue, 18 Feb 2020 14:46:10 GMT
14:46:10 == Server: Apache/2.4.29 (Ubuntu)
14:46:10 == Content-Length: 374
14:46:10 == Connection: close
14:46:10 == Content-Type: text/html; charset=iso-8859-1
14:46:10 ==
14:46:10 == <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
14:46:10 == <html><head>
14:46:10 == <title>503 Service Unavailable</title>
14:46:10 == </head><body>
14:46:10 == <h1>Service Unavailable</h1>
14:46:10 == <p>The server is temporarily unable to service your
14:46:10 == request due to maintenance downtime or capacity
14:46:10 == problems. Please try again later.</p>
14:46:10 == <hr>
14:46:10 == <address>Apache/2.4.29 (Ubuntu) Server at 127.0.0.1 Port 80</address>
14:46:11 == </body></html>
14:46:11 ...

[Mauricio Faria de Oliveira (mfo)]

Uploaded changes to B/X.

There are no other mod-wsgi uploads in the unapproved queue nor waiting in -proposed/pending SRU.

mod-wsgi (source)
 4.5.17-1ubuntu1 main python medium ubuntu-server Proposed 56 seconds ago

 mod-wsgi (source)
 4.3.0-1.1ubuntu1 main python medium ubuntu-server Proposed 53 seconds ago

[Łukasz Zemczak (sil2100)]

Hello Dariusz, or anyone else affected,

Accepted mod-wsgi into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mod-wsgi/4.5.17-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Dariusz, or anyone else affected,

Accepted mod-wsgi into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mod-wsgi/4.3.0-1.1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dariusz Gadomski (dgadomski)]

Bionic verification:
After installing libapache2-mod-wsgi 4.5.17-1ubuntu1 and setting
WSGISocketRotation Off
in /etc/apache2/mods-available/wsgi.conf I confirm there is no socket rotation after reload:

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4963.u33.1.sock
$ sudo systemctl reload apache2.service
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4963.u33.1.sock

[Dariusz Gadomski (dgadomski)]

Xenial verification:
Similarly to bionic with Xenial version of libapache2-mod-wsgi 4.3.0-1.1ubuntu1 and setting
WSGISocketRotation Off
in /etc/apache2/conf-enabled/wsgi.conf there's no socket rotation:
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.3170.u33.1.sock
$ sudo systemctl reload apache2.service
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.3170.u33.1.sock
$ apt-cache policy libapache2-mod-wsgi | grep Installed
  Installed: 4.3.0-1.1ubuntu1

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted mod-wsgi (4.5.17-1ubuntu1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

nova/2:17.0.12-0ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#mod-wsgi

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for mod-wsgi has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mod-wsgi - 4.5.17-1ubuntu1

---------------
mod-wsgi (4.5.17-1ubuntu1) bionic; urgency=medium

  * d/p/allow-disabling-daemon-socket-rotation-on-restart.patch:
    Add option for disabling daemon socket rotation on restarts (LP: #1863232)

 -- Dariusz Gadomski <email address hidden> Fri, 14 Feb 2020 12:08:41 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package mod-wsgi - 4.3.0-1.1ubuntu1

---------------
mod-wsgi (4.3.0-1.1ubuntu1) xenial; urgency=medium

  * d/p/allow-disabling-daemon-socket-rotation-on-restart.patch:
    Add option for disabling daemon socket rotation on restarts (LP: #1863232)

 -- Dariusz Gadomski <email address hidden> Fri, 14 Feb 2020 12:36:00 +0100