Apache 2.4.x: mod_http2 sends empty response after MaxRequestsPerChild

[Antonios Tsianakas]

[ Impact ]

Apache2 users with mod_http2 enabled can face problems when the MaxRequestsPerChild limit is reached. In this scenario, apache2 will send a GOAWAY packet too early in the connection, which will cause no data to be transferred to the client (i.e., an empty HTTP response from the server).

[ Test Plan ]

This problem reproduces on Bionic, Focal and Impish. The instructions are almost the same for all 3, but on Bionic an extra step is necessary to make sure apache2 will use HTTP2.

$ lxc launch ubuntu-daily:focal bug1969629-focal
$ lxc shell bug1969629-focal
# apt update
# apt install -y apache2
# a2enmod http2
# [ $(lsb_release -sc) = "bionic" ] && (echo -e "\nProtocols h2 h2c http/1.1" >> /etc/apache2/apache2.conf)
# cat > /etc/apache2/mods-available/mpm_event.conf << __EOF__
<IfModule mpm_event_module>
        StartServers 2
        MaxRequestsPerChild 3
</IfModule>
__EOF__
# systemctl restart apache2.service
# while true; do if [[ $(curl -o /dev/null -s -k --http2 http://localhost/ -w '%{size_download}') == 0 ]]; then echo "BUG DETECTED"; break; fi; done
BUG DETECTED

The "while" command above should never stop, but as can be seen, it does stop because the size of the downloaded file (index.html, in this case) is zero. This can also be confirmed by looking at the server logs:

127.0.0.1 - - [26/Apr/2022:20:13:46 +0000] "GET / HTTP/1.1" 101 125 "-" "curl/7.58.0"
127.0.0.1 - - "GET / HTTP/2.0" 200 0 "-" "curl/7.58.0"

[ Where problems could occur ]

Even though the patch is relatively simple, it touches an area that can be a bit complex: session processing of an HTTP request. There are upstream tests that exercise this area and I believe we can be reasonably confident that this change does the right thing. If a regression is to occur, it would likely manifest in the form of a bug in the HTTP2 session handling, and the quicker solution here would be to revert the change while revisiting the patch to determine what went wrong (possibly with upstream help).

[ Original Description ]

We run into a bug in mod_http2 for Apache 2.4.x.

Every time MaxRequestsPerChild is reached, an empty response is sent to the client:
https://bz.apache.org/bugzilla/show_bug.cgi?id=65731

This bug was fixed and backported to v2.4.52:
https://github.com/apache/httpd/pull/281

Any chance that this fix get backported to the Bionic and Focal Package?

[Antonios-m (antonios-m)]

When requesting a simple image file with HTTP2 every time MaxRequestsPerChild is reached, an empty response is sent to the client. This is even reproducible.

Server Version: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f
Server MPM: event
OS: Ubuntu 20.04

Server Version: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1
Server MPM: worker
OS: Ubuntu 18.04

Same issue exists with mpm_event and mpm_worker.

My configuration:

<IfModule mpm_event_module>
    StartServers 8
    MinSpareThreads 4
    MaxSpareThreads 12
    ThreadLimit 64
    ThreadsPerChild 16
    MaxRequestsPerChild 100
    ServerLimit 64
    MaxClients 64
</IfModule>

mod_http2 no custom configuration in use.

Test command:
for i in {1..2000}; do if [[ $(curl -o /dev/null -s -k --http2 https://example.net/2101583.jpg?test=$i -w '%{size_download}') == 0 ]]; then break; fi; done

Access log output:
8.8.8.8 - - [08/Dec/2021:18:20:32 +0100] "GET /2101583.jpg?fakeVar=33 HTTP/2.0" 200 17115 "-" "curl/7.68.0"
8.8.8.8 - - [08/Dec/2021:18:20:33 +0100] "GET /2101583.jpg?fakeVar=34 HTTP/2.0" 200 17115 "-" "curl/7.68.0"
8.8.8.8 - - [08/Dec/2021:18:20:33 +0100] "GET /2101583.jpg?fakeVar=35 HTTP/2.0" 200 0 "-" "curl/7.68.0"
8.8.8.8 - - [08/Dec/2021:18:20:33 +0100] "GET /2101583.jpg?fakeVar=36 HTTP/2.0" 200 17115 "-" "curl/7.68.0"
8.8.8.8 - - [08/Dec/2021:18:20:33 +0100] "GET /2101583.jpg?fakeVar=37 HTTP/2.0" 200 17115 "-" "curl/7.68.0"
8.8.8.8 - - [08/Dec/2021:18:20:33 +0100] "GET /2101583.jpg?fakeVar=38 HTTP/2.0" 200 17115 "-" "curl/7.68.0"
8.8.8.8 - - [08/Dec/2021:18:20:33 +0100] "GET /2101583.jpg?fakeVar=39 HTTP/2.0" 200 17115 "-" "curl/7.68.0"
...
8.8.8.8 - - [08/Dec/2021:18:20:37 +0100] "GET /2101583.jpg?fakeVar=135 HTTP/2.0" 200 0 "-" "curl/7.68.0"
...
8.8.8.8 - - [08/Dec/2021:18:20:42 +0100] "GET /2101583.jpg?fakeVar=235 HTTP/2.0" 200 0 "-" "curl/7.68.0"
...
8.8.8.8 - - [08/Dec/2021:18:20:47 +0100] "GET /2101583.jpg?fakeVar=335 HTTP/2.0" 200 0 "-" "curl/7.68.0"

[Rpluem (rpluem)]

Does this happen as well with 2.4.51 compiled from source?

[Antonios-m (antonios-m)]

Created attachment 38125
Apache Trace8 Log

[Antonios-m (antonios-m)]

Yes, and with 2.4.51 the responses with an empty body wasn't logged in the access log.

When this happens curl reports "curl: (16) Error in the HTTP2 framing layer". Attached is the trace8 output of such a request.

Server version: Apache/2.4.51 (Unix)
Server built: Dec 9 2021 12:24:50
Server's Module Magic Number: 20120211:118
Server loaded: APR 1.6.5, APR-UTIL 1.6.1
Compiled using: APR 1.6.5, APR-UTIL 1.6.1
Architecture: 64-bit
Server MPM: event
  threaded: yes (fixed thread count)
    forked: yes (variable process count)

[Icing (icing)]

Thanks for the log. I can see that mod_http2 sees the MPM stopping and sends a GOAWAY frame *before* the HTTP/2 session has been fully started. That is indeed a violation of the H2 protocol.

Working on a test and patch.

[Icing (icing)]

Created attachment 38127
patch on 2.4.x code base

Proposed fix for 2.4.x code base.

[Icing (icing)]

I was able to reproduce this in a test case and made a fix to be backported to 2.4.x in the next release.

I attached the patch for the 2.4.x codebase, if you can build from source. If you want to build from the github module version, there is <https://github.com/icing/mod_h2/releases/tag/v1.15.27>.

[Minfrin-s (minfrin-s)]

Backported to v2.4.52:

 *) mod_http2: fixes PR65731 and https://github.com/icing/mod_h2/issues/212
    trunk patch: na, fixed on 2.4.x source base
    backport PR: https://github.com/apache/httpd/pull/281
    +1: icing, minfrin, ylavic

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Paride Legovini (paride)]

Hi Antonios and thanks for this bug report. I'm marking this report as Fix Released as according to the version numbers this bug is fixed in Ubuntu 22.04 LTS (shipping apache2 2.4.52). I am however adding tasks for the stable releases, as by looking a the patch you pointed to (thanks!) I think this is good SRU material.

[Brian Murray (brian-murray)]

Hello Antonios, or anyone else affected,

Accepted apache2 into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.48-3.1ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Antonios, or anyone else affected,

Accepted apache2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.41-4ubuntu3.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Antonios, or anyone else affected,

Accepted apache2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.29-1ubuntu4.23 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apache2 (2.4.48-3.1ubuntu3.4) for impish have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.47.91-1ubuntu1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/impish/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apache2 (2.4.29-1ubuntu4.23) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

cacti/1.1.38+ds1-1 (amd64, arm64)
pinto/0.97+dfsg-4ubuntu1 (i386)
redmine/3.4.4-1ubuntu0.1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apache2 (2.4.41-4ubuntu3.11) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.44.1-1ubuntu1 (ppc64el, amd64, arm64)
nova/2:21.2.4-0ubuntu1 (armhf)
passenger/5.0.30-1.1ubuntu3 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Sergio Durigan Junior (sergiodj)]

Sorry about the delay. Here's the verification for Bionic:

First, verifying that the bug is reproducible:

# apt policy apache2
apache2:
  Installed: 2.4.29-1ubuntu4.22
  Candidate: 2.4.29-1ubuntu4.22
  Version table:
 *** 2.4.29-1ubuntu4.22 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.29-1ubuntu4 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

# while true; do if [[ $(curl -o /dev/null -s -k --http2 http://localhost/ -w '%{size_download}') == 0 ]]; then echo "BUG DETECTED"; break; fi; done
BUG DETECTED

Now, installing the package from -proposed and verifying that it fixes the bug:

# apt policy apache2
apache2:
  Installed: 2.4.29-1ubuntu4.23
  Candidate: 2.4.29-1ubuntu4.23
  Version table:
 *** 2.4.29-1ubuntu4.23 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.29-1ubuntu4.22 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     2.4.29-1ubuntu4 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

# while true; do if [[ $(curl -o /dev/null -s -k --http2 http://localhost/ -w '%{size_download}') == 0 ]]; then echo "BUG DETECTED"; break; fi; done
^C
#

As can be seen, the loop did not return right away and instead had to be manually stopped.

This concludes the verification for Bionic.

[Sergio Durigan Junior (sergiodj)]

Verification for Focal.

First, verifying that the bug is reproducible:

# apt policy apache2
apache2:
  Installed: 2.4.41-4ubuntu3.10
  Candidate: 2.4.41-4ubuntu3.10
  Version table:
 *** 2.4.41-4ubuntu3.10 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.41-4ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

# while true; do if [[ $(curl -o /dev/null -s -k --http2 http://localhost/ -w '%{size_download}') == 0 ]]; then echo "BUG DETECTED"; break; fi; done
BUG DETECTED

Now, installing the package from -proposed and verifying that it fixes the bug:

# apt policy apache2
apache2:
  Installed: 2.4.41-4ubuntu3.11
  Candidate: 2.4.41-4ubuntu3.11
  Version table:
 *** 2.4.41-4ubuntu3.11 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.41-4ubuntu3.10 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     2.4.41-4ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

# while true; do if [[ $(curl -o /dev/null -s -k --http2 http://localhost/ -w '%{size_download}') == 0 ]]; then echo "BUG DETECTED"; break; fi; done
^C
#

As can be seen, the loop did not return right away and instead had to be manually stopped.

This concludes the verification for Focal.

[Sergio Durigan Junior (sergiodj)]

Verification for Impish.

First, verifying that the bug is reproducible:

# apt policy apache2
apache2:
  Installed: 2.4.48-3.1ubuntu3.3
  Candidate: 2.4.48-3.1ubuntu3.3
  Version table:
 *** 2.4.48-3.1ubuntu3.3 500
        500 http://archive.ubuntu.com/ubuntu impish-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu impish-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.48-3.1ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu impish/main amd64 Packages

# while true; do if [[ $(curl -o /dev/null -s -k --http2 http://localhost/ -w '%{size_download}') == 0 ]]; then echo "BUG DETECTED"; break; fi; done
BUG DETECTED

Now, installing the package from -proposed and verifying that it fixes the bug:

# apt policy apache2
apache2:
  Installed: 2.4.48-3.1ubuntu3.4
  Candidate: 2.4.48-3.1ubuntu3.4
  Version table:
 *** 2.4.48-3.1ubuntu3.4 500
        500 http://archive.ubuntu.com/ubuntu impish-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.48-3.1ubuntu3.3 500
        500 http://archive.ubuntu.com/ubuntu impish-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu impish-security/main amd64 Packages
     2.4.48-3.1ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu impish/main amd64 Packages

# while true; do if [[ $(curl -o /dev/null -s -k --http2 http://localhost/ -w '%{size_download}') == 0 ]]; then echo "BUG DETECTED"; break; fi; done
^C
#

As can be seen, the loop did not return right away and instead had to be manually stopped.

This concludes the verification for Impish.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.48-3.1ubuntu3.4

---------------
apache2 (2.4.48-3.1ubuntu3.4) impish; urgency=medium

  * d/p/mod_http2-Don-t-send-GOAWAY-too-early-when-MaxReques.patch:
    Don't send GOAWAY too early on new connections when
    MaxRequestsPerChild has been reached. (LP: #1969629)

 -- Sergio Durigan Junior <email address hidden> Tue, 26 Apr 2022 15:55:37 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for apache2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.41-4ubuntu3.11

---------------
apache2 (2.4.41-4ubuntu3.11) focal; urgency=medium

  * d/p/mod_http2-Don-t-send-GOAWAY-too-early-when-MaxReques.patch:
    Don't send GOAWAY too early on new connections when
    MaxRequestsPerChild has been reached. (LP: #1969629)

 -- Sergio Durigan Junior <email address hidden> Tue, 26 Apr 2022 14:02:11 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.29-1ubuntu4.23

---------------
apache2 (2.4.29-1ubuntu4.23) bionic; urgency=medium

  * d/p/mod_http2-Don-t-send-GOAWAY-too-early-when-MaxReques.patch:
    Don't send GOAWAY too early on new connections when
    MaxRequestsPerChild has been reached. (LP: #1969629)

 -- Sergio Durigan Junior <email address hidden> Mon, 25 Apr 2022 20:46:43 -0400