Apache 2.4.x: mod_http2 sends empty response after MaxRequestsPerChild

Bug #1969629 reported by Antonios Tsianakas
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Apache2 Web Server
Fix Released
Medium
apache2 (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Committed
Undecided
Sergio Durigan Junior
Focal
Fix Committed
Undecided
Sergio Durigan Junior
Impish
Fix Committed
Undecided
Sergio Durigan Junior

Bug Description

[ Impact ]

Apache2 users with mod_http2 enabled can face problems when the MaxRequestsPerChild limit is reached. In this scenario, apache2 will send a GOAWAY packet too early in the connection, which will cause no data to be transferred to the client (i.e., an empty HTTP response from the server).

[ Test Plan ]

This problem reproduces on Bionic, Focal and Impish. The instructions are almost the same for all 3, but on Bionic an extra step is necessary to make sure apache2 will use HTTP2.

$ lxc launch ubuntu-daily:focal bug1969629-focal
$ lxc shell bug1969629-focal
# apt update
# apt install -y apache2
# a2enmod http2
# [ $(lsb_release -sc) = "bionic" ] && (echo -e "\nProtocols h2 h2c http/1.1" >> /etc/apache2/apache2.conf)
# cat > /etc/apache2/mods-available/mpm_event.conf << __EOF__
<IfModule mpm_event_module>
        StartServers 2
        MaxRequestsPerChild 3
</IfModule>
__EOF__
# systemctl restart apache2.service
# while true; do if [[ $(curl -o /dev/null -s -k --http2 http://localhost/ -w '%{size_download}') == 0 ]]; then echo "BUG DETECTED"; break; fi; done
BUG DETECTED

The "while" command above should never stop, but as can be seen, it does stop because the size of the downloaded file (index.html, in this case) is zero. This can also be confirmed by looking at the server logs:

127.0.0.1 - - [26/Apr/2022:20:13:46 +0000] "GET / HTTP/1.1" 101 125 "-" "curl/7.58.0"
127.0.0.1 - - "GET / HTTP/2.0" 200 0 "-" "curl/7.58.0"

[ Where problems could occur ]

Even though the patch is relatively simple, it touches an area that can be a bit complex: session processing of an HTTP request. There are upstream tests that exercise this area and I believe we can be reasonably confident that this change does the right thing. If a regression is to occur, it would likely manifest in the form of a bug in the HTTP2 session handling, and the quicker solution here would be to revert the change while revisiting the patch to determine what went wrong (possibly with upstream help).

[ Original Description ]

We run into a bug in mod_http2 for Apache 2.4.x.

Every time MaxRequestsPerChild is reached, an empty response is sent to the client:
https://bz.apache.org/bugzilla/show_bug.cgi?id=65731

This bug was fixed and backported to v2.4.52:
https://github.com/apache/httpd/pull/281

Any chance that this fix get backported to the Bionic and Focal Package?

Related branches

Revision history for this message
In , Antonios-m (antonios-m) wrote :

When requesting a simple image file with HTTP2 every time MaxRequestsPerChild is reached, an empty response is sent to the client. This is even reproducible.

Server Version: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f
Server MPM: event
OS: Ubuntu 20.04

Server Version: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1
Server MPM: worker
OS: Ubuntu 18.04

Same issue exists with mpm_event and mpm_worker.

My configuration:

<IfModule mpm_event_module>
    StartServers 8
    MinSpareThreads 4
    MaxSpareThreads 12
    ThreadLimit 64
    ThreadsPerChild 16
    MaxRequestsPerChild 100
    ServerLimit 64
    MaxClients 64
</IfModule>

mod_http2 no custom configuration in use.

Test command:
for i in {1..2000}; do if [[ $(curl -o /dev/null -s -k --http2 https://example.net/2101583.jpg?test=$i -w '%{size_download}') == 0 ]]; then break; fi; done

Access log output:
8.8.8.8 - - [08/Dec/2021:18:20:32 +0100] "GET /2101583.jpg?fakeVar=33 HTTP/2.0" 200 17115 "-" "curl/7.68.0"
8.8.8.8 - - [08/Dec/2021:18:20:33 +0100] "GET /2101583.jpg?fakeVar=34 HTTP/2.0" 200 17115 "-" "curl/7.68.0"
8.8.8.8 - - [08/Dec/2021:18:20:33 +0100] "GET /2101583.jpg?fakeVar=35 HTTP/2.0" 200 0 "-" "curl/7.68.0"
8.8.8.8 - - [08/Dec/2021:18:20:33 +0100] "GET /2101583.jpg?fakeVar=36 HTTP/2.0" 200 17115 "-" "curl/7.68.0"
8.8.8.8 - - [08/Dec/2021:18:20:33 +0100] "GET /2101583.jpg?fakeVar=37 HTTP/2.0" 200 17115 "-" "curl/7.68.0"
8.8.8.8 - - [08/Dec/2021:18:20:33 +0100] "GET /2101583.jpg?fakeVar=38 HTTP/2.0" 200 17115 "-" "curl/7.68.0"
8.8.8.8 - - [08/Dec/2021:18:20:33 +0100] "GET /2101583.jpg?fakeVar=39 HTTP/2.0" 200 17115 "-" "curl/7.68.0"
...
8.8.8.8 - - [08/Dec/2021:18:20:37 +0100] "GET /2101583.jpg?fakeVar=135 HTTP/2.0" 200 0 "-" "curl/7.68.0"
...
8.8.8.8 - - [08/Dec/2021:18:20:42 +0100] "GET /2101583.jpg?fakeVar=235 HTTP/2.0" 200 0 "-" "curl/7.68.0"
...
8.8.8.8 - - [08/Dec/2021:18:20:47 +0100] "GET /2101583.jpg?fakeVar=335 HTTP/2.0" 200 0 "-" "curl/7.68.0"

Revision history for this message
In , Rpluem (rpluem) wrote :

Does this happen as well with 2.4.51 compiled from source?

Revision history for this message
In , Antonios-m (antonios-m) wrote :

Created attachment 38125
Apache Trace8 Log

Revision history for this message
In , Antonios-m (antonios-m) wrote :

Yes, and with 2.4.51 the responses with an empty body wasn't logged in the access log.

When this happens curl reports "curl: (16) Error in the HTTP2 framing layer". Attached is the trace8 output of such a request.

Server version: Apache/2.4.51 (Unix)
Server built: Dec 9 2021 12:24:50
Server's Module Magic Number: 20120211:118
Server loaded: APR 1.6.5, APR-UTIL 1.6.1
Compiled using: APR 1.6.5, APR-UTIL 1.6.1
Architecture: 64-bit
Server MPM: event
  threaded: yes (fixed thread count)
    forked: yes (variable process count)

Revision history for this message
In , Icing (icing) wrote :

Thanks for the log. I can see that mod_http2 sees the MPM stopping and sends a GOAWAY frame *before* the HTTP/2 session has been fully started. That is indeed a violation of the H2 protocol.

Working on a test and patch.

Revision history for this message
In , Icing (icing) wrote :

Created attachment 38127
patch on 2.4.x code base

Proposed fix for 2.4.x code base.

Revision history for this message
In , Icing (icing) wrote :

I was able to reproduce this in a test case and made a fix to be backported to 2.4.x in the next release.

I attached the patch for the 2.4.x codebase, if you can build from source. If you want to build from the github module version, there is <https://github.com/icing/mod_h2/releases/tag/v1.15.27>.

Revision history for this message
In , Minfrin-s (minfrin-s) wrote :

Backported to v2.4.52:

 *) mod_http2: fixes PR65731 and https://github.com/icing/mod_h2/issues/212
    trunk patch: na, fixed on 2.4.x source base
    backport PR: https://github.com/apache/httpd/pull/281
    +1: icing, minfrin, ylavic

summary: - mod_http2: fixes 2 regressions in server limit handling.
+ Apache 2.4.x: mod_http2 sends empty response after MaxRequestsPerChild
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apache2 (Ubuntu):
status: New → Confirmed
Revision history for this message
Paride Legovini (paride) wrote :

Hi Antonios and thanks for this bug report. I'm marking this report as Fix Released as according to the version numbers this bug is fixed in Ubuntu 22.04 LTS (shipping apache2 2.4.52). I am however adding tasks for the stable releases, as by looking a the patch you pointed to (thanks!) I think this is good SRU material.

Changed in apache2 (Ubuntu):
status: Confirmed → Fix Released
Changed in apache2 (Ubuntu Bionic):
status: New → Triaged
Changed in apache2 (Ubuntu Focal):
status: New → Triaged
Changed in apache2 (Ubuntu Impish):
status: New → Triaged
tags: added: server-todo
description: updated
Changed in apache2 (Ubuntu Bionic):
assignee: nobody → Sergio Durigan Junior (sergiodj)
Changed in apache2 (Ubuntu Focal):
assignee: nobody → Sergio Durigan Junior (sergiodj)
Changed in apache2 (Ubuntu Impish):
assignee: nobody → Sergio Durigan Junior (sergiodj)
Changed in apache2 (Ubuntu Bionic):
status: Triaged → In Progress
Changed in apache2 (Ubuntu Focal):
status: Triaged → In Progress
Changed in apache2 (Ubuntu Impish):
status: Triaged → In Progress
Changed in apache2:
importance: Unknown → Medium
status: Unknown → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Antonios, or anyone else affected,

Accepted apache2 into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.48-3.1ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in apache2 (Ubuntu Impish):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-impish
Changed in apache2 (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Antonios, or anyone else affected,

Accepted apache2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.41-4ubuntu3.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in apache2 (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Antonios, or anyone else affected,

Accepted apache2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.29-1ubuntu4.23 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (apache2/2.4.48-3.1ubuntu3.4)

All autopkgtests for the newly accepted apache2 (2.4.48-3.1ubuntu3.4) for impish have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.47.91-1ubuntu1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/impish/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (apache2/2.4.29-1ubuntu4.23)

All autopkgtests for the newly accepted apache2 (2.4.29-1ubuntu4.23) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

cacti/1.1.38+ds1-1 (amd64, arm64)
pinto/0.97+dfsg-4ubuntu1 (i386)
redmine/3.4.4-1ubuntu0.1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (apache2/2.4.41-4ubuntu3.11)

All autopkgtests for the newly accepted apache2 (2.4.41-4ubuntu3.11) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.44.1-1ubuntu1 (ppc64el, amd64, arm64)
nova/2:21.2.4-0ubuntu1 (armhf)
passenger/5.0.30-1.1ubuntu3 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.