Regression in Apache 2.4.52-1ubuntu4.10 causes intermittent errors in mod_proxy_http2 backend

[Sebastian Marsching]

Version 2.4.52-1ubuntu4.10 of the apache2 package that was released in order to fix the issues described in USN-6885-1 introduced a regression that leads to intermittent 502 errors when proxying requests to a HTTP/2 server. This issue is not present in version 2.4.52-1ubuntu4.9 of the package, so it must have been introduced as a side effect of the security fixes.

The affected server uses the following configuration to proxy requests to an Apache HTTPd version 2.4.61 that is running inside a Docker container:

  ProxyPass "/" "h2c://localhost:8080/"
  ProxyPassReverse "/" "http://localhost:8080/"
  ProxyPreserveHost On
  RequestHeader set "X-Forwarded-Proto" "expr=%{REQUEST_SCHEME}"
  RequestHeader unset "X-Forwarded-For"
  RequestHeader unset "X-Forwarded-Host"
  RequestHeader unset "X-Forwarded-Server"

This configuration works without any problems when using Apache 2.4.52-1ubuntu4.9. After upgrading to Apache 2.4.52-1ubuntu4.10, the following error message is intermittently displayed when trying to access a proxied resource:

  Bad Gateway

  The proxy server received an invalid response from an upstream server.

  The proxy server could not handle the request

  Reason: URI has no hostname: /…

  Apache/2.4.52 (Ubuntu) Server at … Port 443

The request does not show up in the log of the server running inside the Docker container, so it seems like it is not forwarded correctly.

This problem does not appear when changing

  ProxyPass "/" "h2c://localhost:8080/"

to

  ProxyPass "/" "http://localhost:8080/"

so it only seems to appear when using the HTTP/2 backend. My best guess is that one of the changes that were introduced to address the security vulnerabilities also introduced a race condition that is only triggered when using HTTP/2 that is more parallel in nature.

This seems to be corroborated by my observation that this problem seems to appear more frequently when many requests are forwarded in parallel or at least quick succession.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: apache2 2.4.52-1ubuntu4.10
ProcVersionSignature: Ubuntu 5.15.0-113.123-generic 5.15.152
Uname: Linux 5.15.0-113-generic x86_64
Apache2ConfdDirListing: False
Apache2Modules: httpd (pid 580607) already running
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: unknown
Date: Wed Jul 10 14:25:29 2024
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: apache2
UpgradeStatus: Upgraded to jammy on 2024-01-05 (187 days ago)

[Marc Deslauriers (mdeslaur)]

Is this all your are getting? "Reason: URI has no hostname: /…"
Or did you edit that to remove sensitive info?

[Marc Deslauriers (mdeslaur)]

This is a stab in the dark but perhaps this is part of the problem:

https://github.com/apache/httpd/commit/4d3a308014be26e5407113b4c827a1ea2882bf38

Would you be willing to try a test package if I build one?

[Sebastian Marsching (sebastian-marsching)]

Thanks for picking this up.

I redacted the URI because some of the examples contained sensitive data and the the value depends on the actual request, but it always seems to be the actual request URI including the query string. Here are a few examples that do not contain sensitive data:

URI has no hostname: /?module=Login
URI has no hostname: /index.php?date=yesterday&module=API&method=API.getWidgetMetadata&filter_limit=-1&format=JSON&deep=1&idSite=1&period=day
URI has no hostname: /index.php?date=yesterday&module=API&format=json&method=API.getReportPagesMetadata&filter_limit=-1&segment=&idSite=1&period=day

If you send me a link to packages with a potential fix, I am happy to test them. The problem appears so frequently that I can easily reproduce it within a few minutes.

[Marc Deslauriers (mdeslaur)]

I have uploaded a test package to the security team PPA here:

https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+packages

If you could give it a spin and see if it fixes the issue, that would be great. If it does, I'll release it as a regression update.

Thanks!

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sebastian Marsching (sebastian-marsching)]

I can confirm that the package that you provided fixes the issue for me.

Thank you very much for the quick help!

Your intuition seems to better than mine, because I have to admit that it took me a while how the commit that you linked fixes the issue and how it was previously introduced by the security fix.

For those readers who are interested and do not want to read all the relevant sections of the source code themselves, here is the explanation:

The patch for CVE-2024-38477 introduced a check that verifies that the URI contains a hostname. As long as this check was missing, a dormant bug in proxy_http2_handler that in case of a retry caused ap_proxy_determine_connection to be called for a URL that had already been processed by this function, did not cause any immediate problems (I guess the URL got somewhat mangled, but this did not matter because the affected parts of were not used after that). With the introduction of the check of the hostname, this bug suddenly became visible.

As this bug was in mod_proxy_http2, proxy connections to HTTP/1 servers were not affected, which lead to this funny effect that HTTP/2 connections occassionally failed while HTTP/1 connections were unaffected, even though the security patch itself did not make any changes to the HTTP/2 logic.

[Marc Deslauriers (mdeslaur)]

Awesome, thanks for testing it, I'll prepare updated packages to release ASAP!

That's an excellent write-up of what is happening here, and is why I suspected that commit was the cause.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.58-1ubuntu8.3

---------------
apache2 (2.4.58-1ubuntu8.3) noble-security; urgency=medium

  * SECURITY REGRESSION: regression when proxying http2 (LP: #2072648)
    - debian/patches/CVE-2024-38477-2.patch: restart from the original URL
      on reconnect in modules/http2/mod_proxy_http2.c.

 -- Marc Deslauriers <email address hidden> Thu, 11 Jul 2024 10:41:54 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.41-4ubuntu3.20

---------------
apache2 (2.4.41-4ubuntu3.20) focal-security; urgency=medium

  * SECURITY REGRESSION: regression when proxying http2 (LP: #2072648)
    - debian/patches/CVE-2024-38477-2.patch: restart from the original URL
      on reconnect in modules/http2/mod_proxy_http2.c.

 -- Marc Deslauriers <email address hidden> Thu, 11 Jul 2024 10:39:23 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.52-1ubuntu4.11

---------------
apache2 (2.4.52-1ubuntu4.11) jammy-security; urgency=medium

  * SECURITY REGRESSION: regression when proxying http2 (LP: #2072648)
    - debian/patches/CVE-2024-38477-2.patch: restart from the original URL
      on reconnect in modules/http2/mod_proxy_http2.c.

 -- Marc Deslauriers <email address hidden> Thu, 11 Jul 2024 08:20:46 -0400

[Marc Deslauriers (mdeslaur)]

Regression fix USN has now been published:

https://ubuntu.com/security/notices/USN-6885-2

Thanks!