functionality loss in mod_proxy rewritten path

[Daniele Gozzi]

= ubuntu release =

Description: Ubuntu 22.04.3 LTS
Release: 22.04

= package version =

2.4.52-1ubuntu4.11

= what I expected to happen =

I have a RewriteRule as such
  ErrorDocument 503 /cgi-bin/503_cgi/503_cgi.py
  RewriteRule ^/cgi-bin/503_cgi/503_cgi.py$ "unix:/run/instance_starter-cgi-instance_starter/socket|fcgi:/home/instance_starter/503_cgi/503_cgi.py" [QSA,P,L,NE]
  RewriteCond %{HTTP_HOST} ^([^.]+)\.${APACHE_RP_DOMAIN}(:[0-9]+)?$
  RewriteRule ^(.+) %{HTTP_HOST}$1 [C]
  RewriteRule ^([^.]+)\.${APACHE_RP_DOMAIN}(:[0-9]+)?(.*)$ "http://$1:8000$3" [P,L]

(/run/instance_starter-cgi-instance_starter/socket being the listening socket of a FastCGI daemon)

I expect that setup to serve a specific FastCGI script in case of a 503 error, proxying to an external HTTP server normally otherwise. The name of the proxied HTTP server is taken as the first part of the hostname in the original request, separating that with dots.

= what happened instead =

After the last security updates this configuration is no longer working. It is instead logging in apache:
AH01083: error parsing URL http://: Invalid host/port

I think I can actually pinpoint the release that introduced the bug as 2.4.52-1ubuntu4.10 , based on my logs.

Actually, there are probably two incompatibilities with this configuration:

Incompatibility 1. The last RewriteRule is the first one producing the AH01083 error and it had to be rewritten with a leading slash on the search pattern as in
  RewriteRule ^/([^.]+)\.${APACHE_RP_DOMAIN}(:[0-9]+)?(.*)$ "http://$1:8000$3" [P,L]

which is less than optimal given my aim to have the hostname as part of the pattern matching that I could use in the rewritten path - the slash is preposterous but I could live with that, as it works.

Incompatibility 2. The first RewriteRule that uses unix: as the start of the rewritten path

I had not been able to ever use mod_proxy_fcgi in this way again. Even changing The RewriteRule as I did in several other mod_proxy_http setups, the path was only reachable through the canonical request path and would not trigger as an effect of a 503 error.

Just for context I'm explaining a bit the mod_proxy_http changes I had to perform as a consequence of the same security patches.

I have several other apache web servers which use an internal web application server with a RewriteRule as such:
  RewriteRule ^(/.*)$ "proxy:unix:/run/username-gunicorn-username/socket|http://%{HTTP_HOST}$1" [QSA,L,NE]

They all had to be rewritten as
  RewriteRule ^(/.*)$ "unix:/run/username-gunicorn-username/socket|http://%{HTTP_HOST}$1" [QSA,L,NE,P]

And I kind of see the point. I could also live with that. But a similar bug in the same package bit us two years ago and I would really appreciate some regression tests on the security fixes applied with the Canonical backports.
As far as I can tell, my configuration was not inherently vulnerable so the incident could be avoided having tests which cover similar integrations with external HTTP/FastCGI server server processes on Unix sockets.

This problem also affects the backports for Ubuntu 20.04.

[Marc Deslauriers (mdeslaur)]

Thanks for reporting these issues. There are been a few regression fixes upstream following the security updates from their 2.4.60, 2.4.61 and 2.4.62 releases. I will attempt to reproduce your issue and see if it has been fixed upstream yet.

[Marc Deslauriers (mdeslaur)]

I'm having difficulty reproducing your exact issue. Could you please run apache with "rewrite:trace3" and show working and non-working requests from error.log? Thanks.

[Daniele Gozzi (gozzid)]

Sure. No problem. This should show how adding a leading slash is breaking the last RewriteRule. It would be harder to reproduce Incompatibility #2 as I don't have the FCGI daemon any more, as I'm using an HTTP application server now instead. However, if you need, I could circumvent the #1 incompatibility adding a leading slash to the last RewriteRule to see what happens with the mod_proxy_fcgi RewriteRule.

However, as for now, incompatibility #1 (domain redacted with "example.com"):

== Non-working ==
Configuration for reference:
  RewriteEngine On
  RewriteRule ^/cgi-bin/503_cgi/503_cgi.py$ "unix:/run/instance_starter-cgi-instance_starter/socket|fcgi:/home/instance_starter/503_cgi/503_cgi.py" [QSA,P,L,NE]
  RewriteCond %{HTTP_HOST} ^([^.]+)\.${APACHE_RP_DOMAIN}(:[0-9]+)?$
  RewriteRule ^(.+) %{HTTP_HOST}$1 [C]
  RewriteRule ^([^.]+)\.${APACHE_RP_DOMAIN}(:[0-9]+)?(.*)$ "http://$1:8000$3" [P,L]

Apache error log with LogLevel rewrite:trace3:

[Mon Jul 22 14:22:33.089889 2024] [rewrite:trace2] [pid 122148:tid 140373984781888] mod_rewrite.c(491): [client 192.168.80.12:52434] 192.168.80.12 - - [asd.example.com/sid#7fab670e8940][rid#7fab672330a0/initial] init rewrite engine with requested uri /
[Mon Jul 22 14:22:33.089979 2024] [rewrite:trace3] [pid 122148:tid 140373984781888] mod_rewrite.c(491): [client 192.168.80.12:52434] 192.168.80.12 - - [asd.example.com/sid#7fab670e8940][rid#7fab672330a0/initial] applying pattern '^/cgi-bin/503_cgi/503_cgi.py$' to uri '/'
[Mon Jul 22 14:22:33.089995 2024] [rewrite:trace3] [pid 122148:tid 140373984781888] mod_rewrite.c(491): [client 192.168.80.12:52434] 192.168.80.12 - - [asd.example.com/sid#7fab670e8940][rid#7fab672330a0/initial] applying pattern '^(.+)' to uri '/'
[Mon Jul 22 14:22:33.090014 2024] [rewrite:trace2] [pid 122148:tid 140373984781888] mod_rewrite.c(491): [client 192.168.80.12:52434] 192.168.80.12 - - [asd.example.com/sid#7fab670e8940][rid#7fab672330a0/initial] rewrite '/' -> 'asd.example.com/'
[Mon Jul 22 14:22:33.090021 2024] [rewrite:trace3] [pid 122148:tid 140373984781888] mod_rewrite.c(491): [client 192.168.80.12:52434] 192.168.80.12 - - [asd.example.com/sid#7fab670e8940][rid#7fab672330a0/initial] add root prefix: asd.example.com/ -> /asd.example.com/
[Mon Jul 22 14:22:33.090027 2024] [rewrite:trace2] [pid 122148:tid 140373984781888] mod_rewrite.c(491): [client 192.168.80.12:52434] 192.168.80.12 - - [asd.example.com/sid#7fab670e8940][rid#7fab672330a0/initial] setting lastsub to rule with output %{HTTP_HOST}$1
[Mon Jul 22 14:22:33.090033 2024] [rewrite:trace3] [pid 122148:tid 140373984781888] mod_rewrite.c(491): [client 192.168.80.12:52434] 192.168.80.12 - - [asd.example.com/sid#7fab670e8940][rid#7fab672330a0/initial] applying pattern '^([^.]+)\\.example.com(:[0-9]+)?(.*)$' to uri '/asd.example.com/'
[Mon Jul 22 14:22:33.090043 2024] [rewrite:trace2] [pid 122148:tid 140373984781888] mod_rewrite.c(491): [client 192.168.80.12:52434] 192.168.80.12 - - [asd.example.com/sid#7fab670e8940][rid#7fab672330a0/initial] rewrite '/asd.example.com/' -> 'http:///asd:8000/'
[Mon Jul 22 14:22:33.090052 2024] [rewrite:trace2] [pid 122148:tid 140373984781888] mod_rewrite.c(491): [client 192.168.80.12:52434] 19...

[Marc Deslauriers (mdeslaur)]

Thanks, that is very helpful, so we're getting an extra slash here:

rewrite '/asd.example.com/' -> 'http:///asd:8000/'

instead of:

rewrite '/asd.example.com/' -> 'http://asd:8000/'

and I can confirm this issue exists with upstream's 2.4.62 release too, so I will keep an eye out for new fixes they have been adding post-release.

[Daniele Gozzi (gozzid)]

Yes, that is the first problem we are getting. It is probably introduced by the "add root prefix: asd.example.com/ -> /asd.example.com/" step.

The triple slash is introduced by my last RewriteRule which does not account for a leading slash (RewriteRule ^([^.]+)).

My use case with the RewriteRule chaining ([C] flag) that puts the hostname in front of the path to be rewritten is possibly a vulnerable configuration if the HTTP Host header is put together as concatenating a first URL, newlines then the usual hostname. It could be one of the HTTP response splitting vulnerabilities addressed by the upstream patches. That is why I'm redacting the actual hostname.