Apache's mod_remoteip: IP address spoofing via X-Forwarded-For when mod_rewrite rule is triggered

[Andrey]

[Impact]

 * remoteip/mod_rewrite and a proxy might allow faking the source URL

 * Fix by backporting an upstream change added in 2.4.24 and later (was not
   changed since the fix).

 * The fix is small and only changes behavior in a very special case that
   formerly was broken (if there was a useragent_addr on re-processing).
   For other cases the behavior is unchanged.

[Test Case]

$ apt install apache2 libapache2-mod-php

define /etc/apache2/sites-enabled/000-default.conf as:
<VirtualHost *:8080>
        DocumentRoot /var/www/html

        <IfModule mod_remoteip.c>
                RemoteIPInternalProxy 127.0.0.1
                RemoteIPHeader X-Forwarded-For
        </IfModule>

        <Directory /var/www/html>
                RewriteEngine On
                RewriteRule .* index.php [L,QSA]
        </Directory>
</VirtualHost>

In File /etc/apache2/ports.conf change
Listen 80
to
Listen 8080

$ sudo a2enmod rewrite
$ sudo a2enmod remoteip
$ sudo a2enmod php7.0
$ systemctl restart apache2

$ apt install nginx

define file /etc/nginx/sites-enabled/default as:
server {
        listen 80 default_server;
        root /var/www/html;
        server_name _;

        location / {
                proxy_pass http://127.0.0.1:8080;
                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
}

$ systemctl restart nginx

define file /var/www/html/index.php as:
<?php
echo $_SERVER['REMOTE_ADDR'] . "\n";
?>

You need to do the following from a second host in the same network, do not use "localhost" as there it can't differentiate the processing no matter if fixed or not.

$ curl http://<IP>/index.php
127.0.0.1
$ curl http://<IP>/seo-friendly-url
127.0.0.1
$ curl http://<IP>/seo-friendly-url -H "X-Forwarded-For: 1.1.1.1"
1.1.1.1

Expected:
- the last one should not be the fake 1.1.1.1
- if you do this from a remote host it should show the remote IP for all three.

[Regression Potential]

 * If remote_ip isn' enabled (the common case) the change should be a no-
   op. It only is important when using remote_ip and processing things
   twice e.g. on an error handler. There due to an issue it allowed to fake
   the RemoteIP. Fixing that should fix the issue, but not break other
   things - if anywhere then remoteIP handling would be the one expected to
   see regressions of any sort, but most likely only if people started to
   rely on the bad behavior.

[Other Info]

 * one can debate if this is a security issue (crafting of wrong origin
   logs) or not but I'll leave that to other people.

---

There is a bug in mod_remoteip (a part of Apache Web Server): https://bz.apache.org/bugzilla/show_bug.cgi?id=60251
Although the status of this bug is "NEW", actually it was fixed in Apache 2.4.24.
Although a CVE id was not requested yet, actually it is a vulnerability.

The fix was not backported to Ubuntu 16.04 (xenial).

Impact: if a victim uses Apache rewrite rules, then an attacker can spoof his IP address for logs and PHP scripts.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: apache2 2.4.18-2ubuntu3.14
ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
Apache2ConfdDirListing: False
ApportVersion: 2.20.1-0ubuntu2.23
Architecture: amd64
Date: Mon Apr 27 13:17:43 2020
SourcePackage: apache2
UpgradeStatus: No upgrade log present (probably fresh install)
error.log:

modified.conffile..etc.apache2.apache2.conf: [modified]
modified.conffile..etc.apache2.mods-available.dir.conf: [modified]
modified.conffile..etc.apache2.mods-available.ssl.conf: [modified]
modified.conffile..etc.apache2.ports.conf: [modified]
modified.conffile..etc.apache2.sites-available.000-default.conf: [modified]
modified.conffile..etc.apache2.sites-available.default-ssl.conf: [modified]
mtime.conffile..etc.apache2.apache2.conf: 2020-04-23T15:45:48.416970
mtime.conffile..etc.apache2.mods-available.dir.conf: 2020-04-23T12:03:13.711062
mtime.conffile..etc.apache2.mods-available.ssl.conf: 2020-04-23T12:02:44.854484
mtime.conffile..etc.apache2.ports.conf: 2020-04-23T15:45:48.169037
mtime.conffile..etc.apache2.sites-available.000-default.conf: 2020-04-23T15:45:48.197030
mtime.conffile..etc.apache2.sites-available.default-ssl.conf: 2020-04-23T15:45:48.225022

[Alex Murray (alexmurray)]

If you feel this is a real security vulnerability which has not received a CVE, you could try discuss with the apache developers and once a CVE has been assigned the Ubuntu Security team can fix it via a security update for Apache. Otherwise, this could be addressed via the Stable Release Update (SRU) process otherwise - https://wiki.ubuntu.com/StableReleaseUpdates

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Next steps:

1) We need to check if this problem is fixed in the current development release of Ubuntu, and if a fix is needed in any other stable releases.

2) We need a step-by-step test case to reproduce the problem.

If you could help with either of these, this would be appreciated and help us land a fix.

[Robie Basak (racb)]

(alternatively if you get a CVE then it can follow the separate security update process based on Alex's comment)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Marcus (pathmissing)]

Hello Robie. I originally reported this issue to Andrey. I will attempt to provide some additional information for reproducing this bug.

As already stated by Andrey, this issue affects apache versions prior to 2.4.24 and therefore distributions like ubuntu 16.04 and debian 8 seem to be vulnerable.

In order to reproduce this issue, a simple setup with apache2 as backend server and nginx as reverse proxy is sufficcient. I will provide the example configuration files.

The apache2 modules remoteip, rewrite and php have to be enabled by hand.

[Marcus (pathmissing)]

After configuring nginx and apache, the file index.php has to be placed in the document root directory (/var/www/html). This PHP file outputs the value of the variable $_SERVER['REMOTE_ADDR'] which should always carry the client's real IP address and should always contain trustworthy values when being processed by a webserver.

In order to keep this simple, the following IPv4 addresses will be used:

Server IP: 192.168.1.1
Client IP: 192.168.2.2

Now our client wants to access our site by using the following command:

curl http://192.168.1.1/index.php
Output: 192.168.2.2 (As expected, the real IP address of the requesting client)

Let's assume the client accesses a SEO friendly URL and should be internally redirected to the file index.php. To test this, the following curl command can be used:

curl http://192.168.1.1/seo-friendly-url/
Output: 192.168.2.2 (As expected)

So far so good. Let's assume our client provides the header X-Forwarded-For with an arbitrary IP address:

curl http://192.168.1.1/seo-friendly-url/ -H "X-Forwarded-For: 1.1.1.1"
Output: 1.1.1.1 (Unexpected; should contain the real IP address instead of the Header value)

I hope the supplied configuration files and this short explanation can be of help. I am happy to provide additional information if needed.

[Christian Ehrhardt  (paelzer)]

Hi Marcus,
I was following your howto step by step and wanted to thank you already to provide that much details.
I was trying to simplify it further to not reach out to extra files, adding the apt install steps and everything else.

$ apt install apache2 libapache2-mod-php

define /etc/apache2/sites-enabled/000-default.conf as:
<VirtualHost *:8080>
        DocumentRoot /var/www/html

        <IfModule mod_remoteip.c>
                RemoteIPInternalProxy 127.0.0.1
                RemoteIPHeader X-Forwarded-For
        </IfModule>

        <Directory /var/www/html>
                RewriteEngine On
                RewriteRule .* index.php [L,QSA]
        </Directory>
</VirtualHost>

In File /etc/apache2/ports.conf change
Listen 80
to
Listen 8080

$ sudo a2enmod rewrite
$ a2enmod php7.0
$ systemctl restart apache2

$ apt install nginx

define file /etc/nginx/sites-enabled/default as:
server {
        listen 80 default_server;
        root /var/www/html;
        server_name _;

        location / {
                proxy_pass http://127.0.0.1:8080;
                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
}

$ systemctl restart nginx

define file /var/www/html/index.php as:
<?php
echo $_SERVER['REMOTE_ADDR'] . "\n";
?>

$ curl http://localhost/index.php
127.0.0.1
$ curl http://localhost/seo-friendly-url
127.0.0.1
$ curl http://localhost/seo-friendly-url -H "X-Forwarded-For: 1.1.1.1"
127.0.0.1

It doesn't seem to matter if I do the curl from localhost, or the same curl from another system onto this `curl 10.253.194.202/foo -H "X-Forwarded-For: 1.1.1.1"`. In my cases I never got the original client IP, nor the Faked IP - I always got the one of the nginx server.

Do you spot a difference between my approach above and your setup that would explain?

[Marcus (pathmissing)]

Hi Christian,

it looks like you took the excact setup steps as I did. But just to make sure: Have you enabled the apache2 module remote_ip?

$ a2enmod remoteip

mod_rewrite seems to be enabled, otherwise your curl commands would not return the output of the PHP file.
Other than that everything looks similar

Regards

[Christian Ehrhardt  (paelzer)]

Here you go, that was it.
I had not remoteip enabled.
Added to my howto (which will be part of the SRU template).

[Christian Ehrhardt  (paelzer)]

Updated SRU template of the bug and thrown a test build into PPA [1]
An MP with the proposed changes is at [2].

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4098/
[2]: https://code.launchpad.net/~paelzer/ubuntu/+source/apache2/+git/apache2/+merge/385748

[Christian Ehrhardt  (paelzer)]

The PPA build with the fix as referenced by the upstream bug did not fix it in my tests :-/
Anyone up to try this as well if it is local to me?

[Marcus (pathmissing)]

I could test it in the evening. I assume I'd just have to add the PPA source, apt update and reinstall apache2?

[Marcus (pathmissing)]

Just tested it and the issue still exists. I added the repository, reinstalled apache2 and restartet the service but still had the same outcome.

apache2/xenial,now 2.4.18-2ubuntu3.15~ppa1 amd64 [installed]

[Christian Ehrhardt  (paelzer)]

Ok, that means we need more than "just" the fix to https://bz.apache.org/bugzilla/show_bug.cgi?id=60251.
Probably something else that was between 2.4.18 and 2.4.24 ...

The only other change to remoteip itself was [1], but that doesn't seem to be what we miss.

This issue seemed nice - and I wanted to help to drive it to conclusion even thou it didn't seem too severe - as the fix was already identified, but right not this will need deeper analysis what fix(es) we actually need :-/

@Marcus - if you have more suggestions which might fix it I'm happy to re-build a PPA for us to test.

[1]: https://github.com/apache/httpd/commit/1618a30e0d7797af2dc8f9bd18136d55c20d8f70

[Marcus (pathmissing)]

Sadly, I don't have any other ideas at the moment.

But I do think that this issue does pose a security risk that should not be overlooked. Applications relying in any form on the source IP address for authentication or rate limitations might be affected.

I will have another look at the sources later on but I am not familiar with the remoteip codebase so I would not expect too much from that.

[Christian Ehrhardt  (paelzer)]

2.4.24 30060fb18dcbb189d66bcc5a6f66f50fe7d5d3d4
2.4.18 b82d3c64494b7a59c13f03d169631177c1f6170d

Notes to not-nice-but-working build from git :-):
$ git clean -x -f -d
$ wget https://apache.mirror.digionline.de//apr/apr-util-1.6.1.tar.gz -O /tmp/apru.tgz
$ wget https://apache.mirror.digionline.de//apr/apr-1.7.0.tar.gz -O /tmp/apr.tgz
$ tar -xf /tmp/apr.tgz -C srclib/
$ tar -xf /tmp/apru.tgz -C srclib/
$ mv srclib/apr-1.7.0 srclib/apr
$ mv srclib/apr-util-1.6.1 srclib/apr-util
$ ./buildconf
$ ./configure --enable-layout=Debian --enable-so --with-program-name=apache2 --enable-suexec --with-suexec-caller=www-data --with-suexec-bin=/usr/lib/apache2/suexec --with-suexec-docroot=/var/www --with-suexec-userdir=public_html --with-suexec-logfile=/var/log/apache2/suexec.log --with-suexec-uidmin=100 --enable-suexec=shared --enable-log-config=static --with-apr=/usr/bin/apr-1-config --with-apr-util=/usr/bin/apu-1-config --with-pcre=yes --enable-pie --enable-mpms-shared=all --enable-mods-shared="all cgi ident proxy_fdpass" --enable-mods-static="unixd logio watchdog version" CFLAGS="-pipe" LDFLAGS="-Wl,--as-needed" LTFLAGS="--no-silent"
$ printf "all:\n\techo yeah\ninstall:\n\techo yeah" > srclib/Makefile
$ make
$ make install
$ md5sum /usr/lib/apache2/modules/mod_remoteip.so /usr/lib/apache2/modules/mod_rewrite.so /usr/sbin/apache2; ll /usr/lib/apache2/modules/mod_remoteip.so /usr/lib/apache2/modules/mod_rewrite.so /usr/sbin/apache2
$ systemctl restart apache2

2.2.24
a956ffb3146d4d573c9d5b558f425543 /usr/lib/apache2/modules/mod_remoteip.so
3da6c1a5cd722a5929dbcd7aec3b25da /usr/lib/apache2/modules/mod_rewrite.so
d1bdb21f7aaf9863fd50edd0f47686ab /usr/sbin/apache2
-rwxr-xr-x 1 root root 18288 Jun 16 08:04 /usr/lib/apache2/modules/mod_remoteip.so*
-rwxr-xr-x 1 root root 96384 Jun 16 08:05 /usr/lib/apache2/modules/mod_rewrite.so*
-rwxr-xr-x 1 root root 933200 Jun 16 08:05 /usr/sbin/apache2*

=> works

2.2.18
8bbf7be2a5a718045f85eb88e5cf3c98 /usr/lib/apache2/modules/mod_remoteip.so
bbb2b45e6c1da8e346b19e223fafe6c5 /usr/lib/apache2/modules/mod_rewrite.so
2877029808bfd25df927bc1068d6f4c0 /usr/sbin/apache2
-rwxr-xr-x 1 root root 18288 Jun 16 08:28 /usr/lib/apache2/modules/mod_remoteip.so*
-rwxr-xr-x 1 root root 96264 Jun 16 08:28 /usr/lib/apache2/modules/mod_rewrite.so*
-rwxr-xr-x 1 root root 898528 Jun 16 08:28 /usr/sbin/apache2*

=> Fails

This already looked like git-bisect land, when on testing these builds I found that the local test:
$ curl http://localhost/seo-friendly-url -H "X-Forwarded-For: 1.1.1.1"
can still be fooled
while at the same time any remote access like
$ curl 10.253.194.202/foo -H "X-Forwarded-For: 1.1.1.1"
responds to the fix as expected.

Once that was learned I can say that my fix in the PPA actually works, the only change is that the curl test has to be from another host.

@Marcus - can you confirm that in your testing?

[Marcus (pathmissing)]

@Christian I can confirm, it works as expected for external requests.

[Christian Ehrhardt  (paelzer)]

Uploaded to xenial-unapproved for the SRU Team to consider

[Marcus (pathmissing)]

Glad to hear! Just out of curiosity: What will be the next steps? Should this issue also be reported to the debian bug tracker because it affects debain 8? If I remember correctly, the support for debian 8 should end this month.

And also: Should there be a discussion about the security impact? If so, where would be a good place to start?

Thanks!

[Christian Ehrhardt  (paelzer)]

The security Team is subscribed here, they would know where to discuss security.
But I guess Alex has given the answer in [1] already.

Next steps (for you to know) are following the SRU process [2].
TL;DR:
1. the SRU Team reviews my upload and approves/rejects
2. it will build for xenial-proposed
3. regression tests will run there
4. we need to verify the fix really works as expected
5. some maturing time needs to pass
6. the update is releases into xenial-updates

[1]: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1875299/comments/2
[2]: https://wiki.ubuntu.com/StableReleaseUpdates

[Robie Basak (racb)]

My understanding of Alex's suggestion in comment 2 is that upstream don't consider this to be a security vulnerability and in Ubuntu the security team doesn't see a reason to diverge from that opinion. So we'll treat this as a non-security fix for now and follow the process for a regular bugfix.

Note that this means that users who opt for security updates only will not receive this fix.

If this position changes (for example if you convince upstream that it is a security issue and a CVE is warranted) then the Ubuntu security team can always rebuild and push the fix to the security pocket later, to also give the fix to users opting for security updates only.

[Robie Basak (racb)]

Hello Andrey, or anyone else affected,

Accepted apache2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.18-2ubuntu3.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Marcus (pathmissing)]

Hello Robie,

I just had a look at the proposed package and it fixes the issue for me.
To test the patch I followed the steps as explained in the bug description, first with apache2/xenial-updates,now 2.4.18-2ubuntu3.14 amd64 and the issue could be reproduced with the command

$ curl 192.168.1.129/test -H 'X-Forwarded-For: 1.1.1.1'
> 1.1.1.1

After upgrading the package to apache2/xenial-proposed,now 2.4.18-2ubuntu3.15 amd64 and running the same curl command as before, the output was the expected real IP addresss of the client.

$ curl 192.168.1.129/test -H 'X-Forwarded-For: 1.1.1.1'
> 192.168.1.100

Regards

[Christian Ehrhardt  (paelzer)]

Thanks for the test, marking as verified

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.18-2ubuntu3.15

---------------
apache2 (2.4.18-2ubuntu3.15) xenial; urgency=medium

  * d/p/lp-1875299-Merge-r1688399-from-trunk.patch: use r_useragent_addr as
    the root trusted address (LP: #1875299)

 -- Christian Ehrhardt <email address hidden> Mon, 15 Jun 2020 16:09:55 +0200

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for apache2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.