Apache's mod_remoteip: IP address spoofing via X-Forwarded-For when mod_rewrite rule is triggered
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| apache2 (Debian) |
New
|
Undecided
|
Unassigned | |
| apache2 (Ubuntu) |
Undecided
|
Unassigned | ||
| Xenial |
Undecided
|
Unassigned |
Bug Description
[Impact]
* remoteip/
* Fix by backporting an upstream change added in 2.4.24 and later (was not
changed since the fix).
* The fix is small and only changes behavior in a very special case that
formerly was broken (if there was a useragent_addr on re-processing).
For other cases the behavior is unchanged.
[Test Case]
$ apt install apache2 libapache2-mod-php
define /etc/apache2/
<VirtualHost *:8080>
<IfModule mod_remoteip.c>
</IfModule>
<Directory /var/www/html>
</VirtualHost>
In File /etc/apache2/
Listen 80
to
Listen 8080
$ sudo a2enmod rewrite
$ sudo a2enmod remoteip
$ sudo a2enmod php7.0
$ systemctl restart apache2
$ apt install nginx
define file /etc/nginx/
server {
listen 80 default_server;
root /var/www/html;
server_name _;
location / {
}
}
$ systemctl restart nginx
define file /var/www/
<?php
echo $_SERVER[
?>
You need to do the following from a second host in the same network, do not use "localhost" as there it can't differentiate the processing no matter if fixed or not.
$ curl http://<IP>/index.php
127.0.0.1
$ curl http://<IP>/seo-
127.0.0.1
$ curl http://<IP>/seo-
1.1.1.1
Expected:
- the last one should not be the fake 1.1.1.1
- if you do this from a remote host it should show the remote IP for all three.
[Regression Potential]
* If remote_ip isn' enabled (the common case) the change should be a no-
op. It only is important when using remote_ip and processing things
twice e.g. on an error handler. There due to an issue it allowed to fake
the RemoteIP. Fixing that should fix the issue, but not break other
things - if anywhere then remoteIP handling would be the one expected to
see regressions of any sort, but most likely only if people started to
rely on the bad behavior.
[Other Info]
* one can debate if this is a security issue (crafting of wrong origin
logs) or not but I'll leave that to other people.
---
There is a bug in mod_remoteip (a part of Apache Web Server): https:/
Although the status of this bug is "NEW", actually it was fixed in Apache 2.4.24.
Although a CVE id was not requested yet, actually it is a vulnerability.
The fix was not backported to Ubuntu 16.04 (xenial).
Impact: if a victim uses Apache rewrite rules, then an attacker can spoof his IP address for logs and PHP scripts.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: apache2 2.4.18-2ubuntu3.14
ProcVersionSign
Uname: Linux 4.4.0-22-generic x86_64
Apache2ConfdDir
ApportVersion: 2.20.1-0ubuntu2.23
Architecture: amd64
Date: Mon Apr 27 13:17:43 2020
SourcePackage: apache2
UpgradeStatus: No upgrade log present (probably fresh install)
error.log:
modified.
modified.
modified.
modified.
modified.
modified.
mtime.conffile.
mtime.conffile.
mtime.conffile.
mtime.conffile.
mtime.conffile.
mtime.conffile.
Related branches
- Rafael David Tinoco: Approve on 2020-06-16
- Canonical Server Team: Pending requested 2020-06-15
- Canonical Server packageset reviewers: Pending requested 2020-06-15
-
Diff: 79 lines (+57/-0)3 files modifieddebian/changelog (+7/-0)
debian/patches/lp-1875299-Merge-r1688399-from-trunk.patch (+49/-0)
debian/patches/series (+1/-0)
Andrey (iqlyf) wrote : | #1 |
summary: |
- Apache's mod_repoteip: IP spoofing via X-Forwarded-For when mod_rewrite + Apache's mod_remoteip: IP spoofing via X-Forwarded-For when mod_rewrite rule is triggered |
summary: |
- Apache's mod_remoteip: IP spoofing via X-Forwarded-For when mod_rewrite - rule is triggered + Apache's mod_remoteip: IP address spoofing via X-Forwarded-For when + mod_rewrite rule is triggered |
information type: | Private Security → Public Security |
Alex Murray (alexmurray) wrote : | #2 |
Robie Basak (racb) wrote : | #3 |
Thank you for taking the time to report this bug and helping to make Ubuntu better.
Next steps:
1) We need to check if this problem is fixed in the current development release of Ubuntu, and if a fix is needed in any other stable releases.
2) We need a step-by-step test case to reproduce the problem.
If you could help with either of these, this would be appreciated and help us land a fix.
tags: | added: server-next |
Changed in apache2 (Ubuntu Xenial): | |
status: | New → Triaged |
Robie Basak (racb) wrote : | #4 |
(alternatively if you get a CVE then it can follow the separate security update process based on Alex's comment)
Launchpad Janitor (janitor) wrote : | #5 |
Status changed to 'Confirmed' because the bug affects multiple users.
Changed in apache2 (Ubuntu): | |
status: | New → Confirmed |
Marcus (pathmissing) wrote : | #6 |
Hello Robie. I originally reported this issue to Andrey. I will attempt to provide some additional information for reproducing this bug.
As already stated by Andrey, this issue affects apache versions prior to 2.4.24 and therefore distributions like ubuntu 16.04 and debian 8 seem to be vulnerable.
In order to reproduce this issue, a simple setup with apache2 as backend server and nginx as reverse proxy is sufficcient. I will provide the example configuration files.
The apache2 modules remoteip, rewrite and php have to be enabled by hand.
Marcus (pathmissing) wrote : | #7 |
Marcus (pathmissing) wrote : | #8 |
Marcus (pathmissing) wrote : | #9 |
After configuring nginx and apache, the file index.php has to be placed in the document root directory (/var/www/html). This PHP file outputs the value of the variable $_SERVER[
In order to keep this simple, the following IPv4 addresses will be used:
Server IP: 192.168.1.1
Client IP: 192.168.2.2
Now our client wants to access our site by using the following command:
curl http://
Output: 192.168.2.2 (As expected, the real IP address of the requesting client)
Let's assume the client accesses a SEO friendly URL and should be internally redirected to the file index.php. To test this, the following curl command can be used:
curl http://
Output: 192.168.2.2 (As expected)
So far so good. Let's assume our client provides the header X-Forwarded-For with an arbitrary IP address:
curl http://
Output: 1.1.1.1 (Unexpected; should contain the real IP address instead of the Header value)
I hope the supplied configuration files and this short explanation can be of help. I am happy to provide additional information if needed.
Changed in apache2 (Ubuntu): | |
status: | Confirmed → Fix Released |
Christian Ehrhardt (paelzer) wrote : | #10 |
Hi Marcus,
I was following your howto step by step and wanted to thank you already to provide that much details.
I was trying to simplify it further to not reach out to extra files, adding the apt install steps and everything else.
$ apt install apache2 libapache2-mod-php
define /etc/apache2/
<VirtualHost *:8080>
<IfModule mod_remoteip.c>
</IfModule>
<Directory /var/www/html>
</VirtualHost>
In File /etc/apache2/
Listen 80
to
Listen 8080
$ sudo a2enmod rewrite
$ a2enmod php7.0
$ systemctl restart apache2
$ apt install nginx
define file /etc/nginx/
server {
listen 80 default_server;
root /var/www/html;
server_name _;
location / {
}
}
$ systemctl restart nginx
define file /var/www/
<?php
echo $_SERVER[
?>
$ curl http://
127.0.0.1
$ curl http://
127.0.0.1
$ curl http://
127.0.0.1
It doesn't seem to matter if I do the curl from localhost, or the same curl from another system onto this `curl 10.253.194.202/foo -H "X-Forwarded-For: 1.1.1.1"`. In my cases I never got the original client IP, nor the Faked IP - I always got the one of the nginx server.
Do you spot a difference between my approach above and your setup that would explain?
Changed in apache2 (Ubuntu Xenial): | |
status: | Triaged → Incomplete |
Marcus (pathmissing) wrote : | #11 |
Hi Christian,
it looks like you took the excact setup steps as I did. But just to make sure: Have you enabled the apache2 module remote_ip?
$ a2enmod remoteip
mod_rewrite seems to be enabled, otherwise your curl commands would not return the output of the PHP file.
Other than that everything looks similar
Regards
Christian Ehrhardt (paelzer) wrote : | #12 |
Here you go, that was it.
I had not remoteip enabled.
Added to my howto (which will be part of the SRU template).
description: | updated |
description: | updated |
Christian Ehrhardt (paelzer) wrote : | #13 |
Updated SRU template of the bug and thrown a test build into PPA [1]
An MP with the proposed changes is at [2].
[1]: https:/
[2]: https:/
Christian Ehrhardt (paelzer) wrote : | #14 |
The PPA build with the fix as referenced by the upstream bug did not fix it in my tests :-/
Anyone up to try this as well if it is local to me?
Marcus (pathmissing) wrote : | #15 |
I could test it in the evening. I assume I'd just have to add the PPA source, apt update and reinstall apache2?
Marcus (pathmissing) wrote : | #16 |
Just tested it and the issue still exists. I added the repository, reinstalled apache2 and restartet the service but still had the same outcome.
apache2/xenial,now 2.4.18-
Christian Ehrhardt (paelzer) wrote : | #17 |
Ok, that means we need more than "just" the fix to https:/
Probably something else that was between 2.4.18 and 2.4.24 ...
The only other change to remoteip itself was [1], but that doesn't seem to be what we miss.
This issue seemed nice - and I wanted to help to drive it to conclusion even thou it didn't seem too severe - as the fix was already identified, but right not this will need deeper analysis what fix(es) we actually need :-/
@Marcus - if you have more suggestions which might fix it I'm happy to re-build a PPA for us to test.
[1]: https:/
Marcus (pathmissing) wrote : | #18 |
Sadly, I don't have any other ideas at the moment.
But I do think that this issue does pose a security risk that should not be overlooked. Applications relying in any form on the source IP address for authentication or rate limitations might be affected.
I will have another look at the sources later on but I am not familiar with the remoteip codebase so I would not expect too much from that.
Christian Ehrhardt (paelzer) wrote : | #19 |
2.4.24 30060fb18dcbb18
2.4.18 b82d3c64494b7a5
Notes to not-nice-
$ git clean -x -f -d
$ wget https:/
$ wget https:/
$ tar -xf /tmp/apr.tgz -C srclib/
$ tar -xf /tmp/apru.tgz -C srclib/
$ mv srclib/apr-1.7.0 srclib/apr
$ mv srclib/
$ ./buildconf
$ ./configure --enable-
$ printf "all:\n\techo yeah\ninstall:
$ make
$ make install
$ md5sum /usr/lib/
$ systemctl restart apache2
2.2.24
a956ffb3146d4d5
3da6c1a5cd722a5
d1bdb21f7aaf986
-rwxr-xr-x 1 root root 18288 Jun 16 08:04 /usr/lib/
-rwxr-xr-x 1 root root 96384 Jun 16 08:05 /usr/lib/
-rwxr-xr-x 1 root root 933200 Jun 16 08:05 /usr/sbin/apache2*
=> works
2.2.18
8bbf7be2a5a7180
bbb2b45e6c1da8e
2877029808bfd25
-rwxr-xr-x 1 root root 18288 Jun 16 08:28 /usr/lib/
-rwxr-xr-x 1 root root 96264 Jun 16 08:28 /usr/lib/
-rwxr-xr-x 1 root root 898528 Jun 16 08:28 /usr/sbin/apache2*
=> Fails
This already looked like git-bisect land, when on testing these builds I found that the local test:
$ curl http://
can still be fooled
while at the same time any remote access like
$ curl 10.253.194.202/foo -H "X-Forwarded-For: 1.1.1.1"
responds to the fix as expected.
Once that was learned I can say that my fix in the PPA actually works, the only change is that the curl test has to be from another host.
@Marcus - can you confirm that in your testing?
description: | updated |
Marcus (pathmissing) wrote : | #20 |
@Christian I can confirm, it works as expected for external requests.
Christian Ehrhardt (paelzer) wrote : | #21 |
Uploaded to xenial-unapproved for the SRU Team to consider
Marcus (pathmissing) wrote : | #22 |
Glad to hear! Just out of curiosity: What will be the next steps? Should this issue also be reported to the debian bug tracker because it affects debain 8? If I remember correctly, the support for debian 8 should end this month.
And also: Should there be a discussion about the security impact? If so, where would be a good place to start?
Thanks!
Christian Ehrhardt (paelzer) wrote : | #23 |
The security Team is subscribed here, they would know where to discuss security.
But I guess Alex has given the answer in [1] already.
Next steps (for you to know) are following the SRU process [2].
TL;DR:
1. the SRU Team reviews my upload and approves/rejects
2. it will build for xenial-proposed
3. regression tests will run there
4. we need to verify the fix really works as expected
5. some maturing time needs to pass
6. the update is releases into xenial-updates
[1]: https:/
[2]: https:/
Robie Basak (racb) wrote : | #24 |
My understanding of Alex's suggestion in comment 2 is that upstream don't consider this to be a security vulnerability and in Ubuntu the security team doesn't see a reason to diverge from that opinion. So we'll treat this as a non-security fix for now and follow the process for a regular bugfix.
Note that this means that users who opt for security updates only will not receive this fix.
If this position changes (for example if you convince upstream that it is a security issue and a CVE is warranted) then the Ubuntu security team can always rebuild and push the fix to the security pocket later, to also give the fix to users opting for security updates only.
Changed in apache2 (Ubuntu Xenial): | |
status: | Incomplete → Fix Committed |
tags: | added: verification-needed verification-needed-xenial |
Hello Andrey, or anyone else affected,
Accepted apache2 into xenial-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
Marcus (pathmissing) wrote : | #26 |
Hello Robie,
I just had a look at the proposed package and it fixes the issue for me.
To test the patch I followed the steps as explained in the bug description, first with apache2/
$ curl 192.168.1.129/test -H 'X-Forwarded-For: 1.1.1.1'
> 1.1.1.1
After upgrading the package to apache2/
$ curl 192.168.1.129/test -H 'X-Forwarded-For: 1.1.1.1'
> 192.168.1.100
Regards
Christian Ehrhardt (paelzer) wrote : | #27 |
Thanks for the test, marking as verified
tags: |
added: verification-done verification-done-xenial removed: verification-needed verification-needed-xenial |
Launchpad Janitor (janitor) wrote : | #28 |
This bug was fixed in the package apache2 - 2.4.18-2ubuntu3.15
---------------
apache2 (2.4.18-
* d/p/lp-
the root trusted address (LP: #1875299)
-- Christian Ehrhardt <email address hidden> Mon, 15 Jun 2020 16:09:55 +0200
Changed in apache2 (Ubuntu Xenial): | |
status: | Fix Committed → Fix Released |
The verification of the Stable Release Update for apache2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.
If you feel this is a real security vulnerability which has not received a CVE, you could try discuss with the apache developers and once a CVE has been assigned the Ubuntu Security team can fix it via a security update for Apache. Otherwise, this could be addressed via the Stable Release Update (SRU) process otherwise - https:/ /wiki.ubuntu. com/StableRelea seUpdates