Apache2 Balancer Manager mod_proxy_balancer not working after Update
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | Apache2 Web Server |
Fix Released
|
Medium
|
||
| | apache2 (Debian) |
Fix Released
|
Unknown
|
||
| | apache2 (Ubuntu) |
Medium
|
Steve Beattie | ||
| | Xenial |
Undecided
|
Unassigned | ||
| | Bionic |
Undecided
|
Unassigned | ||
| | Disco |
Undecided
|
Unassigned | ||
Bug Description
OS
Description: Ubuntu 18.04.3 LTS
Release: 18.04
Codename: bionic
I use this kind of configuration to reache the Balancer Manager.
-------------
|Bastian Host |
|Apache Proxy | -----------> LB Apache Balancer Manger
-------------
After Apache Update
from: 2.4.29-1ubuntu4.8
to: 2.4.29-1ubuntu4.10
The Balancer Manager behind a Proxy is not Working and i think this is comming with
the fix CVE-2019-10092
https:/
http://
I strip down the configuration to try and explain the situation.
Install new Ubuntu 18.04 VirtualBox. From an another VM i saved the prior
Apache Packages from /var/cache/
:~# apt-get install libapr1 libaprutil1 libaprutil1-
:~# dpkg -i apache2_
:~# dpkg -l | grep apache2
ii apache2 2.4.29-1ubuntu4.8 amd64 Apache HTTP Server
ii apache2-bin 2.4.29-1ubuntu4.8 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.29-1ubuntu4.8 all Apache HTTP Server (common files)
ii apache2-utils 2.4.29-1ubuntu4.8 amd64 Apache HTTP Server (utility programs for web servers)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
:~# vim /etc/apache2/
<VirtualHost 192.168.56.211:81 127.0.0.1:81>
Servername 127.0.0.1
ServerAdmin root@localhost
<Location /balancer-manager>
SetHandler balancer-manager
Require local
#Require ip 192.168.56.0/24 127.0.0.1/24
Require all granted
</Location>
LogLevel warn
ErrorLog ${APACHE_
CustomLog ${APACHE_
</VirtualHost>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
:~# vim /etc/apache2/
<Proxy "balancer://test">
ProxySet lbmethod=bybusyness
</Proxy>
<VirtualHost 127.0.0.1:8100>
ServerAdmin root@localhost
ServerName testapp01
ServerAlias 127.0.0.1:8100
ProxyPass "/test" "balancer://test"
ProxyPassRe
CustomLog ${APACHE_
ErrorLog ${APACHE_
</VirtualHost>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
:~# a2enmod proxy_balancer proxy_http lbmethod_bybusyness lbmethod_byrequests
:~# a2ensite management proxytest
:~# vim /etc/apache2/
[...]
Listen 81
Listen 8100
:~# systemctl restart apache2
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
At that point i install also some console Browsers for testing.
:~# apt-get install lynx elinks
:~# tail -f /var/log/
:~# elinks 127.0.0.
:~# lynx 127.0.0.
i can do update the Load and made changes. i also connect from outside with
Firefox
http://
all this creates no error log entrys, the log is still empty
-------
update apache
:~# apt-get update
:~# apt-get upgrade
:~# dpkg -l | grep apache2
ii apache2 2.4.29-1ubuntu4.10 amd64 Apache HTTP Server
ii apache2-bin 2.4.29-1ubuntu4.10 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.29-1ubuntu4.10 all Apache HTTP Server (common files)
ii apache2-utils 2.4.29-1ubuntu4.10 amd64 Apache HTTP Server (utility programs for web servers)
do the same with all the Browsers and have the error log in view.
http://
:~# tail -f /var/log/
[Wed Sep 04 12:24:55.740457 2019] [proxy_
:~# elinks 127.0.0.
:~# tail -f /var/log/
[Wed Sep 04 12:27:45.423011 2019] [proxy_
Firefox and elinks creat one single entry and updates from load etc. looks like
working but with
:~# lynx 127.0.0.
:~# tail -f /var/log/
[Wed Sep 04 12:28:58.249737 2019] [proxy_
[Wed Sep 04 12:29:09.585221 2019] [proxy_
[Wed Sep 04 12:29:15.435690 2019] [proxy_
[Wed Sep 04 12:29:29.771322 2019] [proxy_
every singel submit will create an entry and for example
the Load change will not made in the balancer manager.
The string from the Log Entry is in the newest Version from
https:/
http://
a downgrade to the prior Version to the Apache Packages solved the Problem.
Regards Horst
CVE References
|
|
#10 |
(In reply to Armin Abfalterer from comment #0)
> The new CSRF protection of the Balancer Manager breaks editing functionality
> for browsers that lowercase hostnames in the Referer: header; e.g. Chrome
>
> The error is based on the usage of strcmp() in the safe_referer() function
>
> https:/
> c#L1107
thanks for the report and sorry for the inconvenience. Trunk r1865749 and proposing for backport to 2.4.x.
| Paride Legovini (paride) wrote : | #1 |
Thanks for your bug report. The "ignoring params in balancer-manager cross-site access" error message has been introduced as part of the patchset fixing CVE-2019-10092, see [1], so this definitely looks like a regression.
| Paride Legovini (paride) wrote : | #2 |
| tags: | added: server-triage-discuss |
| tags: | added: server-next |
| tags: |
added: regression-update removed: server-next |
| tags: |
added: server-next removed: server-triage-discuss |
| Paride Legovini (paride) wrote : | #3 |
I subscribed and pinged ubuntu-security on this one, let's see if they chime in and what their opinion is.
hi all,
mybe i found a kind of same problem. in my configuration i used
the balancer manager behind a proxy
-------------
|Bastian Host |
|Apache Proxy | -----------> LB Apache Balancer Manger
-------------
and i struggle in the problem with an update from Ubuntu 18.04
which i described in the following bug report
https:/
on localhost with lynx i figure out error log entrys with every
submit and no update from the load data etc.
:~# tail -f /var/log/
[Wed Sep 04 12:28:58.249737 2019] [proxy_
[Wed Sep 04 12:29:09.585221 2019] [proxy_
i can reproduce this within debian 10
:~# apt-get install apache2
:~# dpkg -l | grep apache2
ii apache2 2.4.38-3+deb10u1
ii apache2-bin 2.4.38-3+deb10u1
ii apache2-data 2.4.38-3+deb10u1
ii apache2-utils 2.4.38-3+deb10u1
i download the prior version from the module and compile that version
http://
:~# apxs2 -c -i mod_proxy_
:~# systemctl restart apache2
:~# lynx 127.0.0.
:~# elinks 127.0.0.
Browser: http://
i can use the balancer manager as expected without any error log entry
i compiled also that version from https:/
but with no luck.
regards horst
| Horst Platz (hp-localhorst) wrote : | #4 |
i found https:/
|
|
#12 |
can you try something like this since you had a sandbox env:
Index: modules/
=======
--- modules/
+++ modules/
@@ -1185,7 +1185,7 @@
/* Ignore parameters if this looks like XSRF */
ref = apr_table_
if (apr_table_
- && (!ref || !safe_referer(r, ref))) {
+ && (ref && !safe_referer(r, ref))) {
No referer should pass through IIUC.
|
|
#13 |
Well, Eric you suggested it, so maybe I got it wrong ;)
In all valid requests to the balancer-manager the previous URI should be the balancer-manager page, and hence Referer should be set. So ignoring params if Referer is not present was definitely deliberate.
So OP are you're saying this fails with lynx and works with elinks?
|
|
#14 |
BTW I can't comment on that Ubuntu page without creating an account, so please point them to this comment
The referenced change to mod_proxy/
CVE-2019-10092 is fixed by https:/
if i came from ousite with firefox and on the console with elinks the first connect
:~# http://
:~# tail -f /var/log/
[Sat Sep 07 12:37:39.907268 2019] [proxy_
:~# elinks 127.0.0.
:~# tail -f /var/log/
[Sat Sep 07 12:40:42.786775 2019] [proxy_
creates on error log entry but it workes with lynx first connect and every submit creates the log entry
:~# lynx 127.0.0.
:~# tail -f /var/log/
[Sat Sep 07 12:41:43.620865 2019] [proxy_
[Sat Sep 07 12:42:20.582399 2019] [proxy_
[Sat Sep 07 12:42:33.611602 2019] [proxy_
[Sat Sep 07 12:42:37.749409 2019] [proxy_
so yes lynx ist not working
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
with your patch on
http://
i got no errog log entrys and lynx is also working for my sandbox env.
thx horst
i create the hint on the ubuntu page
| Horst Platz (hp-localhorst) wrote : | #5 |
https:/
and there is a Patch available
| Horst Platz (hp-localhorst) wrote : | #6 |
with that patch from here
https:/
and with the ubuntu 18.04 apache2 sources
:~$ apt-get source apache2
:~$ find . -name mod_proxy_
./apache2-
./apache2-
./apache2-
i copy and patched only that single file
:~$ cp ./apache2-
:~$ patch mod_proxy_
patching file mod_proxy_
Hunk #1 succeeded at 1078 (offset -107 lines).
compile it
:~# apxs2 -c -i mod_proxy_
i got also no more error log entries in my sandbox env from above. i copied that compiled binary away and i try early next week with that one if the initial problem behinde the proxy is also solved.
|
|
#16 |
Sorry I'm struggling to parse your comemnts, Horst.
From a quick search it looks some versions of Lynx don't produce Referer headers. They won't work with mod_proxy_balancer since 2.4.41, because we made tightened up the XSRF protection. This is unfortunate but we don't have a better way to protect against XSRF.
| Christian Ehrhardt (paelzer) wrote : | #7 |
Thanks for linking the upstream bug and your experiments Horst!
In the bug there it was mentioned that this would not be related to the CVE fix CVE-2019-10092.
But it made me think as Horst clearly found it to be related to that update.
I did some of the same checks Horst did (in which patch is the balancer touched).
There are three patches in the package referenced for this CVE:
- debian/
- debian/
big changes (but not part of the upstream CVE change)
- debian/
This last one is what brings changes to proxy/mod_
It is not directly tied to CVE-2019-10092 but seems to be picked up in that context.
That at least somewhat explains upstreams confusion on "referenced change to mod_proxy/
And if I got Horst right in the former comment he confirmed that if he drops that change it seems to work again.
But it seems (other than the mis-tag to CVE-2019-10092) this hardening to XSRF was an intended change by upstream [5].
I wasn't able to follow all comments of the upstream bug, they mentioned lynx might be incompatible to it- but does that apply to some proxies as well then?
In that case this might be a hard call on security-SRUing this into Bionic and breaking things. But while this is a no-go for normal SRUs security sometimes required changes like that.
@sbeattie - could you outline what was going on in the CVE discussions when this XSRF protection was added. And if you have any known discussions on adding XSRF protection that includes balancing those proxies/browsers.
[1]: https:/
[2]: https:/
[3]: https:/
[4]: https:/
[5]: https:/
| Christian Ehrhardt (paelzer) wrote : | #8 |
@Horst
I have put a preliminary build of the packaged Apache to the PPA [1] with the fix that was suggested on the upstream bug [2]. Could you give that one a try?
[1]: https:/
[2]: https:/
| Changed in apache2: | |
| importance: | Unknown → Medium |
| status: | Unknown → Confirmed |
| Steve Beattie (sbeattie) wrote : | #17 |
Sorry for the problems that people are experiencing.
Christian, the Ubuntu Security Team will sometimes incorporate a hardening measure like the extra XSRF that upstream included in the 2.4.41 release, if it appears to address similar issues as the original vulnerability. Looking at the history of modules/proxy/ in the 2.4.x branch made it look like they were mildly related. Unfortunately, upstream did not make explicitly clear in the 2.4.x branch which commits specifically addressed each vulnerability (and in fact, upstream managed to silently break an embargo with their fix for CVE-2019-9517).
The debian/
The issues with the https:/
I've made available pacakges which incorporate both patches mentioned in the upstream bug report (the one for the strcasecmp change and the change in the referrer test) in the ppa https:/
Thanks, and again, my apologies.
| Christian Ehrhardt (paelzer) wrote : | #18 |
Thanks for the explanations Steve.
I almost assumed something like this (adding related hardening) and this should not have been any blaming. I was just dissecting the case one step at a time.
Thanks for doing the next step already with the builds for all affected releases.
In that case I can stop the coding myself but I want to continue to help. And that might be with some testing instead.
I'll try if I can set up and repro the issues that were reported ...
| Christian Ehrhardt (paelzer) wrote : | #19 |
First of all, thanks to the great steps by Horst I was able to reproduce this on X/B/D releases.
like:
[Tue Sep 10 06:39:37.715128 2019] [proxy_
With that set up I upgraded all those to the PPA [1] and retried the access.
It works without the AH10187 error now.
I also found no other new issue in the logs triggered by the update - at least not for this setup.
In an SRU sense I'd now call it verified.
I hope that helps to get this processed further, since it needs pushing to the -security pocket I can't help much further.
[1]: https:/
| Horst Platz (hp-localhorst) wrote : | #20 |
sorry i can't use your PPAs in the production. for a quick test i used my patched compiled module where only one line is changing from the patch i discribed above
:$ diff mod_proxy_
1081c1081
< && (!ref || !safe_referer(r, ref))) {
---
> && (ref && !safe_referer(r, ref))) {
updated on of my production machine with the apache packages 2.4.29-1ubuntu4.10 and copy that module. lynx ist start working but the initial problem from outside over the bastian host proxy is not solved.
i will try to create a more better test env to use your ppas behinde a proxy but i'am sorry this needs a while.
thx for all your work, horst
| Horst Platz (hp-localhorst) wrote : | #21 |
unfortunately the ppa solve also not the behind a proxy problem.
usualy in my produktion in front (bastion/proxy host) is debian 9
so i test both with debian 9 and ubuntu 18.04 ppa at on the proxy
host.
i modified a littel the configuration to get closer for the
production env.
VM with LB Manager IP:192.168.56.211
i start again with the old apache version
:~# apt-get install libapr1 libaprutil1 libaprutil1-
:~# dpkg -i apache2_
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
:~# vim /etc/apache2/
<VirtualHost 192.168.56.211:81 127.0.0.1:81>
Servername 127.0.0.1
ServerAdmin root@localhost
<Location /balancer-manager>
SetHandler balancer-manager
Require local
#Require ip 192.168.56.0/24 127.0.0.1/24
Require all granted
</Location>
<Location /test-web01/
SetHandler balancer-manager
Require local
#Require ip 192.168.56.0/24 127.0.0.1/24
Require all granted
</Location>
LogLevel warn
ErrorLog ${APACHE_
CustomLog ${APACHE_
</VirtualHost>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
:~# vim /etc/apache2/
<Proxy "balancer://test">
ProxySet lbmethod=bybusyness
</Proxy>
<VirtualHost 127.0.0.1:8100>
ServerAdmin root@localhost
ServerName testapp01
ServerAlias 127.0.0.1:8100
ProxyPass "/test" "balancer://test"
ProxyPassRe
CustomLog ${APACHE_
ErrorLog ${APACHE_
</VirtualHost>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
:~# a2enmod proxy_balancer proxy_http lbmethod_bybusyness lbmethod_byrequests
:~# a2ensite management proxytest
:~# vim /etc/apache2/
[...]
Listen 81
Listen 8100
:~# systemctl restart apache2
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
check localhost LB Manager
:~# apt-get install lynx
:~# lynx 127.0.0.
:~# lynx 127.0.0.
:~# tail -f /var/log/
-> worked as expectet
-> no log entries on the LB Manager VM
-------
Bastion Host Proxy VM IP:192.168.56.230
:~# apt-get install apache2 lynx
check from proxy VM that LB Manager is working without a proxy config in
front of them.
:~# lynx 192.168.
:~# lynx 192.168.
:~# tail -f /var/log/
-> no log entries on the LB Manager VM
start proxy configuration
:~# vim /etc...
i'am sorry for that...
on the ubuntu page is a further more discussion and i explain and test the initilal problem LB manager behind a proxy ist not working after the update. mybe this helped out to get my problem more clear.
lynx is only comming to the playground for debug the initial problem. lynx with your patch ist start working again. but good to know that in the future lynx is mybe not a good choice for debugging purpose on that point.
| Changed in apache2 (Ubuntu): | |
| status: | New → Triaged |
| importance: | Undecided → Medium |
| Changed in apache2 (Ubuntu): | |
| assignee: | nobody → Steve Beattie (sbeattie) |
| Launchpad Janitor (janitor) wrote : | #23 |
This bug was fixed in the package apache2 - 2.4.18-2ubuntu3.13
---------------
apache2 (2.4.18-
* SECURITY REGRESSION: mod_proxy balancer XSS/CSRF hardening broke
browsers which change case in headers and breaks balancers
loading in some configurations (LP: #1842701)
- drop d/p/CVE-
-- Steve Beattie <email address hidden> Mon, 16 Sep 2019 06:13:53 -0700
| Changed in apache2 (Ubuntu): | |
| status: | Triaged → Fix Released |
| Launchpad Janitor (janitor) wrote : | #24 |
This bug was fixed in the package apache2 - 2.4.29-1ubuntu4.11
---------------
apache2 (2.4.29-
* SECURITY REGRESSION: mod_proxy balancer XSS/CSRF hardening broke
browsers which change case in headers and breaks balancers
loading in some configurations (LP: #1842701)
- drop d/p/CVE-
-- Steve Beattie <email address hidden> Mon, 16 Sep 2019 05:58:48 -0700
| Changed in apache2 (Ubuntu): | |
| status: | Triaged → Fix Released |
| Launchpad Janitor (janitor) wrote : | #25 |
This bug was fixed in the package apache2 - 2.4.38-2ubuntu2.3
---------------
apache2 (2.4.38-2ubuntu2.3) disco-security; urgency=medium
* SECURITY REGRESSION: mod_proxy balancer XSS/CSRF hardening broke
browsers which change case in headers and breaks balancers
loading in some configurations (LP: #1842701)
- drop d/p/CVE-
-- Steve Beattie <email address hidden> Mon, 16 Sep 2019 05:36:25 -0700
| Changed in apache2 (Ubuntu): | |
| status: | Triaged → Fix Released |
| Horst Platz (hp-localhorst) wrote : | #26 |
with the new packages my problem is solved.
on more question in the next Ubuntu release for example 20.04 with a newer apache version. it is possible that this kind of problem is comming back again? because the patches are in the newer version from apache.org.
thx again, regards horst
| Christian Ehrhardt (paelzer) wrote : | #27 |
Hi Horst,
yes I checked and the issue is in Eoan 2.4.41 - I checked that already last week and let Steve now.
Steve wanted to track the upstream discussions on this as going forward we most likely want to follow upstreams guidance on this (e.g. want to have it broken for better security).
But thanks for the ping, we might want to mark the bug tasks accordingly to make this clear.
| Changed in apache2 (Ubuntu): | |
| status: | Fix Released → Confirmed |
| Changed in apache2 (Ubuntu Xenial): | |
| status: | New → Fix Released |
| Changed in apache2 (Ubuntu Bionic): | |
| status: | New → Fix Released |
| Changed in apache2 (Ubuntu Disco): | |
| status: | New → Fix Released |
| Horst Platz (hp-localhorst) wrote : | #28 |
hi Christian,
thx for the info and please let me know if there is a posibility solution for the future releases.
| Christian Ehrhardt (paelzer) wrote : | #29 |
I'll if I hear something, but I'll leave that task mostly to Steve who said that he wanted to keep an eye on it (for potentially backporting the hardening once we know how to handle the regression).
| Changed in apache2 (Debian): | |
| status: | Unknown → New |
| Changed in apache2 (Debian): | |
| status: | New → Fix Committed |
| Changed in apache2 (Debian): | |
| status: | Fix Committed → Fix Released |
This has been backported in 2.4.x branch in r1865966
This is part of 2.4.42.
Not sure that comments in comment #2 and below are related.
If needed, please open a new bug report for it.
| Changed in apache2: | |
| status: | Confirmed → Fix Released |
| Robie Basak (racb) wrote : | #31 |
Looks like this is still open for Groovy, but will be resolved when we merge 2.4.42.
| tags: | removed: server-next |
The new CSRF protection of the Balancer Manager breaks editing functionality for browsers that lowercase hostnames in the Referer: header; e.g. Chrome
The error is based on the usage of strcmp() in the safe_referer() function
https:/