Apache2 Balancer Manager not working after dist-upgrade to focal behind a Proxy

[Horst Platz]

Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal

Approximately two years ago i creat the following bug report.

https://bugs.launchpad.net/apache2/+bug/1842701/

At the end talk to that bug report is a hin "that bug is still opend in
apache 2.4.41" and further on "it should be fixed in 2.4.42". mybe this is
now forgotten to fix in focal and it is a regression bug or the
"(e.g. want to have it broken for better security)." ist happend.

any suggestion is appreciated

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

To Reproduced i create two new Virtual Box VMs on my local Machine

 -------------
|Bastian Host |
|Apache Proxy | -----------> LB Apache Balancer Manger
 -------------

Debian 10 -> Bastion Host (Proxy) / 192.168.56.90
Ubuntu 18.04 -> LB Manager / 192.168.56.160

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Ubuntu 18.04 with LB Manager

:~# apt-get install apache2

:~# dpkg -l | grep apache2
ii apache2 2.4.29-1ubuntu4.16 amd64 Apache HTTP Server
ii apache2-bin 2.4.29-1ubuntu4.16 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.29-1ubuntu4.16 all Apache HTTP Server (common files)
ii apache2-utils 2.4.29-1ubuntu4.16 amd64 Apache HTTP Server (utility programs for web servers)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

:~# vim /etc/apache2/sites-available/management.conf
<VirtualHost 192.168.56.160:81 127.0.0.1:81>
    Servername 127.0.0.1
    ServerAdmin root@localhost

    <Location /balancer-manager>
        SetHandler balancer-manager
        Require local
        #Require ip 192.168.56.0/24 127.0.0.1/24
        Require all granted
    </Location>

    <Location /test-web01/balancer-manager>
        SetHandler balancer-manager
        Require local
        #Require ip 192.168.56.0/24 127.0.0.1/24
        Require all granted
    </Location>

    LogLevel warn
    ErrorLog ${APACHE_LOG_DIR}/management_error.log
    CustomLog ${APACHE_LOG_DIR}/management_access.log combined

</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

:~# vim /etc/apache2/sites-available/proxytest.conf
<Proxy "balancer://test">
        BalancerMember "http://192.168.168.130/test"
        BalancerMember "http://192.168.168.131/test" status=+H
        ProxySet lbmethod=bybusyness
</Proxy>

<VirtualHost 127.0.0.1:8100>
ServerAdmin root@localhost
ServerName testapp01
ServerAlias 127.0.0.1:8100

    ProxyPass "/test" "balancer://test"
    ProxyPassReverse "/test" "balancer://test"

    CustomLog ${APACHE_LOG_DIR}/test-access.log combined
    ErrorLog ${APACHE_LOG_DIR}/test-error.log

</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

:~# a2enmod proxy_balancer proxy_http lbmethod_bybusyness lbmethod_byrequests
:~# a2ensite management proxytest

:~# vim /etc/apache2/ports.conf
[...]
Listen 81
Listen 8100

:~# systemctl restart apache2

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

At that point i install also some console Browsers for testing.

:~# apt-get install lynx elinks

:~# tail -f /var/log/apache2/management_error.log

:~# elinks http://127.0.0.1:81/balancer-manager
:~# lynx http://127.0.0.1:81/balancer-manager

LB Manager -> "Edit worker settings for..." is visible and usable, no error log entrys

- - - - - - - - - - - - - - - - - - - - - - - - -

Connect with firefox from outside

http://192.168.56.160:81/balancer-manager

LB Manager -> "Edit worker settings for..." is visible and usable, no error log entrys

-------------------------------------------------------------------------

Create Debian 10 Proxy VM

:~# apt-get install apache2 lynx elinks

:~# dpkg -l | grep apache
ii apache2 2.4.38-3+deb10u5 amd64 Apache HTTP Server
ii apache2-bin 2.4.38-3+deb10u5 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.38-3+deb10u5 all Apache HTTP Server (common files)
ii apache2-utils 2.4.38-3+deb10u5 amd64 Apache HTTP Server (utility programs for web servers)

Check from that VM that LB Manager is usable

:~# elinks 192.168.56.160:81/balancer-manager
:~# elinks 192.168.56.160:81/test-web01/balancer-manager

check direkt in Firefox
http://192.168.56.160:81/balancer-manager

:~# tail -f /var/log/apache2/management_error.log
LB Manager -> "Edit worker settings for..." is visible and usable, no error log entrys

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Create Proxy Config

~# vim /etc/apache2/sites-enabled/000-default.conf
[...]
       <Location /test-web01>
                ProxyPass http://192.168.56.160:81/test-web01
                ProxyPassReverse http://192.168.56.160:81/test-web01
                SetOutputFilter INFLATE;SUBSTITUTE
                Substitute "s|http://192.168.56.90:81|http://192.168.56.160|i"
       </Location>
[...]

:~# a2enmod proxy_http substitute
:~# systemctl restart apache2

Check now over proxy config

:~# elinks 192.168.56.90/test-web01/balancer-manager
-> "Edit worker settings for..." is visible and usable, no error log entrys

Browser: Firefox and Chrome

http://192.168.56.90/test-web01/balancer-manager

:~# tail -f /var/log/apache2/management_error.log
LB Manager -> "Edit worker settings for..." is visible and usable, no error log entrys

=> Conclusion everything worked for me as expected. LB Manager is usable
   through the Proxy Bastion Host.

-------------------------------------------------------------------------

dist-upgrade from the LB Manager VM 18.04 -> 20.04

:~# do-release-upgrade
:~# shutdown -r now

:~# dpkg -l | grep apache2
ii apache2 2.4.41-4ubuntu3.4 amd64 Apache HTTP Server
ii apache2-bin 2.4.41-4ubuntu3.4 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.41-4ubuntu3.4 all Apache HTTP Server (common files)
ii apache2-utils 2.4.41-4ubuntu3.4 amd64 Apache HTTP Server (utility programs for web servers)

Check from the LB Manager VM directly now 20.04

:~# elinks http://127.0.0.1:81/balancer-manager
:~# lynx http://127.0.0.1:81/balancer-manager

:~# tail -f /var/log/apache2/management_error.log
[Thu Aug 12 11:07:45.381180 2021] [proxy_balancer:error] [pid 709:tid 139675415078656] [client 127.0.0.1:44158] AH10187: ignoring params in balancer-manager cross-site access

check direct in Firefox
http://192.168.56.160:81/balancer-manager

:~# tail -f /var/log/apache2/management_error.log
[Thu Aug 12 11:09:10.362535 2021] [proxy_balancer:error] [pid 709:tid 139675381507840] [client 192.168.56.1:5684] AH10187: ignoring params in balancer-manager cross-site access

every connect makes a single Log entry, but LB Manager directly is usable

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Check over Proxy on Debian 10 VM

:~# elinks http://192.168.56.90/test-web01/balancer-manager

[Thu Aug 12 11:11:41.379048 2021] [proxy_balancer:error] [pid 709:tid 139675230439168] [client 192.168.56.90:48336] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager?b=test&w=http://192.168.168.130/test&nonce=71c9c136-9639-0ce8-7cb7-e545ae00c50f

LB Manager ist not usable

The "Edit worker settings for..." is not visible

- - - - - - - - - - - - - - - - - - - - - - - - -

Browser: Firefox and Chrome

http://192.168.56.90/test-web01/balancer-manager

[Thu Aug 12 11:12:48.035479 2021] [proxy_balancer:error] [pid 709:tid 139675423471360] [client 192.168.56.90:48338] AH10187: ignoring params in balancer-manager cross-site access
[Thu Aug 12 11:12:50.842842 2021] [proxy_balancer:error] [pid 709:tid 139675406685952] [client 192.168.56.90:48338] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager
[Thu Aug 12 11:12:51.653153 2021] [proxy_balancer:error] [pid 709:tid 139675398293248] [client 192.168.56.90:48338] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager?b=test&w=http://192.168.168.130/test&nonce=71c9c136-9639-0ce8-7cb7-e545ae00c50f
[Thu Aug 12 11:12:53.183729 2021] [proxy_balancer:error] [pid 709:tid 139675389900544] [client 192.168.56.90:48338] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager?b=test&w=http://192.168.168.131/test&nonce=71c9c136-9639-0ce8-7cb7-e545ae00c50f
[Thu Aug 12 11:12:53.639131 2021] [proxy_balancer:error] [pid 709:tid 139675501659904] [client 192.168.56.90:48338] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager?b=test&w=http://192.168.168.130/test&nonce=71c9c136-9639-0ce8-7cb7-e545ae00c50f

First connect and every singel klick create an error entry.

LB Manager is not usable

The "Edit worker settings for..." is not visible

=> Conclusion after Update to focal i can't use the LB Manager behind
   that Proxy Bastion Host. Unfortunately for me there is no other
   way to jump in the Production environment.

As i say any suggestion is appreciated is this now a bug (regression) or
is it now a security feature. That will be for me a realy bad news.

Regard Horst

[Horst Platz (hp-localhorst)]

mybe i thought with a newer apache the problem is solved but with no luck.

LB Manager VM dist-upgrade 21.04 / hirsute

:~# vim /etc/update-manager/release-upgrades
[...]
#Prompt=lts
Prompt=normal

:~# do-release-upgrade
:~# shutdown -r now

:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 21.04
Release: 21.04
Codename: hirsute

:~# dpkg -l | grep apache2
ii apache2 2.4.46-4ubuntu1.1 amd64 Apache HTTP Server
ii apache2-bin 2.4.46-4ubuntu1.1 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.46-4ubuntu1.1 all Apache HTTP Server (common files)
ii apache2-utils 2.4.46-4ubuntu1.1 amd64 Apache HTTP Server (utility programs for web servers)

Browser: Firefox and Chrome

http://192.168.56.90/test-web01/balancer-manager

:~# tail -f /var/log/apache2/management_error.log
[Fri Aug 13 16:56:31.818801 2021] [proxy_balancer:error] [pid 717:tid 140627126277696] [client 192.168.56.90:44938] AH10187: ignoring params in balancer-manager cross-site access
[Fri Aug 13 16:56:34.088176 2021] [proxy_balancer:error] [pid 717:tid 140627117884992] [client 192.168.56.90:44938] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager
[Fri Aug 13 16:56:34.663154 2021] [proxy_balancer:error] [pid 717:tid 140627109492288] [client 192.168.56.90:44938] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager?b=test&w=http://192.168.168.130/test&nonce=936feca7-14cd-c118-d222-c6b172c9729e

[Paride Legovini (paride)]

Hello Horst and thanks for your bug report. I think I'm able to reproduce the issue and hopefully I came up with a "more minimal" configuration. As a first step to better understand what's going on here I'd like to check with you that my config does indeed cause the problem.

On a clean Impish system I did the folowing:

--- reproducer (tentative) ---

1. apt install apache2
2. Create/etc/apache2/sites-available/management.conf with:

<VirtualHost 127.0.0.1:80>
    <Location /balancer-manager>
        SetHandler balancer-manager
    </Location>

    LogLevel debug
    ErrorLog ${APACHE_LOG_DIR}/management_error.log
    CustomLog ${APACHE_LOG_DIR}/management_access.log combined
</VirtualHost>

3. a2enmod proxy_balancer
4. a2ensite management
5. systemctl restart apache2

6. With a brower (in my case: Firefox) go to http://127.0.0.1/balancer-manager. A "Load Balancer Manager for 127.0.0.1" page opens.
7. Check /var/log/apache2/management_error.log and verify that is has an error line like:

[client 127.0.0.1:54490] AH10187: ignoring params in balancer-manager cross-site access

which is the problem this bug report is about.

--- end ---

Is this in your opinion a good reproducer? Is there an Ubuntu (or Debian) release where the same set of steps does *not* generate that AH10187 error?

Waiting for your reply I'm marking this bug report as Incomplete; please comment back and we'll look at it again. Thanks!

Note: I didn't mark this report as a duplicate of LP: #1842701 because I'm not confident enough it is the very same problem, but it probably is.

[Horst Platz (hp-localhorst)]

Hello Paride, thank you for your reply. I'am not sure that your minimal configuration reproduce the hole issue. I reproduce your config with 18.04/bionic and 20.04/focal. What i can say is yes with focal i saw also the error

[...]
[Wed Aug 18 08:22:14.241792 2021] [authz_core:debug] [pid 4199:tid 140445086639872] mod_authz_core.c(817): [client 10.0.2.2:2171] AH01626: authorization result of Require all granted: granted
[Wed Aug 18 08:22:14.241843 2021] [authz_core:debug] [pid 4199:tid 140445086639872] mod_authz_core.c(817): [client 10.0.2.2:2171] AH01626: authorization result of <RequireAny>: granted
[Wed Aug 18 08:22:14.241863 2021] [proxy_balancer:error] [pid 4199:tid 140445086639872] [client 10.0.2.2:2171] AH10187: ignoring params in balancer-manager cross-site access
[Wed Aug 18 08:22:14.241873 2021] [proxy_balancer:debug] [pid 4199:tid 140445086639872] mod_proxy_balancer.c(1451): [client 10.0.2.2:2171] AH01204: genning page
[Wed Aug 18 08:22:14.242740 2021] [deflate:debug] [pid 4199:tid 140445086639872] mod_deflate.c(854): [client 10.0.2.2:2171] AH01384: Zlib: Compressed 994 to 466 : URL /balancer-manager[...]

and with bionic your minimal config the error is not comming up.

[...]
[Wed Aug 18 10:21:10.436566 2021] [authz_core:debug] [pid 2527:tid 139950091626240] mod_authz_core.c(809): [client 10.0.2.2:2161] AH01626: authorization result of Require all granted: granted
[Wed Aug 18 10:21:10.436751 2021] [authz_core:debug] [pid 2527:tid 139950091626240] mod_authz_core.c(809): [client 10.0.2.2:2161] AH01626: authorization result of <RequireAny>: granted
[Wed Aug 18 10:21:10.436795 2021] [proxy_balancer:debug] [pid 2527:tid 139950091626240] mod_proxy_balancer.c(1319): [client 10.0.2.2:2161] AH01204: genning page
[Wed Aug 18 10:21:10.437269 2021] [deflate:debug] [pid 2527:tid 139950091626240] mod_deflate.c(854): [client 10.0.2.2:2161] AH01384: Zlib: Compressed 994 to 465 : URL /balancer-manager
[...]

But with your minimal config the LB Manager is not "functional" it appear no

LoadBalancer Status for ...

or

Edit worker settings for ...

So you can do nothing with the LB Manager.

If i go ahead an create the "second part /etc/apache2/sites-available/proxytest.conf" the problem is when you are on the same box without the proxy in front. You saw that the error cames one time after the first contact. But the LB Manager is fully functional and produces no further error entrys. If you go ahead and use a proxy in front than the LB Manager is not Working and every single klick create a log error.

If you are able to fix focal with your minmal configuration and the error disapear like in bionic mybe this will also fix the hole issue with the proxy in front of it.

If i can do anything more let me know that.

thx horst

[Horst Platz (hp-localhorst)]

Hello Paride, today i recreate my bigger configuration with Debian 10 and 11

Debian 10 Proxy and Debian 10 LB Manager witch Apache 2.4.38-3+deb10u5 every thing worked as expected no error log entrys. LB Manager over proxy is fully workable.

- - - - - - - - - - - - - - - - - - - - - - - - -

LB Manager Upgrade -> Debian 11 with Apache 2.4.48-3.1+deb11u1

If i go to the LB Manager directly one log entry after first connect

:~# tail -f /var/log/apache2/management_error.log
[Thu Aug 19 12:51:20.867073 2021] [proxy_balancer:error] [pid 459:tid 139790420895488] [client 192.168.56.1:13139] AH10187: ignoring params in balancer-manager cross-site access

and LB Manager is operational.

But if i go over Debian 10 Proxy first connect and every single klick will generate error log entry and LB Manager is not workable.

[Thu Aug 19 12:54:49.382870 2021] [proxy_balancer:error] [pid 459:tid 139790280275712] [client 192.168.56.91:47906] AH10187: ignoring params in balancer-manager cross-site access
[Thu Aug 19 12:54:53.580821 2021] [proxy_balancer:error] [pid 459:tid 139790288668416] [client 192.168.56.91:47906] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.91/test-web01/balancer-manager
[Thu Aug 19 12:54:54.820184 2021] [proxy_balancer:error] [pid 459:tid 139790263490304] [client 192.168.56.91:47906] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.91/test-web01/balancer-manager?b=test&w=http://192.168.168.130/test&nonce=1141264e-846f-cc1c-7876-7f345241b1ea

- - - - - - - - - - - - - - - - - - - - - - - - -

Proxy Upgrade -> Debian 11 with Apache 2.4.48-3.1+deb11u1

Makes no different first connect and single klicks generate logs and LB Manager not functional.

[Thu Aug 19 13:19:12.763111 2021] [proxy_balancer:error] [pid 459:tid 139790179628800] [client 192.168.56.91:55964] AH10187: ignoring params in balancer-manager cross-site access
[Thu Aug 19 13:19:15.498160 2021] [proxy_balancer:error] [pid 459:tid 139790162843392] [client 192.168.56.91:55964] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.91/test-web01/balancer-manager
[Thu Aug 19 13:19:16.154956 2021] [proxy_balancer:error] [pid 459:tid 139790171236096] [client 192.168.56.91:55964] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.91/test-web01/balancer-manager?b=test&w=http://192.168.168.130/test&nonce=1141264e-846f-cc1c-7876-7f345241b1ea

[Horst Platz (hp-localhorst)]

Hello Paride,

any kind of suggestion on that issue.

thx Horst

[Paride Legovini (paride)]

Hello Horst and sorry for the late reply. I tried again to setup an Ubuntu-based reproducer, but I failed to draw any useful conclusion. A few questions come to my mind at this point:

1. Can you still reproduce the issue with apache2 from Impish or Jammy?

2. You identified https://bz.apache.org/bugzilla/show_bug.cgi?id=63688 as describing this issue in the version of apache2 shipped with Focal. As Hirsute should have a fixed version, but in your experience it does not, I think it's worth checking with the upstream Apache developers once again about the issue.

3. I fail to tell if the behavior you describe is due to a bug or to a local configuration issue. Without first clearly identifying the issue as a bug we can't really begin working on it. If you think there's actually a bug in Ubuntu here, what we need are some minimal but complete steps to reproduce it locally, ideally in LXD containers.

I'm marking this as Incomplete for the moment.

[Horst Platz (hp-localhorst)]

I came across from here

https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1939678

After Update from Ubuntu 18.04 apache2 2.4.29 to Ubuntu 20.04 apache2 2.4.41

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

To Reproduce i create two new Virtual Box VMs on my local Machine and compile apach2 from the sources 2.4.38 -> 2.4.51

 -------------
|Bastian Host |
|Apache Proxy | -----------> LB Apache Balancer Manger
 -------------

Debian 11 -> Bastion Host (Proxy) / 192.168.56.70
Ubuntu 20.04 -> LB Manager / 192.168.56.170

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

I use the Proxy as Bastion Host to reache several LB Manager from one Point.

Debian 11 Proxy strip down config for reproduce the issue.

:~# apt-get install apache2

~# vim /etc/apache2/sites-enabled/000-default.conf
[...]
       <Location /balancer-manager>
                ProxyPass http://192.168.56.170:81/balancer-manager
                ProxyPassReverse http://192.168.56.170:81/balancer-manager
                SetOutputFilter INFLATE;SUBSTITUTE
                Substitute "s|http://192.168.56.70:81|http://192.168.56.170|i"
       </Location>
[...]

:~# a2enmod proxy_http substitute
:~# systemctl restart apache2

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Second VM with Ubuntu 20.04 for testing and reproduction. I compile apache from the sources. I Start with apache 2.4.38

wget http://archive.apache.org/dist/httpd/httpd-2.4.38.tar.gz
configure / make / make install

:~# vim /usr/local/apache2/conf/httpd.conf
ServerRoot "/usr/local/apache2"
ServerName "localhost"
ServerAdmin <email address hidden>

Listen 80
Listen 81
Listen 8100

LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule filter_module modules/mod_filter.so
LoadModule substitute_module modules/mod_substitute.so
LoadModule mime_module modules/mod_mime.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule headers_module modules/mod_headers.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule dir_module modules/mod_dir.so
LoadModule alias_module modules/mod_...

[Horst Platz (hp-localhorst)]

Hello Paride,

i am sorry but i am not familiar with LXD containers and yes i can reproduce the issue. So i use my config from above

The Proxy Machine is now Debian 11 apache2 2.4.51-1~deb11u1

18.04/bionic apache2 2.4.29-1ubuntu4.20 -> is working as expected
20.04/focal apache2 2.4.41-4ubuntu3.8 -> not working as expected
21.04/hirsute apache2 2.4.46-4ubuntu1.4 -> not working as expected
21.10/impish apache2 2.4.48-3.1ubuntu3.1 -> not working as expected

So i go ahead and compile apache from source to see how the upstream apache looks like and i came to the same result. With apache 2.4.41 starts the issue and the last version 2.4.51 is also affected from my perception. I create a bug report there.

https://bz.apache.org/bugzilla/show_bug.cgi?id=65737

And hopfully there comming some more information or results.

[Paride Legovini (paride)]

Thanks, I added a new Apache2 project task to this bug, tracking the upstream bug you filed, so we'll get notified of changes there.

[Horst Platz (hp-localhorst)]

There is an copy & paste bug in my Substitute it must be.

[...]
                Substitute "s|http://192.168.56.170:81|http://192.168.56.70|i"
[...]

And i investigate further on i grab some httpd versions i can get from

wget http://archive.apache.org/dist/httpd/httpd-2.4.39.tar.gz
wget http://archive.apache.org/dist/httpd/httpd-2.4.41.tar.gz
wget http://archive.apache.org/dist/httpd/httpd-2.4.51.tar.gz

And i diff mod_proxy_balancer.c from version to version for instance.

:~$ diff httpd-2.4.39/modules/proxy/mod_proxy_balancer.c httpd-2.4.41/modules/proxy/mod_proxy_balancer.c

In version 2.4.41 mod_proxy_balancer.c and in the further versions i found that peace of code. And it looks like that trigger the error.

:~$ vim mod_proxy_balancer.c
[...]
    /* Ignore parameters if this looks like XSRF */
    ref = apr_table_get(r->headers_in, "Referer");
    if (apr_table_elts(params)
        && (!ref || !safe_referer(r, ref))) {
        ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r, APLOGNO(10187)
                      "ignoring params in balancer-manager cross-site access");
        apr_table_clear(params);
    }
[...]

If i delete that and recompile than it looks like anything is working again with the proxy in front of the LB Manager. But i'am not a Developer and i don't know is that an good idea to delete that code. And mybe on different places happening other bad things.

Any suggestion is appreciated.

Thx Horst

[Horst Platz (hp-localhorst)]

Hello Paride,

for a kinde of integrity i did the same with the ubuntu 20.04 Apache.

:~$ apt-get source apache2
:~$ cp apache2-2.4.41/modules/proxy/mod_proxy_balancer.c .
:~$ vim mod_proxy_balancer.c
[...]
    /* Ignore parameters if this looks like XSRF */
    ref = apr_table_get(r->headers_in, "Referer");
    if (apr_table_elts(params)
        && (!ref || !safe_referer(r, ref))) {
        ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r, APLOGNO(10187)
                      "ignoring params in balancer-manager cross-site access");
        apr_table_clear(params);
    }
[...]

Delete that part and compile the module only

:~# apxs2 -c -i mod_proxy_balancer.c

:~# systemctl restart apache2

And it looks like for me everything ist working again as expected. But also the I'am a non Developer hint and mybe this todo so is a really bad idea.

Thx Horst

[Grelaud (fabrice-grelaud)]

Hi all,

what about this bug ?

I got three VMs behind an apache-balancer and i got the same issue.

Distributor ID: Ubuntu
Description: Ubuntu 20.04.5 LTS
Release: 20.04
Codename: focal

ii apache2 2.4.41-4ubuntu3.12 amd64 Apache HTTP Server

Is there a fix for this package ?

Regards,
Fabrice

[Grelaud (fabrice-grelaud)]

Hi all,

what about this bug ?

I got three VMs behind an apache-balancer and i got the same issue.

Distributor ID: Ubuntu
Description: Ubuntu 20.04.5 LTS
Release: 20.04
Codename: focal

ii apache2 2.4.41-4ubuntu3.12 amd64 Apache HTTP Server

Is there a fix for this package ?

Regards,
Fabrice

[Paride Legovini (paride)]

Hello Grelaud,

This bug report is Incomplete because we weren't unable to triage it as a bug in Ubuntu. What we need here is a minimal reproducer, with instructions on how to recreate the problem from fresh Ubuntu systems. LXD containers can be used to quickly create separate systems for the balancer and the actual webservers.

If you are not sure you're are facing the same issue initially reported here I suggest you to file a separate bug report, again possibly with instructions on how to reproduce it.

[Ustymenko-andrew (ustymenko-andrew)]

Hi all,

Have the same issue in 2.4.55. Apparently the code mentioned earlier

:~$ vim mod_proxy_balancer.c
[...]
    /* Ignore parameters if this looks like XSRF */
    ref = apr_table_get(r->headers_in, "Referer");
    if (apr_table_elts(params)
        && (!ref || !safe_referer(r, ref))) {
        ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r, APLOGNO(10187)
                      "ignoring params in balancer-manager cross-site access");
        apr_table_clear(params);
    }
[...]

is checking for the Referer header in the request. The correct Referer can be taken from DevTools when accessing balancer-manager page in the browser and clicking on the corresponding backend.

Adding that to the request will fix the issue.

[Jorton-9 (jorton-9)]

Yes, you need to have a valid Referer: header in the request since 2.4.41 - this is a deliberate security feature to prevent XSS/XSRF attacks. So I don't see a bug here.

[Horst Platz (hp-localhorst)]

ok thx for the infos and in my example case. the referre header must in the proxy configuration.

~# vim /etc/apache2/sites-enabled/000-default.conf
[...]
<Location /balancer-manager>
  ProxyPass http://192.168.56.170:81/balancer-manager
  ProxyPassReverse http://192.168.56.170:81/balancer-manager
  RequestHeader set Referer http://192.168.56.170:81/balancer-manager
  SetOutputFilter INFLATE;SUBSTITUTE
  Substitute "s|http://192.168.56.170:81|http://192.168.56.70|i"
</Location>
[...]

[Horst Platz (hp-localhorst)]

It is not a bug it is a security feature. See at

https://bz.apache.org/bugzilla/show_bug.cgi?id=65737

Regards Horst

[Paride Legovini (paride)]

Thanks for the update. Do you think we can closed this bug report as Invalid (= not a bug in Ubuntu) then?

[Horst Platz (hp-localhorst)]

Yes i think so.