Apache2 Balancer Manager not working after dist-upgrade to focal behind a Proxy

Bug #1939678 reported by Horst Platz
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apache2 (Ubuntu)
Undecided
Unassigned

Bug Description

Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal

Approximately two years ago i creat the following bug report.

https://bugs.launchpad.net/apache2/+bug/1842701/

At the end talk to that bug report is a hin "that bug is still opend in
apache 2.4.41" and further on "it should be fixed in 2.4.42". mybe this is
now forgotten to fix in focal and it is a regression bug or the
"(e.g. want to have it broken for better security)." ist happend.

any suggestion is appreciated

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

To Reproduced i create two new Virtual Box VMs on my local Machine

 -------------
|Bastian Host |
|Apache Proxy | -----------> LB Apache Balancer Manger
 -------------

Debian 10 -> Bastion Host (Proxy) / 192.168.56.90
Ubuntu 18.04 -> LB Manager / 192.168.56.160

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Ubuntu 18.04 with LB Manager

:~# apt-get install apache2

:~# dpkg -l | grep apache2
ii apache2 2.4.29-1ubuntu4.16 amd64 Apache HTTP Server
ii apache2-bin 2.4.29-1ubuntu4.16 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.29-1ubuntu4.16 all Apache HTTP Server (common files)
ii apache2-utils 2.4.29-1ubuntu4.16 amd64 Apache HTTP Server (utility programs for web servers)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

:~# vim /etc/apache2/sites-available/management.conf
<VirtualHost 192.168.56.160:81 127.0.0.1:81>
    Servername 127.0.0.1
    ServerAdmin root@localhost

    <Location /balancer-manager>
        SetHandler balancer-manager
        Require local
        #Require ip 192.168.56.0/24 127.0.0.1/24
        Require all granted
    </Location>

    <Location /test-web01/balancer-manager>
        SetHandler balancer-manager
        Require local
        #Require ip 192.168.56.0/24 127.0.0.1/24
        Require all granted
    </Location>

    LogLevel warn
    ErrorLog ${APACHE_LOG_DIR}/management_error.log
    CustomLog ${APACHE_LOG_DIR}/management_access.log combined

</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

:~# vim /etc/apache2/sites-available/proxytest.conf
<Proxy "balancer://test">
        BalancerMember "http://192.168.168.130/test"
        BalancerMember "http://192.168.168.131/test" status=+H
        ProxySet lbmethod=bybusyness
</Proxy>

<VirtualHost 127.0.0.1:8100>
ServerAdmin root@localhost
ServerName testapp01
ServerAlias 127.0.0.1:8100

    ProxyPass "/test" "balancer://test"
    ProxyPassReverse "/test" "balancer://test"

    CustomLog ${APACHE_LOG_DIR}/test-access.log combined
    ErrorLog ${APACHE_LOG_DIR}/test-error.log

</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

:~# a2enmod proxy_balancer proxy_http lbmethod_bybusyness lbmethod_byrequests
:~# a2ensite management proxytest

:~# vim /etc/apache2/ports.conf
[...]
Listen 81
Listen 8100

:~# systemctl restart apache2

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

At that point i install also some console Browsers for testing.

:~# apt-get install lynx elinks

:~# tail -f /var/log/apache2/management_error.log

:~# elinks http://127.0.0.1:81/balancer-manager
:~# lynx http://127.0.0.1:81/balancer-manager

LB Manager -> "Edit worker settings for..." is visible and usable, no error log entrys

- - - - - - - - - - - - - - - - - - - - - - - - -

Connect with firefox from outside

http://192.168.56.160:81/balancer-manager

LB Manager -> "Edit worker settings for..." is visible and usable, no error log entrys

-------------------------------------------------------------------------

Create Debian 10 Proxy VM

:~# apt-get install apache2 lynx elinks

:~# dpkg -l | grep apache
ii apache2 2.4.38-3+deb10u5 amd64 Apache HTTP Server
ii apache2-bin 2.4.38-3+deb10u5 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.38-3+deb10u5 all Apache HTTP Server (common files)
ii apache2-utils 2.4.38-3+deb10u5 amd64 Apache HTTP Server (utility programs for web servers)

Check from that VM that LB Manager is usable

:~# elinks 192.168.56.160:81/balancer-manager
:~# elinks 192.168.56.160:81/test-web01/balancer-manager

check direkt in Firefox
http://192.168.56.160:81/balancer-manager

:~# tail -f /var/log/apache2/management_error.log
LB Manager -> "Edit worker settings for..." is visible and usable, no error log entrys

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Create Proxy Config

~# vim /etc/apache2/sites-enabled/000-default.conf
[...]
       <Location /test-web01>
                ProxyPass http://192.168.56.160:81/test-web01
                ProxyPassReverse http://192.168.56.160:81/test-web01
                SetOutputFilter INFLATE;SUBSTITUTE
                Substitute "s|http://192.168.56.90:81|http://192.168.56.160|i"
       </Location>
[...]

:~# a2enmod proxy_http substitute
:~# systemctl restart apache2

Check now over proxy config

:~# elinks 192.168.56.90/test-web01/balancer-manager
-> "Edit worker settings for..." is visible and usable, no error log entrys

Browser: Firefox and Chrome

http://192.168.56.90/test-web01/balancer-manager

:~# tail -f /var/log/apache2/management_error.log
LB Manager -> "Edit worker settings for..." is visible and usable, no error log entrys

=> Conclusion everything worked for me as expected. LB Manager is usable
   through the Proxy Bastion Host.

-------------------------------------------------------------------------

dist-upgrade from the LB Manager VM 18.04 -> 20.04

:~# do-release-upgrade
:~# shutdown -r now

:~# dpkg -l | grep apache2
ii apache2 2.4.41-4ubuntu3.4 amd64 Apache HTTP Server
ii apache2-bin 2.4.41-4ubuntu3.4 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.41-4ubuntu3.4 all Apache HTTP Server (common files)
ii apache2-utils 2.4.41-4ubuntu3.4 amd64 Apache HTTP Server (utility programs for web servers)

Check from the LB Manager VM directly now 20.04

:~# elinks http://127.0.0.1:81/balancer-manager
:~# lynx http://127.0.0.1:81/balancer-manager

:~# tail -f /var/log/apache2/management_error.log
[Thu Aug 12 11:07:45.381180 2021] [proxy_balancer:error] [pid 709:tid 139675415078656] [client 127.0.0.1:44158] AH10187: ignoring params in balancer-manager cross-site access

check direct in Firefox
http://192.168.56.160:81/balancer-manager

:~# tail -f /var/log/apache2/management_error.log
[Thu Aug 12 11:09:10.362535 2021] [proxy_balancer:error] [pid 709:tid 139675381507840] [client 192.168.56.1:5684] AH10187: ignoring params in balancer-manager cross-site access

every connect makes a single Log entry, but LB Manager directly is usable

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Check over Proxy on Debian 10 VM

:~# elinks http://192.168.56.90/test-web01/balancer-manager

[Thu Aug 12 11:11:41.379048 2021] [proxy_balancer:error] [pid 709:tid 139675230439168] [client 192.168.56.90:48336] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager?b=test&w=http://192.168.168.130/test&nonce=71c9c136-9639-0ce8-7cb7-e545ae00c50f

LB Manager ist not usable

The "Edit worker settings for..." is not visible

- - - - - - - - - - - - - - - - - - - - - - - - -

Browser: Firefox and Chrome

http://192.168.56.90/test-web01/balancer-manager

[Thu Aug 12 11:12:48.035479 2021] [proxy_balancer:error] [pid 709:tid 139675423471360] [client 192.168.56.90:48338] AH10187: ignoring params in balancer-manager cross-site access
[Thu Aug 12 11:12:50.842842 2021] [proxy_balancer:error] [pid 709:tid 139675406685952] [client 192.168.56.90:48338] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager
[Thu Aug 12 11:12:51.653153 2021] [proxy_balancer:error] [pid 709:tid 139675398293248] [client 192.168.56.90:48338] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager?b=test&w=http://192.168.168.130/test&nonce=71c9c136-9639-0ce8-7cb7-e545ae00c50f
[Thu Aug 12 11:12:53.183729 2021] [proxy_balancer:error] [pid 709:tid 139675389900544] [client 192.168.56.90:48338] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager?b=test&w=http://192.168.168.131/test&nonce=71c9c136-9639-0ce8-7cb7-e545ae00c50f
[Thu Aug 12 11:12:53.639131 2021] [proxy_balancer:error] [pid 709:tid 139675501659904] [client 192.168.56.90:48338] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager?b=test&w=http://192.168.168.130/test&nonce=71c9c136-9639-0ce8-7cb7-e545ae00c50f

First connect and every singel klick create an error entry.

LB Manager is not usable

The "Edit worker settings for..." is not visible

=> Conclusion after Update to focal i can't use the LB Manager behind
   that Proxy Bastion Host. Unfortunately for me there is no other
   way to jump in the Production environment.

As i say any suggestion is appreciated is this now a bug (regression) or
is it now a security feature. That will be for me a realy bad news.

Regard Horst

Revision history for this message
Horst Platz (hp-localhorst) wrote :

mybe i thought with a newer apache the problem is solved but with no luck.

LB Manager VM dist-upgrade 21.04 / hirsute

:~# vim /etc/update-manager/release-upgrades
[...]
#Prompt=lts
Prompt=normal

:~# do-release-upgrade
:~# shutdown -r now

:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 21.04
Release: 21.04
Codename: hirsute

:~# dpkg -l | grep apache2
ii apache2 2.4.46-4ubuntu1.1 amd64 Apache HTTP Server
ii apache2-bin 2.4.46-4ubuntu1.1 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.46-4ubuntu1.1 all Apache HTTP Server (common files)
ii apache2-utils 2.4.46-4ubuntu1.1 amd64 Apache HTTP Server (utility programs for web servers)

Browser: Firefox and Chrome

http://192.168.56.90/test-web01/balancer-manager

:~# tail -f /var/log/apache2/management_error.log
[Fri Aug 13 16:56:31.818801 2021] [proxy_balancer:error] [pid 717:tid 140627126277696] [client 192.168.56.90:44938] AH10187: ignoring params in balancer-manager cross-site access
[Fri Aug 13 16:56:34.088176 2021] [proxy_balancer:error] [pid 717:tid 140627117884992] [client 192.168.56.90:44938] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager
[Fri Aug 13 16:56:34.663154 2021] [proxy_balancer:error] [pid 717:tid 140627109492288] [client 192.168.56.90:44938] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.90/test-web01/balancer-manager?b=test&w=http://192.168.168.130/test&nonce=936feca7-14cd-c118-d222-c6b172c9729e

Revision history for this message
Paride Legovini (paride) wrote :

Hello Horst and thanks for your bug report. I think I'm able to reproduce the issue and hopefully I came up with a "more minimal" configuration. As a first step to better understand what's going on here I'd like to check with you that my config does indeed cause the problem.

On a clean Impish system I did the folowing:

--- reproducer (tentative) ---

1. apt install apache2
2. Create/etc/apache2/sites-available/management.conf with:

<VirtualHost 127.0.0.1:80>
    <Location /balancer-manager>
        SetHandler balancer-manager
    </Location>

    LogLevel debug
    ErrorLog ${APACHE_LOG_DIR}/management_error.log
    CustomLog ${APACHE_LOG_DIR}/management_access.log combined
</VirtualHost>

3. a2enmod proxy_balancer
4. a2ensite management
5. systemctl restart apache2

6. With a brower (in my case: Firefox) go to http://127.0.0.1/balancer-manager. A "Load Balancer Manager for 127.0.0.1" page opens.
7. Check /var/log/apache2/management_error.log and verify that is has an error line like:

[client 127.0.0.1:54490] AH10187: ignoring params in balancer-manager cross-site access

which is the problem this bug report is about.

--- end ---

Is this in your opinion a good reproducer? Is there an Ubuntu (or Debian) release where the same set of steps does *not* generate that AH10187 error?

Waiting for your reply I'm marking this bug report as Incomplete; please comment back and we'll look at it again. Thanks!

Note: I didn't mark this report as a duplicate of LP: #1842701 because I'm not confident enough it is the very same problem, but it probably is.

Revision history for this message
Horst Platz (hp-localhorst) wrote :

Hello Paride, thank you for your reply. I'am not sure that your minimal configuration reproduce the hole issue. I reproduce your config with 18.04/bionic and 20.04/focal. What i can say is yes with focal i saw also the error

[...]
[Wed Aug 18 08:22:14.241792 2021] [authz_core:debug] [pid 4199:tid 140445086639872] mod_authz_core.c(817): [client 10.0.2.2:2171] AH01626: authorization result of Require all granted: granted
[Wed Aug 18 08:22:14.241843 2021] [authz_core:debug] [pid 4199:tid 140445086639872] mod_authz_core.c(817): [client 10.0.2.2:2171] AH01626: authorization result of <RequireAny>: granted
[Wed Aug 18 08:22:14.241863 2021] [proxy_balancer:error] [pid 4199:tid 140445086639872] [client 10.0.2.2:2171] AH10187: ignoring params in balancer-manager cross-site access
[Wed Aug 18 08:22:14.241873 2021] [proxy_balancer:debug] [pid 4199:tid 140445086639872] mod_proxy_balancer.c(1451): [client 10.0.2.2:2171] AH01204: genning page
[Wed Aug 18 08:22:14.242740 2021] [deflate:debug] [pid 4199:tid 140445086639872] mod_deflate.c(854): [client 10.0.2.2:2171] AH01384: Zlib: Compressed 994 to 466 : URL /balancer-manager[...]

and with bionic your minimal config the error is not comming up.

[...]
[Wed Aug 18 10:21:10.436566 2021] [authz_core:debug] [pid 2527:tid 139950091626240] mod_authz_core.c(809): [client 10.0.2.2:2161] AH01626: authorization result of Require all granted: granted
[Wed Aug 18 10:21:10.436751 2021] [authz_core:debug] [pid 2527:tid 139950091626240] mod_authz_core.c(809): [client 10.0.2.2:2161] AH01626: authorization result of <RequireAny>: granted
[Wed Aug 18 10:21:10.436795 2021] [proxy_balancer:debug] [pid 2527:tid 139950091626240] mod_proxy_balancer.c(1319): [client 10.0.2.2:2161] AH01204: genning page
[Wed Aug 18 10:21:10.437269 2021] [deflate:debug] [pid 2527:tid 139950091626240] mod_deflate.c(854): [client 10.0.2.2:2161] AH01384: Zlib: Compressed 994 to 465 : URL /balancer-manager
[...]

But with your minimal config the LB Manager is not "functional" it appear no

LoadBalancer Status for ...

or

Edit worker settings for ...

So you can do nothing with the LB Manager.

If i go ahead an create the "second part /etc/apache2/sites-available/proxytest.conf" the problem is when you are on the same box without the proxy in front. You saw that the error cames one time after the first contact. But the LB Manager is fully functional and produces no further error entrys. If you go ahead and use a proxy in front than the LB Manager is not Working and every single klick create a log error.

If you are able to fix focal with your minmal configuration and the error disapear like in bionic mybe this will also fix the hole issue with the proxy in front of it.

If i can do anything more let me know that.

thx horst

Revision history for this message
Horst Platz (hp-localhorst) wrote :

Hello Paride, today i recreate my bigger configuration with Debian 10 and 11

Debian 10 Proxy and Debian 10 LB Manager witch Apache 2.4.38-3+deb10u5 every thing worked as expected no error log entrys. LB Manager over proxy is fully workable.

- - - - - - - - - - - - - - - - - - - - - - - - -

LB Manager Upgrade -> Debian 11 with Apache 2.4.48-3.1+deb11u1

If i go to the LB Manager directly one log entry after first connect

:~# tail -f /var/log/apache2/management_error.log
[Thu Aug 19 12:51:20.867073 2021] [proxy_balancer:error] [pid 459:tid 139790420895488] [client 192.168.56.1:13139] AH10187: ignoring params in balancer-manager cross-site access

and LB Manager is operational.

But if i go over Debian 10 Proxy first connect and every single klick will generate error log entry and LB Manager is not workable.

[Thu Aug 19 12:54:49.382870 2021] [proxy_balancer:error] [pid 459:tid 139790280275712] [client 192.168.56.91:47906] AH10187: ignoring params in balancer-manager cross-site access
[Thu Aug 19 12:54:53.580821 2021] [proxy_balancer:error] [pid 459:tid 139790288668416] [client 192.168.56.91:47906] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.91/test-web01/balancer-manager
[Thu Aug 19 12:54:54.820184 2021] [proxy_balancer:error] [pid 459:tid 139790263490304] [client 192.168.56.91:47906] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.91/test-web01/balancer-manager?b=test&w=http://192.168.168.130/test&nonce=1141264e-846f-cc1c-7876-7f345241b1ea

- - - - - - - - - - - - - - - - - - - - - - - - -

Proxy Upgrade -> Debian 11 with Apache 2.4.48-3.1+deb11u1

Makes no different first connect and single klicks generate logs and LB Manager not functional.

[Thu Aug 19 13:19:12.763111 2021] [proxy_balancer:error] [pid 459:tid 139790179628800] [client 192.168.56.91:55964] AH10187: ignoring params in balancer-manager cross-site access
[Thu Aug 19 13:19:15.498160 2021] [proxy_balancer:error] [pid 459:tid 139790162843392] [client 192.168.56.91:55964] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.91/test-web01/balancer-manager
[Thu Aug 19 13:19:16.154956 2021] [proxy_balancer:error] [pid 459:tid 139790171236096] [client 192.168.56.91:55964] AH10187: ignoring params in balancer-manager cross-site access, referer: http://192.168.56.91/test-web01/balancer-manager?b=test&w=http://192.168.168.130/test&nonce=1141264e-846f-cc1c-7876-7f345241b1ea

Revision history for this message
Horst Platz (hp-localhorst) wrote :

Hello Paride,

any kind of suggestion on that issue.

thx Horst

Revision history for this message
Paride Legovini (paride) wrote :

Hello Horst and sorry for the late reply. I tried again to setup an Ubuntu-based reproducer, but I failed to draw any useful conclusion. A few questions come to my mind at this point:

1. Can you still reproduce the issue with apache2 from Impish or Jammy?

2. You identified https://bz.apache.org/bugzilla/show_bug.cgi?id=63688 as describing this issue in the version of apache2 shipped with Focal. As Hirsute should have a fixed version, but in your experience it does not, I think it's worth checking with the upstream Apache developers once again about the issue.

3. I fail to tell if the behavior you describe is due to a bug or to a local configuration issue. Without first clearly identifying the issue as a bug we can't really begin working on it. If you think there's actually a bug in Ubuntu here, what we need are some minimal but complete steps to reproduce it locally, ideally in LXD containers.

I'm marking this as Incomplete for the moment.

Changed in apache2 (Ubuntu):
status: New → Incomplete
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.