AH02808 when using long (> 96 chars) ProxyPass worker name

Bug #1964829 reported by Sergio Durigan Junior
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Apache2 Web Server
Fix Released
Critical
apache2 (Ubuntu)
Fix Released
Medium
Unassigned
Focal
Won't Fix
Medium
Unassigned
Jammy
Won't Fix
Medium
Unassigned

Bug Description

Note: This is related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1668474

When using a ProxyPass worker name whose length is greater than 96 chars, one will see the following warning on apache2's logs:

Mar 14 21:27:49 test-apache2 systemd[1]: Starting The Apache HTTP Server...
Mar 14 21:27:49 test-apache2 apachectl[1763]: [Mon Mar 14 21:27:49.850040 2022] [proxy:error] [pid 1763:tid 140217439254400] AH02808: Alert! worker name (https://aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaax) too long; truncated to: https://aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Mar 14 21:27:49 test-apache2 systemd[1]: Started The Apache HTTP Server.

Revision history for this message
In , Mu1059 (mu1059) wrote :

Hi.

I'm tring to change apache from 2.2 to 2.4.2.
And apachectl reports there is error in my configuration file about proxypass.

-----
# /usr/local/apache24/bin/apachectl -t
AH00548: NameVirtualHost has no effect and will be removed in the next release /usr/local/apache24/conf/httpd.conf:101
AH00526: Syntax error on line 615 of /usr/local/apache24/conf/httpd.conf:
ProxyPass worker name (http://localhost:3128/VirtualHostBase/http/www.hogefuga.com:80/fugafugafile/VirtualHostRoot/++resource++Products.HOGEHOGE.public.stylesheets) too long
-----

I read some sorce files and found there is length limit in worker name.
In mod_proxy.h:305 there is define of length of proxy worker name.

----
#define PROXY_WORKER_MAX_NAME_SIZE 96
----

I think in some casees this size is not enough.
In my case (legth is 177 chars.), I doubled this number and works fine.
Would you please increase this size.

Thank you.

Revision history for this message
In , kiorky (kiorky) wrote :

I confirm the bug and second the importance of this bug to be solved.

Revision history for this message
In , Masc2279 (masc2279) wrote :

I can also confirm the bug. This version is completely useless have to downgrade. Even the hack does not work. Just changing the #define PROXY_WORKER_MAX_NAME_SIZE 96 by itself does nothing at least on my side even if you place a number beyond what is needed it still says too long.

Revision history for this message
In , kiorky (kiorky) wrote :

We got our way using PT rewrites, eg

# /-> vhmonster proxyreverse because of redirects !
RewriteRule ^/(.*)$ /VirtualHostBase/http/edit.foo.bar.net:80/Plone/VirtualHostRoot/$1 [L,PT]

# for https://issues.apache.org/bugzilla/show_bug.cgi?id=53218, only use a simple worker url
ProxyPass / http://127.0.0.1:8083/ retry=1
ProxyPassReverse / http://127.0.0.1:8083/VirtualHostBase/http/edit.foo.bar.net:80/Plone/VirtualHostRoot/
ProxyPassReverse /zmiroot/ http://127.0.0.1:8083/VirtualHostBase/http/edit.foo.bar.net:80/VirtualHostRoot/_vh_zmiroot/

Revision history for this message
In , Alan (alan-oneill) wrote :

I can confirm this bug as well on Kubuntu 14.04 LTS 64-bit running Apache 2.4.7. Unfortunately, it is making it impossible for me to serve Plone pages through Apache, which is preventing me from upgrading my production servers to 14.04 LTS. It would be great if this issue could be addressed soon. Thanks.

Revision history for this message
In , kiorky (kiorky) wrote :

Alan, you can use rewrites instead of proxypass, even if it is a bug, this workaround which is more another way to do that workaround will let you upgrade.

If you are just setting up a proxy, you may also opt for something like nginx.

Revision history for this message
In , Sebastian Himberger (shimberger) wrote :

I second that. Especially with unix domain sockets allowed in mod_proxy the names can get quite long.

Revision history for this message
In , V-jiz-h (v-jiz-h) wrote :

I am looking into whether or not that should be a fatal error... we may be able to get around just reporting it, and still using/accepting a truncated copy.

Revision history for this message
In , Rainer Jung (rainer-jung-kippdata) wrote :

Maximum worker name length increased by jim to 256 in trunk (and some other limits increased as well). Revisions r1540318, r1621367, r1621372, r1621373, r1621382.
Proposed for backport to 2.4.x.

Revision history for this message
In , Rainer Jung (rainer-jung-kippdata) wrote :

There's an API compatibility problem with a straight backport proposal. Backport on hold.

Revision history for this message
In , kiorky (kiorky) wrote :

256 is not enougth !

Revision history for this message
In , Rainer Jung (rainer-jung-kippdata) wrote :

There's a workaround available that should be applicable to many situations: combined ProxyPass with RewriteRule [P] flag:

One can do reverse proxying with mod_proxy ProxyPass but also with mod_rewrite RewriteRule [P] flag. The former often is "better", because it uses a pool of persistent connections to the origin server and the characteristics of the connections can be configured in more detail.

What is possible here, is a combination of both approaches: Use a "dummy" ProxyPass with short right side URI to configure a pool to the origin server but RewriteRule for the individual proxy rules containing the long target URLs. As long as the right side of the ProxyPass matches thebeginning of the RewriteRule target URL, the proxying will be done via the connection pool configured with ProxyPass.

An Example:

# This directive must come before the following one in order
# block access to arbitrary URIs on the origin server!
# As an alternative one can also use "RewriteRule /UNUSED - [F]"
ProxyPass /UNUSED !

# Configure a connection pool for the origin server
# http://myserver.example.org:9081
ProxyPass /UNUSED http://myserver.example.org:9081

RewriteEngine On

# Proxy "/long" to a long URI on the origin server,
# [P] flag at end of line is important
RewriteRule /long http://myserver.example.org:9081/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/long.html [P]

# Proxy "/verylong" to an even longer URI on the origin server,
# again [P] flag at end of line is important
RewriteRule /verylong http://myserver.example.org:9081/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc/dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd/verylong.html [P]

Pro:

- you can use long origin server URIs
- the requests are still dispatched to a pool of persistent connections
- You can combine this with ProxyPassReverse and similar directives

Con:

- all requests to this origin server will use the same connection pool, not one pool per target URI. This might be a con in some cases (less separation), in some it might actually be a pro (sharing of resources)
- need a combination of two different syntaxes (harder to understand)
- might not work for any protocol to the origin server (ajp, fcgi, scgi etc.), I simply haven't tested it for those.

Revision history for this message
In , kiorky (kiorky) wrote :

if you had read the bug report, you would notice that i posted myself this workaround.

Revision history for this message
In , Rainer Jung (rainer-jung-kippdata) wrote :

Sorry. I read much more than this bug report (there's also other discussion going on) and overlooked your workaround in comment #3 above.

Note though that

- your recipe opens up the origin server for any URI via the ProxyPass for "/". Often that is not wanted, therefore my recipe contains the additional and important ProxyPass with a "!". This in turn only makes sense by using some "/UNUSED" instead of simply "/". This change has security implications.

- you are using the pass-through flag "PT", I'm using the proxy flag "P". I didn't try "PT" but "P" seems to be more correct. Your mileage may vary.

Regards,

Rainer

Revision history for this message
In , Rpluem (rpluem) wrote :

Even easier: Create a

<Proxy protocol://server/yourseparationpathprefix>
   SetProxy anyoption
</Proxy>

block in your configuration and you are using a connection pool with RewriteRule [P]

Revision history for this message
In , Chris-christopherschultz (chris-christopherschultz) wrote :

+1 for increasing the URL length limit for proxy targets.

I ran into this recently with an https:// proxy where the hostname was long, causing the URLs to exceed this limit. Using the IP address was not possible because the HTTPS connection refused to handshake because of a mismatched hostname (IP != cert hostname).

The workaround in my case was to use ProxyPassMatch which is non-ideal as it adds a regular expression evaluation (multiple, since I have many ProxyPassMatch directives) to every request, plus I can't be as specific as I need to when mapping certain URLs and not others.

I would normally use mod_jk, but the client required the use of HTTPS between the reverse proxy and application server (Tomcat, in this case).

Revision history for this message
In , Emailranjithr (emailranjithr) wrote :

AH02808: Alert! worker name ....too long; truncated to:96 characters

ProxyPass / https://somehost.test.com:8080/service=com.test.myservice/version%=0.0.1/routeOffer=default/envContext=TEST/subContext=web/services/myservice/

ProxyPassReverse / https://somehost.test.com:8080/service=com.test.myservice/version%=0.0.1/routeOffer=default/envContext=TEST/subContext=web/services/myservice/

Actual url length is 189 characters. But Proxy pass restricts to 96 characters. How to increase this length ?

Revision history for this message
In , Graham Leggett (minfrin-y) wrote :

This was worked around in v2.4.12 by supporting truncation of the balancer name:

https://svn.apache.org/viewvc?view=revision&revision=1634520

Revision history for this message
In , Graham Leggett (minfrin-y) wrote :

Fixed and released in v2.4.12.

Revision history for this message
In , kiorky (kiorky) wrote :

Sorry but no, the situation is even now worsier than before.

For very long urls, that per se those that you can have when reverse proxying zope with VHM urls, truncating may make apache reverse proxying portions of the site that you would not want to expose in the first place.

Truncating may expose "upper" resources and is not at all a solution.

Either there is an error, if it is not technically possible,
Or the proxy proxy EXACTLY where we pointed it.

There is no reason to automatically & silently make an assumption where to proxy and even more on a different location...

Revision history for this message
In , V-jiz-h (v-jiz-h) wrote :

But it is NOT silently. It is logged at APLOG_ERR to allow the admin to know that it happened and to decide if it is, or is NOT, a problem.

Revision history for this message
In , kiorky (kiorky) wrote :

(In reply to Jim Jagielski from comment #20)
> But it is NOT silently. It is logged at APLOG_ERR to allow the admin to know
> that it happened and to decide if it is, or is NOT, a problem.

So truncation does not have to happen, and most of the time even fail.

Exposing upper resources is most of the time EVIL...

This commit is at best a workaround, and in my opinion a big and dangerous regression.

Revision history for this message
In , V-jiz-h (v-jiz-h) wrote :

Of course exposing resources unintentionally is EVIL. That is why when there is the possibility, it is logged so allow the admin, who is the final arbitrator, to determine if they are exposed or not.

Revision history for this message
In , kiorky (kiorky) wrote :

The proxypassed ressources, from a security point of view have not to be exposed in another location, if there is a mismatch, it is a fatal configuration error.

The introcuded change is a new (security) bug.

It is not relevant to silently warn the Op behind the scene, this must either proxypass to the aforementioned location or fail, it's as simple as that...

Revision history for this message
In , Chris-christopherschultz (chris-christopherschultz) wrote :

Is there some very sensitive internal structure which cannot be expanded to include some larger, arbitrary number of characters/bytes for the worker-name? If not, can we just get a higher limit on the URL length? Most URL limits in httpd are around 8k characters/bytes, no?

(In reply to Jim Jagielski from comment #22)
> Of course exposing resources unintentionally is EVIL. That is why when there
> is the possibility, it is logged so allow the admin, who is the final
> arbitrator, to determine if they are exposed or not.

Unfortunately, the log is only advisory. httpd continues to start up in what I would describe generously as a "degraded" condition... one where a (likely) larger URL space will be proxied than initially intended.

Revision history for this message
In , Rpluem (rpluem) wrote :

>
> (In reply to Jim Jagielski from comment #22)
> > Of course exposing resources unintentionally is EVIL. That is why when there
> > is the possibility, it is logged so allow the admin, who is the final
> > arbitrator, to determine if they are exposed or not.
>
> Unfortunately, the log is only advisory. httpd continues to start up in what
> I would describe generously as a "degraded" condition... one where a
> (likely) larger URL space will be proxied than initially intended.

I would completely agree if this would be the case, but it isn't. The error messages tells you that the worker name was truncated. This is unrelated to the request URL that is forwarded. The truncated worker name only means that if you have multiple ProxyPass directives and if the truncation of the worker name leads to the same worker name they use the same connection pool for the backend connections. This is no issue at all, contrawise: It saves resources on the backend.

Revision history for this message
In , Chris-christopherschultz (chris-christopherschultz) wrote :

(In reply to Ruediger Pluem from comment #25)
> >
> > (In reply to Jim Jagielski from comment #22)
> > > Of course exposing resources unintentionally is EVIL. That is why when there
> > > is the possibility, it is logged so allow the admin, who is the final
> > > arbitrator, to determine if they are exposed or not.
> >
> > Unfortunately, the log is only advisory. httpd continues to start up in what
> > I would describe generously as a "degraded" condition... one where a
> > (likely) larger URL space will be proxied than initially intended.
>
> I would completely agree if this would be the case, but it isn't. The error
> messages tells you that the worker name was truncated. This is unrelated to
> the request URL that is forwarded. The truncated worker name only means that
> if you have multiple ProxyPass directives and if the truncation of the
> worker name leads to the same worker name they use the same connection pool
> for the backend connections. This is no issue at all, contrawise: It saves
> resources on the backend.

Thanks for that clarification. To re-phrase, the worker name is being truncated, but incoming URLs will not be truncated for comparison, yes?

Could this become a problem is if the hostname of the origin server is something insanely long? For example, if I want two workers like http://super-long-hostname:80/ and http://super-long-hostname:81/ and those "super-long-hostname" names exceed the worker-name limit, they will be considered the same worker, yes? That means that requests that should go to e.g. port 81 might end up instead going to port 80.

Revision history for this message
In , Rpluem (rpluem) wrote :

(In reply to Christopher Schultz from comment #26)
> (In reply to Ruediger Pluem from comment #25)
> > >
> > > (In reply to Jim Jagielski from comment #22)
> > > > Of course exposing resources unintentionally is EVIL. That is why when there
> > > > is the possibility, it is logged so allow the admin, who is the final
> > > > arbitrator, to determine if they are exposed or not.
> > >
> > > Unfortunately, the log is only advisory. httpd continues to start up in what
> > > I would describe generously as a "degraded" condition... one where a
> > > (likely) larger URL space will be proxied than initially intended.
> >
> > I would completely agree if this would be the case, but it isn't. The error
> > messages tells you that the worker name was truncated. This is unrelated to
> > the request URL that is forwarded. The truncated worker name only means that
> > if you have multiple ProxyPass directives and if the truncation of the
> > worker name leads to the same worker name they use the same connection pool
> > for the backend connections. This is no issue at all, contrawise: It saves
> > resources on the backend.
>
> Thanks for that clarification. To re-phrase, the worker name is being
> truncated, but incoming URLs will not be truncated for comparison, yes?

Correct.

>
> Could this become a problem is if the hostname of the origin server is
> something insanely long? For example, if I want two workers like
> http://super-long-hostname:80/ and http://super-long-hostname:81/ and those
> "super-long-hostname" names exceed the worker-name limit, they will be
> considered the same worker, yes? That means that requests that should go to
> e.g. port 81 might end up instead going to port 80.

Yes, this would be a problem. There is a similar PR (PR62085) on this.
But due to ABI constraints this is not easy to fix in a stable version.

Revision history for this message
In , Chris-christopherschultz (chris-christopherschultz) wrote :

(In reply to Ruediger Pluem from comment #27)
> (In reply to Christopher Schultz from comment #26)
> > Could this become a problem is if the hostname of the origin server is
> > something insanely long? For example, if I want two workers like
> > http://super-long-hostname:80/ and http://super-long-hostname:81/ and those
> > "super-long-hostname" names exceed the worker-name limit, they will be
> > considered the same worker, yes? That means that requests that should go to
> > e.g. port 81 might end up instead going to port 80.
>
> Yes, this would be a problem. There is a similar PR (PR62085) on this.
> But due to ABI constraints this is not easy to fix in a stable version.

How about adding a new ProxyPass parameter called "workername" where you can explicitly set the worker name to something both short AND unique? Or is that best requested as a new enhancement?

Revision history for this message
In , Chris-christopherschultz (chris-christopherschultz) wrote :

*** This bug has been confirmed by popular vote. ***

Revision history for this message
In , Operatingknowledge (operatingknowledge) wrote :

Hi.

It seems that this problem is still present in 2.4.48.
Looking to mod_proxy.h, PROXY_WORKER_MAX_NAME_SIZE & PROXY_BALANCER_MAX_NAME_SIZE are still set to 96 char long, causing some "AH02808 Alert! worker name * too long; truncated to *" in our logs

Having a lot of Proxy conf, injected at runtime by other team, stop/start logs are dumping hundreds of these warning lines, though and as far as I can understand, do no arm to Apache runtime behavior.

It would be great if a "reasonnable" longer max length could be set (256?)

BR

Revision history for this message
In , Ylavic-dev (ylavic-dev) wrote :
Revision history for this message
In , Ylavic-dev (ylavic-dev) wrote :

Backported to 2.4.x (r1898588), will be in 2.4.53.

Revision history for this message
In , Lafossas (lafossas) wrote :

Hello,
I am on httpd 2.4.6_97.el7_9.2 and I have this error

httpd[46619]: ProxyPass worker hostname (web-xxxxx-xxxxxx-xxxxxxxxxxxx-xxxx-xxxx.xx-xxx.xxx.xxxxxxxxxx.xx) too long

can someone tell me when this error will be fixed on httpd?

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :
Changed in apache2 (Ubuntu Focal):
status: New → Triaged
importance: Undecided → Medium
Changed in apache2:
importance: Unknown → Critical
status: Unknown → Fix Released
Revision history for this message
Robie Basak (racb) wrote :

According to https://bz.apache.org/bugzilla/show_bug.cgi?id=53218 this was fixed upstream in 2.4.53. Kinetic has 2.4.54-2ubuntu1 so this is presumed fixed in Kinetic, and bug tasks remain open for Focal and Jammy.

Changed in apache2 (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Robie Basak (racb) wrote :

This looks like it might contain an ABI change, so I'm not sure how practical it would be to backport to Focal and Jammy.

Revision history for this message
Athos Ribeiro (athos-ribeiro) wrote :

In this case, should this be closed as WONTFIX?

tags: added: server-triage-discuss
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

apache2-dev ships mod_proxy.h, which means that the proxy_worker_shared struct is part of the public API/ABI. As Robie mentioned, the upstream patch that fixes this issue introduces a new field at the end of the struct mentioned above, which unfortunately characterizes an ABI breakage, especially because the struct is not packed.

+1 for closing it as WONTFIX. I don't know if it's possible to amend Jammy's/Focal's release notes and include a mention there about this bug, why we're not going to address it, and a recommendation to avoid ProxyPass worker names longer than 96 characters.

Changed in apache2 (Ubuntu Focal):
status: Triaged → Won't Fix
Changed in apache2 (Ubuntu Jammy):
status: Triaged → Won't Fix
tags: removed: server-triage-discuss
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.