AH00526 when using long ProxyPass worker name

Bug #1668474 reported by Brian Morton on 2017-02-28
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Apache2 Web Server
Fix Released
Critical
apache2 (Debian)
New
Undecided
Unassigned
apache2 (Ubuntu)
Medium
Brian Morton
Trusty
Medium
Unassigned
Xenial
Medium
Unassigned
Yakkety
Medium
Unassigned

Bug Description

When using a long ProxyPass worker name such as unix:///var/php-fpm/146527084714328.sock|fcgi://localhost/home/mysite/domains/subdomain.com/public_html/$1 Apache issues the fatal error AH00526 and refuses to proceed during reload. This is a typical configuration generated by Virtualmin for a subdomain running php-fpm.

A couple of workarounds are available using mod_rewrite, but they do not use connection pooling for the proxy and aren't available for packaged solutions like Virtualmin. The patch from trunk is fairly straightforward.

Hi.

I'm tring to change apache from 2.2 to 2.4.2.
And apachectl reports there is error in my configuration file about proxypass.

-----
# /usr/local/apache24/bin/apachectl -t
AH00548: NameVirtualHost has no effect and will be removed in the next release /usr/local/apache24/conf/httpd.conf:101
AH00526: Syntax error on line 615 of /usr/local/apache24/conf/httpd.conf:
ProxyPass worker name (http://localhost:3128/VirtualHostBase/http/www.hogefuga.com:80/fugafugafile/VirtualHostRoot/++resource++Products.HOGEHOGE.public.stylesheets) too long
-----

I read some sorce files and found there is length limit in worker name.
In mod_proxy.h:305 there is define of length of proxy worker name.

----
#define PROXY_WORKER_MAX_NAME_SIZE 96
----

I think in some casees this size is not enough.
In my case (legth is 177 chars.), I doubled this number and works fine.
Would you please increase this size.

Thank you.

I confirm the bug and second the importance of this bug to be solved.

I can also confirm the bug. This version is completely useless have to downgrade. Even the hack does not work. Just changing the #define PROXY_WORKER_MAX_NAME_SIZE 96 by itself does nothing at least on my side even if you place a number beyond what is needed it still says too long.

We got our way using PT rewrites, eg

# /-> vhmonster proxyreverse because of redirects !
RewriteRule ^/(.*)$ /VirtualHostBase/http/edit.foo.bar.net:80/Plone/VirtualHostRoot/$1 [L,PT]

# for https://issues.apache.org/bugzilla/show_bug.cgi?id=53218, only use a simple worker url
ProxyPass / http://127.0.0.1:8083/ retry=1
ProxyPassReverse / http://127.0.0.1:8083/VirtualHostBase/http/edit.foo.bar.net:80/Plone/VirtualHostRoot/
ProxyPassReverse /zmiroot/ http://127.0.0.1:8083/VirtualHostBase/http/edit.foo.bar.net:80/VirtualHostRoot/_vh_zmiroot/

I can confirm this bug as well on Kubuntu 14.04 LTS 64-bit running Apache 2.4.7. Unfortunately, it is making it impossible for me to serve Plone pages through Apache, which is preventing me from upgrading my production servers to 14.04 LTS. It would be great if this issue could be addressed soon. Thanks.

Alan, you can use rewrites instead of proxypass, even if it is a bug, this workaround which is more another way to do that workaround will let you upgrade.

If you are just setting up a proxy, you may also opt for something like nginx.

I second that. Especially with unix domain sockets allowed in mod_proxy the names can get quite long.

I am looking into whether or not that should be a fatal error... we may be able to get around just reporting it, and still using/accepting a truncated copy.

Maximum worker name length increased by jim to 256 in trunk (and some other limits increased as well). Revisions r1540318, r1621367, r1621372, r1621373, r1621382.
Proposed for backport to 2.4.x.

There's an API compatibility problem with a straight backport proposal. Backport on hold.

256 is not enougth !

There's a workaround available that should be applicable to many situations: combined ProxyPass with RewriteRule [P] flag:

One can do reverse proxying with mod_proxy ProxyPass but also with mod_rewrite RewriteRule [P] flag. The former often is "better", because it uses a pool of persistent connections to the origin server and the characteristics of the connections can be configured in more detail.

What is possible here, is a combination of both approaches: Use a "dummy" ProxyPass with short right side URI to configure a pool to the origin server but RewriteRule for the individual proxy rules containing the long target URLs. As long as the right side of the ProxyPass matches thebeginning of the RewriteRule target URL, the proxying will be done via the connection pool configured with ProxyPass.

An Example:

# This directive must come before the following one in order
# block access to arbitrary URIs on the origin server!
# As an alternative one can also use "RewriteRule /UNUSED - [F]"
ProxyPass /UNUSED !

# Configure a connection pool for the origin server
# http://myserver.example.org:9081
ProxyPass /UNUSED http://myserver.example.org:9081

RewriteEngine On

# Proxy "/long" to a long URI on the origin server,
# [P] flag at end of line is important
RewriteRule /long http://myserver.example.org:9081/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/long.html [P]

# Proxy "/verylong" to an even longer URI on the origin server,
# again [P] flag at end of line is important
RewriteRule /verylong http://myserver.example.org:9081/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc/dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd/verylong.html [P]

Pro:

- you can use long origin server URIs
- the requests are still dispatched to a pool of persistent connections
- You can combine this with ProxyPassReverse and similar directives

Con:

- all requests to this origin server will use the same connection pool, not one pool per target URI. This might be a con in some cases (less separation), in some it might actually be a pro (sharing of resources)
- need a combination of two different syntaxes (harder to understand)
- might not work for any protocol to the origin server (ajp, fcgi, scgi etc.), I simply haven't tested it for those.

if you had read the bug report, you would notice that i posted myself this workaround.

Sorry. I read much more than this bug report (there's also other discussion going on) and overlooked your workaround in comment #3 above.

Note though that

- your recipe opens up the origin server for any URI via the ProxyPass for "/". Often that is not wanted, therefore my recipe contains the additional and important ProxyPass with a "!". This in turn only makes sense by using some "/UNUSED" instead of simply "/". This change has security implications.

- you are using the pass-through flag "PT", I'm using the proxy flag "P". I didn't try "PT" but "P" seems to be more correct. Your mileage may vary.

Regards,

Rainer

Even easier: Create a

<Proxy protocol://server/yourseparationpathprefix>
   SetProxy anyoption
</Proxy>

block in your configuration and you are using a connection pool with RewriteRule [P]

+1 for increasing the URL length limit for proxy targets.

I ran into this recently with an https:// proxy where the hostname was long, causing the URLs to exceed this limit. Using the IP address was not possible because the HTTPS connection refused to handshake because of a mismatched hostname (IP != cert hostname).

The workaround in my case was to use ProxyPassMatch which is non-ideal as it adds a regular expression evaluation (multiple, since I have many ProxyPassMatch directives) to every request, plus I can't be as specific as I need to when mapping certain URLs and not others.

I would normally use mod_jk, but the client required the use of HTTPS between the reverse proxy and application server (Tomcat, in this case).

Changed in apache2:
importance: Unknown → Critical
status: Unknown → Confirmed
Changed in apache2 (Ubuntu):
assignee: nobody → Brian Morton (rokclimb15)
Brian Morton (rokclimb15) wrote :

Patch attached for trusty. Unsure how to generate a debdiff in this case.

The attachment "Patch for trusty" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Robie Basak (racb) wrote :

Brian, thank you for the patch.

Please see https://wiki.ubuntu.com/StableReleaseUpdates for information on getting Trusty updated. If this patch qualifies to land in Trusty, then please follow as much as you can of https://wiki.ubuntu.com/StableReleaseUpdates#Procedure. In particular, we need to make sure this bug is fixed in Zesty first.

It would be also helpful to know if this impacts Xenial, as I don't think it would be appropriate to fix Trusty without also fixing Xenial (otherwise users will regress when they upgrade).

Don't worry about the debdiff. Your patch is simple enough that we can convert it when we're ready to upload.

Changed in apache2 (Ubuntu Trusty):
status: New → Triaged
tags: added: server-next
Brian Morton (rokclimb15) wrote :

Hi Robie, this fix is in Apache SVN trunk but isn't in upstream stable 2.4 release branch yet so all Ubuntu versions will be affected. In addition, no such patch exists in Debian source. https://anonscm.debian.org/cgit/pkg-apache/apache2.git/tree/modules/proxy/mod_proxy.h#n350 and https://anonscm.debian.org/cgit/pkg-apache/apache2.git/tree/debian/patches

I think the best bet is to add it as a debian patch for Zesty and coordinate it with the Debian Apache maintainers until it lands in a stable upstream release. I adjusted the line number in the patch so it should apply cleanly against Zesty.

Robie Basak (racb) wrote :

Hi Brian,

Do you know why this isn't in upstream's 2.4 branch? I'm wondering if it may result in an ABI break or similar issue.

Brian Morton (rokclimb15) wrote :

Hi Robie,

I do not know for certain. One user on the upstream bug mentions it can't be backported because of an API compat issue, but if there's one present I'm sure not seeing it. The define is entirely contained within that header file. The only thing I would think it could affect would be external modules built against mod_proxy.h

root@apache2-mod-proxy-length-xenial:~/apache2-2.4.18# grep -R 'PROXY_WORKER_MAX_NAME_SIZE' *
modules/proxy/mod_proxy.h:#define PROXY_WORKER_MAX_NAME_SIZE 96
modules/proxy/mod_proxy.h:#define PROXY_BALANCER_MAX_NAME_SIZE PROXY_WORKER_MAX_NAME_SIZE
modules/proxy/mod_proxy.h: char name[PROXY_WORKER_MAX_NAME_SIZE];
modules/proxy/mod_proxy.h: char uds_path[PROXY_WORKER_MAX_NAME_SIZE]; /* path to worker's unix domain socket if applicable */
root@apache2-mod-proxy-length-xenial:~/apache2-2.4.18# grep -R 'PROXY_BALANCER_MAX_NAME_SIZE' *
modules/proxy/mod_proxy.h:#define PROXY_BALANCER_MAX_NAME_SIZE PROXY_WORKER_MAX_NAME_SIZE
modules/proxy/mod_proxy.h: char name[PROXY_BALANCER_MAX_NAME_SIZE];
modules/proxy/mod_proxy.h: char sname[PROXY_BALANCER_MAX_NAME_SIZE];

On Wed, Mar 29, 2017 at 01:36:17AM -0000, Brian Morton wrote:
> The only thing I would think it could affect would be
> external modules built against mod_proxy.h

Right - that's my concern.

Brian Morton (rokclimb15) wrote :

I don't think it is an actual incompatibility since the util function for copying those values takes into consideration the size of the dst buffer. As long as it is being lengthened, I think it's OK. It would be a problem if we were reducing it and src were larger than dst.

#define PROXY_STRNCPY(dst, src) ap_proxy_strncpy((dst), (src), (sizeof(dst)))

if (PROXY_STRNCPY(bshared->name, uri) != APR_SUCCESS) {
         return apr_psprintf(p, "balancer name (%s) too long", uri);
}

Robie Basak (racb) wrote :

OK, but why can't that argument be accepted by upstream in their 2.4 branch first?

Brian Morton (rokclimb15) wrote :

An excellent question/point. That should probably get hashed out on the upstream BZ first. The discussion about it was very minimal. I'll get on there and poke at that issue.

Robie Basak (racb) wrote :

Thanks! So I think our position for the time being is to hold this pending upstream's acceptance of the patch into their 2.4 branch. If something changes so we should reconsider, further discussion is welcome.

tags: removed: server-next
Changed in apache2 (Ubuntu):
status: New → Triaged
Changed in apache2 (Ubuntu Xenial):
status: New → Triaged
Changed in apache2 (Ubuntu Yakkety):
status: New → Triaged
Changed in apache2 (Ubuntu):
importance: Undecided → Medium
Changed in apache2 (Ubuntu Trusty):
importance: Undecided → Medium
Changed in apache2 (Ubuntu Yakkety):
importance: Undecided → Medium
Changed in apache2 (Ubuntu Xenial):
importance: Undecided → Medium

AH02808: Alert! worker name ....too long; truncated to:96 characters

ProxyPass / https://somehost.test.com:8080/service=com.test.myservice/version%=0.0.1/routeOffer=default/envContext=TEST/subContext=web/services/myservice/

ProxyPassReverse / https://somehost.test.com:8080/service=com.test.myservice/version%=0.0.1/routeOffer=default/envContext=TEST/subContext=web/services/myservice/

Actual url length is 189 characters. But Proxy pass restricts to 96 characters. How to increase this length ?

This was worked around in v2.4.12 by supporting truncation of the balancer name:

https://svn.apache.org/viewvc?view=revision&revision=1634520

Fixed and released in v2.4.12.

Sorry but no, the situation is even now worsier than before.

For very long urls, that per se those that you can have when reverse proxying zope with VHM urls, truncating may make apache reverse proxying portions of the site that you would not want to expose in the first place.

Truncating may expose "upper" resources and is not at all a solution.

Either there is an error, if it is not technically possible,
Or the proxy proxy EXACTLY where we pointed it.

There is no reason to automatically & silently make an assumption where to proxy and even more on a different location...

But it is NOT silently. It is logged at APLOG_ERR to allow the admin to know that it happened and to decide if it is, or is NOT, a problem.

(In reply to Jim Jagielski from comment #20)
> But it is NOT silently. It is logged at APLOG_ERR to allow the admin to know
> that it happened and to decide if it is, or is NOT, a problem.

So truncation does not have to happen, and most of the time even fail.

Exposing upper resources is most of the time EVIL...

This commit is at best a workaround, and in my opinion a big and dangerous regression.

Of course exposing resources unintentionally is EVIL. That is why when there is the possibility, it is logged so allow the admin, who is the final arbitrator, to determine if they are exposed or not.

The proxypassed ressources, from a security point of view have not to be exposed in another location, if there is a mismatch, it is a fatal configuration error.

The introcuded change is a new (security) bug.

It is not relevant to silently warn the Op behind the scene, this must either proxypass to the aforementioned location or fail, it's as simple as that...

Is there some very sensitive internal structure which cannot be expanded to include some larger, arbitrary number of characters/bytes for the worker-name? If not, can we just get a higher limit on the URL length? Most URL limits in httpd are around 8k characters/bytes, no?

(In reply to Jim Jagielski from comment #22)
> Of course exposing resources unintentionally is EVIL. That is why when there
> is the possibility, it is logged so allow the admin, who is the final
> arbitrator, to determine if they are exposed or not.

Unfortunately, the log is only advisory. httpd continues to start up in what I would describe generously as a "degraded" condition... one where a (likely) larger URL space will be proxied than initially intended.

>
> (In reply to Jim Jagielski from comment #22)
> > Of course exposing resources unintentionally is EVIL. That is why when there
> > is the possibility, it is logged so allow the admin, who is the final
> > arbitrator, to determine if they are exposed or not.
>
> Unfortunately, the log is only advisory. httpd continues to start up in what
> I would describe generously as a "degraded" condition... one where a
> (likely) larger URL space will be proxied than initially intended.

I would completely agree if this would be the case, but it isn't. The error messages tells you that the worker name was truncated. This is unrelated to the request URL that is forwarded. The truncated worker name only means that if you have multiple ProxyPass directives and if the truncation of the worker name leads to the same worker name they use the same connection pool for the backend connections. This is no issue at all, contrawise: It saves resources on the backend.

(In reply to Ruediger Pluem from comment #25)
> >
> > (In reply to Jim Jagielski from comment #22)
> > > Of course exposing resources unintentionally is EVIL. That is why when there
> > > is the possibility, it is logged so allow the admin, who is the final
> > > arbitrator, to determine if they are exposed or not.
> >
> > Unfortunately, the log is only advisory. httpd continues to start up in what
> > I would describe generously as a "degraded" condition... one where a
> > (likely) larger URL space will be proxied than initially intended.
>
> I would completely agree if this would be the case, but it isn't. The error
> messages tells you that the worker name was truncated. This is unrelated to
> the request URL that is forwarded. The truncated worker name only means that
> if you have multiple ProxyPass directives and if the truncation of the
> worker name leads to the same worker name they use the same connection pool
> for the backend connections. This is no issue at all, contrawise: It saves
> resources on the backend.

Thanks for that clarification. To re-phrase, the worker name is being truncated, but incoming URLs will not be truncated for comparison, yes?

Could this become a problem is if the hostname of the origin server is something insanely long? For example, if I want two workers like http://super-long-hostname:80/ and http://super-long-hostname:81/ and those "super-long-hostname" names exceed the worker-name limit, they will be considered the same worker, yes? That means that requests that should go to e.g. port 81 might end up instead going to port 80.

Changed in apache2:
status: Confirmed → Fix Released

(In reply to Christopher Schultz from comment #26)
> (In reply to Ruediger Pluem from comment #25)
> > >
> > > (In reply to Jim Jagielski from comment #22)
> > > > Of course exposing resources unintentionally is EVIL. That is why when there
> > > > is the possibility, it is logged so allow the admin, who is the final
> > > > arbitrator, to determine if they are exposed or not.
> > >
> > > Unfortunately, the log is only advisory. httpd continues to start up in what
> > > I would describe generously as a "degraded" condition... one where a
> > > (likely) larger URL space will be proxied than initially intended.
> >
> > I would completely agree if this would be the case, but it isn't. The error
> > messages tells you that the worker name was truncated. This is unrelated to
> > the request URL that is forwarded. The truncated worker name only means that
> > if you have multiple ProxyPass directives and if the truncation of the
> > worker name leads to the same worker name they use the same connection pool
> > for the backend connections. This is no issue at all, contrawise: It saves
> > resources on the backend.
>
> Thanks for that clarification. To re-phrase, the worker name is being
> truncated, but incoming URLs will not be truncated for comparison, yes?

Correct.

>
> Could this become a problem is if the hostname of the origin server is
> something insanely long? For example, if I want two workers like
> http://super-long-hostname:80/ and http://super-long-hostname:81/ and those
> "super-long-hostname" names exceed the worker-name limit, they will be
> considered the same worker, yes? That means that requests that should go to
> e.g. port 81 might end up instead going to port 80.

Yes, this would be a problem. There is a similar PR (PR62085) on this.
But due to ABI constraints this is not easy to fix in a stable version.

(In reply to Ruediger Pluem from comment #27)
> (In reply to Christopher Schultz from comment #26)
> > Could this become a problem is if the hostname of the origin server is
> > something insanely long? For example, if I want two workers like
> > http://super-long-hostname:80/ and http://super-long-hostname:81/ and those
> > "super-long-hostname" names exceed the worker-name limit, they will be
> > considered the same worker, yes? That means that requests that should go to
> > e.g. port 81 might end up instead going to port 80.
>
> Yes, this would be a problem. There is a similar PR (PR62085) on this.
> But due to ABI constraints this is not easy to fix in a stable version.

How about adding a new ProxyPass parameter called "workername" where you can explicitly set the worker name to something both short AND unique? Or is that best requested as a new enhancement?

*** This bug has been confirmed by popular vote. ***

Bryce Harrington (bryce) on 2020-07-01
Changed in apache2 (Ubuntu Yakkety):
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.