AH00526 when using long ProxyPass worker name

Bug #1668474 reported by Brian Morton
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Apache2 Web Server
Fix Released
Critical
apache2 (Debian)
New
Undecided
Unassigned
apache2 (Ubuntu)
Medium
Brian Morton
Trusty
Medium
Unassigned
Xenial
Medium
Unassigned
Yakkety
Medium
Unassigned

Bug Description

When using a long ProxyPass worker name such as unix:///var/php-fpm/146527084714328.sock|fcgi://localhost/home/mysite/domains/subdomain.com/public_html/$1 Apache issues the fatal error AH00526 and refuses to proceed during reload. This is a typical configuration generated by Virtualmin for a subdomain running php-fpm.

A couple of workarounds are available using mod_rewrite, but they do not use connection pooling for the proxy and aren't available for packaged solutions like Virtualmin. The patch from trunk is fairly straightforward.

Revision history for this message
In , Mu1059 (mu1059) wrote :

Hi.

I'm tring to change apache from 2.2 to 2.4.2.
And apachectl reports there is error in my configuration file about proxypass.

-----
# /usr/local/apache24/bin/apachectl -t
AH00548: NameVirtualHost has no effect and will be removed in the next release /usr/local/apache24/conf/httpd.conf:101
AH00526: Syntax error on line 615 of /usr/local/apache24/conf/httpd.conf:
ProxyPass worker name (http://localhost:3128/VirtualHostBase/http/www.hogefuga.com:80/fugafugafile/VirtualHostRoot/++resource++Products.HOGEHOGE.public.stylesheets) too long
-----

I read some sorce files and found there is length limit in worker name.
In mod_proxy.h:305 there is define of length of proxy worker name.

----
#define PROXY_WORKER_MAX_NAME_SIZE 96
----

I think in some casees this size is not enough.
In my case (legth is 177 chars.), I doubled this number and works fine.
Would you please increase this size.

Thank you.

Revision history for this message
In , kiorky (kiorky) wrote :

I confirm the bug and second the importance of this bug to be solved.

Revision history for this message
In , Masc2279 (masc2279) wrote :

I can also confirm the bug. This version is completely useless have to downgrade. Even the hack does not work. Just changing the #define PROXY_WORKER_MAX_NAME_SIZE 96 by itself does nothing at least on my side even if you place a number beyond what is needed it still says too long.

Revision history for this message
In , kiorky (kiorky) wrote :

We got our way using PT rewrites, eg

# /-> vhmonster proxyreverse because of redirects !
RewriteRule ^/(.*)$ /VirtualHostBase/http/edit.foo.bar.net:80/Plone/VirtualHostRoot/$1 [L,PT]

# for https://issues.apache.org/bugzilla/show_bug.cgi?id=53218, only use a simple worker url
ProxyPass / http://127.0.0.1:8083/ retry=1
ProxyPassReverse / http://127.0.0.1:8083/VirtualHostBase/http/edit.foo.bar.net:80/Plone/VirtualHostRoot/
ProxyPassReverse /zmiroot/ http://127.0.0.1:8083/VirtualHostBase/http/edit.foo.bar.net:80/VirtualHostRoot/_vh_zmiroot/

Revision history for this message
In , Alan (alan-oneill) wrote :

I can confirm this bug as well on Kubuntu 14.04 LTS 64-bit running Apache 2.4.7. Unfortunately, it is making it impossible for me to serve Plone pages through Apache, which is preventing me from upgrading my production servers to 14.04 LTS. It would be great if this issue could be addressed soon. Thanks.

Revision history for this message
In , kiorky (kiorky) wrote :

Alan, you can use rewrites instead of proxypass, even if it is a bug, this workaround which is more another way to do that workaround will let you upgrade.

If you are just setting up a proxy, you may also opt for something like nginx.

Revision history for this message
In , Sebastian Himberger (shimberger) wrote :

I second that. Especially with unix domain sockets allowed in mod_proxy the names can get quite long.

Revision history for this message
In , V-jiz-h (v-jiz-h) wrote :

I am looking into whether or not that should be a fatal error... we may be able to get around just reporting it, and still using/accepting a truncated copy.

Revision history for this message
In , Rainer Jung (rainer-jung-kippdata) wrote :

Maximum worker name length increased by jim to 256 in trunk (and some other limits increased as well). Revisions r1540318, r1621367, r1621372, r1621373, r1621382.
Proposed for backport to 2.4.x.

Revision history for this message
In , Rainer Jung (rainer-jung-kippdata) wrote :

There's an API compatibility problem with a straight backport proposal. Backport on hold.

Revision history for this message
In , kiorky (kiorky) wrote :

256 is not enougth !

Revision history for this message
In , Rainer Jung (rainer-jung-kippdata) wrote :

There's a workaround available that should be applicable to many situations: combined ProxyPass with RewriteRule [P] flag:

One can do reverse proxying with mod_proxy ProxyPass but also with mod_rewrite RewriteRule [P] flag. The former often is "better", because it uses a pool of persistent connections to the origin server and the characteristics of the connections can be configured in more detail.

What is possible here, is a combination of both approaches: Use a "dummy" ProxyPass with short right side URI to configure a pool to the origin server but RewriteRule for the individual proxy rules containing the long target URLs. As long as the right side of the ProxyPass matches thebeginning of the RewriteRule target URL, the proxying will be done via the connection pool configured with ProxyPass.

An Example:

# This directive must come before the following one in order
# block access to arbitrary URIs on the origin server!
# As an alternative one can also use "RewriteRule /UNUSED - [F]"
ProxyPass /UNUSED !

# Configure a connection pool for the origin server
# http://myserver.example.org:9081
ProxyPass /UNUSED http://myserver.example.org:9081

RewriteEngine On

# Proxy "/long" to a long URI on the origin server,
# [P] flag at end of line is important
RewriteRule /long http://myserver.example.org:9081/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/long.html [P]

# Proxy "/verylong" to an even longer URI on the origin server,
# again [P] flag at end of line is important
RewriteRule /verylong http://myserver.example.org:9081/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc/dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd/verylong.html [P]

Pro:

- you can use long origin server URIs
- the requests are still dispatched to a pool of persistent connections
- You can combine this with ProxyPassReverse and similar directives

Con:

- all requests to this origin server will use the same connection pool, not one pool per target URI. This might be a con in some cases (less separation), in some it might actually be a pro (sharing of resources)
- need a combination of two different syntaxes (harder to understand)
- might not work for any protocol to the origin server (ajp, fcgi, scgi etc.), I simply haven't tested it for those.

Revision history for this message
In , kiorky (kiorky) wrote :

if you had read the bug report, you would notice that i posted myself this workaround.

Revision history for this message
In , Rainer Jung (rainer-jung-kippdata) wrote :

Sorry. I read much more than this bug report (there's also other discussion going on) and overlooked your workaround in comment #3 above.

Note though that

- your recipe opens up the origin server for any URI via the ProxyPass for "/". Often that is not wanted, therefore my recipe contains the additional and important ProxyPass with a "!". This in turn only makes sense by using some "/UNUSED" instead of simply "/". This change has security implications.

- you are using the pass-through flag "PT", I'm using the proxy flag "P". I didn't try "PT" but "P" seems to be more correct. Your mileage may vary.

Regards,

Rainer

Revision history for this message
In , Rpluem (rpluem) wrote :

Even easier: Create a

<Proxy protocol://server/yourseparationpathprefix>
   SetProxy anyoption
</Proxy>

block in your configuration and you are using a connection pool with RewriteRule [P]

Revision history for this message
In , Chris-christopherschultz (chris-christopherschultz) wrote :

+1 for increasing the URL length limit for proxy targets.

I ran into this recently with an https:// proxy where the hostname was long, causing the URLs to exceed this limit. Using the IP address was not possible because the HTTPS connection refused to handshake because of a mismatched hostname (IP != cert hostname).

The workaround in my case was to use ProxyPassMatch which is non-ideal as it adds a regular expression evaluation (multiple, since I have many ProxyPassMatch directives) to every request, plus I can't be as specific as I need to when mapping certain URLs and not others.

I would normally use mod_jk, but the client required the use of HTTPS between the reverse proxy and application server (Tomcat, in this case).

Changed in apache2:
importance: Unknown → Critical
status: Unknown → Confirmed
Changed in apache2 (Ubuntu):
assignee: nobody → Brian Morton (rokclimb15)
Revision history for this message
Brian Morton (rokclimb15) wrote :

Patch attached for trusty. Unsure how to generate a debdiff in this case.

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "Patch for trusty" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Revision history for this message
Robie Basak (racb) wrote :

Brian, thank you for the patch.

Please see https://wiki.ubuntu.com/StableReleaseUpdates for information on getting Trusty updated. If this patch qualifies to land in Trusty, then please follow as much as you can of https://wiki.ubuntu.com/StableReleaseUpdates#Procedure. In particular, we need to make sure this bug is fixed in Zesty first.

It would be also helpful to know if this impacts Xenial, as I don't think it would be appropriate to fix Trusty without also fixing Xenial (otherwise users will regress when they upgrade).

Don't worry about the debdiff. Your patch is simple enough that we can convert it when we're ready to upload.

Changed in apache2 (Ubuntu Trusty):
status: New → Triaged
tags: added: server-next
Revision history for this message
Brian Morton (rokclimb15) wrote :

Hi Robie, this fix is in Apache SVN trunk but isn't in upstream stable 2.4 release branch yet so all Ubuntu versions will be affected. In addition, no such patch exists in Debian source. https://anonscm.debian.org/cgit/pkg-apache/apache2.git/tree/modules/proxy/mod_proxy.h#n350 and https://anonscm.debian.org/cgit/pkg-apache/apache2.git/tree/debian/patches

I think the best bet is to add it as a debian patch for Zesty and coordinate it with the Debian Apache maintainers until it lands in a stable upstream release. I adjusted the line number in the patch so it should apply cleanly against Zesty.

Revision history for this message
Robie Basak (racb) wrote :

Hi Brian,

Do you know why this isn't in upstream's 2.4 branch? I'm wondering if it may result in an ABI break or similar issue.

Revision history for this message
Brian Morton (rokclimb15) wrote :

Hi Robie,

I do not know for certain. One user on the upstream bug mentions it can't be backported because of an API compat issue, but if there's one present I'm sure not seeing it. The define is entirely contained within that header file. The only thing I would think it could affect would be external modules built against mod_proxy.h

root@apache2-mod-proxy-length-xenial:~/apache2-2.4.18# grep -R 'PROXY_WORKER_MAX_NAME_SIZE' *
modules/proxy/mod_proxy.h:#define PROXY_WORKER_MAX_NAME_SIZE 96
modules/proxy/mod_proxy.h:#define PROXY_BALANCER_MAX_NAME_SIZE PROXY_WORKER_MAX_NAME_SIZE
modules/proxy/mod_proxy.h: char name[PROXY_WORKER_MAX_NAME_SIZE];
modules/proxy/mod_proxy.h: char uds_path[PROXY_WORKER_MAX_NAME_SIZE]; /* path to worker's unix domain socket if applicable */
root@apache2-mod-proxy-length-xenial:~/apache2-2.4.18# grep -R 'PROXY_BALANCER_MAX_NAME_SIZE' *
modules/proxy/mod_proxy.h:#define PROXY_BALANCER_MAX_NAME_SIZE PROXY_WORKER_MAX_NAME_SIZE
modules/proxy/mod_proxy.h: char name[PROXY_BALANCER_MAX_NAME_SIZE];
modules/proxy/mod_proxy.h: char sname[PROXY_BALANCER_MAX_NAME_SIZE];

Revision history for this message
Robie Basak (racb) wrote : Re: [Bug 1668474] Re: AH00526 when using long ProxyPass worker name

On Wed, Mar 29, 2017 at 01:36:17AM -0000, Brian Morton wrote:
> The only thing I would think it could affect would be
> external modules built against mod_proxy.h

Right - that's my concern.

Revision history for this message
Brian Morton (rokclimb15) wrote :

I don't think it is an actual incompatibility since the util function for copying those values takes into consideration the size of the dst buffer. As long as it is being lengthened, I think it's OK. It would be a problem if we were reducing it and src were larger than dst.

#define PROXY_STRNCPY(dst, src) ap_proxy_strncpy((dst), (src), (sizeof(dst)))

if (PROXY_STRNCPY(bshared->name, uri) != APR_SUCCESS) {
         return apr_psprintf(p, "balancer name (%s) too long", uri);
}

Revision history for this message
Robie Basak (racb) wrote :

OK, but why can't that argument be accepted by upstream in their 2.4 branch first?

Revision history for this message
Brian Morton (rokclimb15) wrote :

An excellent question/point. That should probably get hashed out on the upstream BZ first. The discussion about it was very minimal. I'll get on there and poke at that issue.

Revision history for this message
Robie Basak (racb) wrote :

Thanks! So I think our position for the time being is to hold this pending upstream's acceptance of the patch into their 2.4 branch. If something changes so we should reconsider, further discussion is welcome.

tags: removed: server-next
Changed in apache2 (Ubuntu):
status: New → Triaged
Changed in apache2 (Ubuntu Xenial):
status: New → Triaged
Changed in apache2 (Ubuntu Yakkety):
status: New → Triaged
Changed in apache2 (Ubuntu):
importance: Undecided → Medium
Changed in apache2 (Ubuntu Trusty):
importance: Undecided → Medium
Changed in apache2 (Ubuntu Yakkety):
importance: Undecided → Medium
Changed in apache2 (Ubuntu Xenial):
importance: Undecided → Medium
Revision history for this message
In , Emailranjithr (emailranjithr) wrote :

AH02808: Alert! worker name ....too long; truncated to:96 characters

ProxyPass / https://somehost.test.com:8080/service=com.test.myservice/version%=0.0.1/routeOffer=default/envContext=TEST/subContext=web/services/myservice/

ProxyPassReverse / https://somehost.test.com:8080/service=com.test.myservice/version%=0.0.1/routeOffer=default/envContext=TEST/subContext=web/services/myservice/

Actual url length is 189 characters. But Proxy pass restricts to 96 characters. How to increase this length ?

Revision history for this message
In , Graham Leggett (minfrin-y) wrote :

This was worked around in v2.4.12 by supporting truncation of the balancer name:

https://svn.apache.org/viewvc?view=revision&revision=1634520

Revision history for this message
In , Graham Leggett (minfrin-y) wrote :

Fixed and released in v2.4.12.

Revision history for this message
In , kiorky (kiorky) wrote :

Sorry but no, the situation is even now worsier than before.

For very long urls, that per se those that you can have when reverse proxying zope with VHM urls, truncating may make apache reverse proxying portions of the site that you would not want to expose in the first place.

Truncating may expose "upper" resources and is not at all a solution.

Either there is an error, if it is not technically possible,
Or the proxy proxy EXACTLY where we pointed it.

There is no reason to automatically & silently make an assumption where to proxy and even more on a different location...

Revision history for this message
In , V-jiz-h (v-jiz-h) wrote :

But it is NOT silently. It is logged at APLOG_ERR to allow the admin to know that it happened and to decide if it is, or is NOT, a problem.

Revision history for this message
In , kiorky (kiorky) wrote :

(In reply to Jim Jagielski from comment #20)
> But it is NOT silently. It is logged at APLOG_ERR to allow the admin to know
> that it happened and to decide if it is, or is NOT, a problem.

So truncation does not have to happen, and most of the time even fail.

Exposing upper resources is most of the time EVIL...

This commit is at best a workaround, and in my opinion a big and dangerous regression.

Revision history for this message
In , V-jiz-h (v-jiz-h) wrote :

Of course exposing resources unintentionally is EVIL. That is why when there is the possibility, it is logged so allow the admin, who is the final arbitrator, to determine if they are exposed or not.

Revision history for this message
In , kiorky (kiorky) wrote :

The proxypassed ressources, from a security point of view have not to be exposed in another location, if there is a mismatch, it is a fatal configuration error.

The introcuded change is a new (security) bug.

It is not relevant to silently warn the Op behind the scene, this must either proxypass to the aforementioned location or fail, it's as simple as that...

Revision history for this message
In , Chris-christopherschultz (chris-christopherschultz) wrote :

Is there some very sensitive internal structure which cannot be expanded to include some larger, arbitrary number of characters/bytes for the worker-name? If not, can we just get a higher limit on the URL length? Most URL limits in httpd are around 8k characters/bytes, no?

(In reply to Jim Jagielski from comment #22)
> Of course exposing resources unintentionally is EVIL. That is why when there
> is the possibility, it is logged so allow the admin, who is the final
> arbitrator, to determine if they are exposed or not.

Unfortunately, the log is only advisory. httpd continues to start up in what I would describe generously as a "degraded" condition... one where a (likely) larger URL space will be proxied than initially intended.

Revision history for this message
In , Rpluem (rpluem) wrote :

>
> (In reply to Jim Jagielski from comment #22)
> > Of course exposing resources unintentionally is EVIL. That is why when there
> > is the possibility, it is logged so allow the admin, who is the final
> > arbitrator, to determine if they are exposed or not.
>
> Unfortunately, the log is only advisory. httpd continues to start up in what
> I would describe generously as a "degraded" condition... one where a
> (likely) larger URL space will be proxied than initially intended.

I would completely agree if this would be the case, but it isn't. The error messages tells you that the worker name was truncated. This is unrelated to the request URL that is forwarded. The truncated worker name only means that if you have multiple ProxyPass directives and if the truncation of the worker name leads to the same worker name they use the same connection pool for the backend connections. This is no issue at all, contrawise: It saves resources on the backend.

Revision history for this message
In , Chris-christopherschultz (chris-christopherschultz) wrote :

(In reply to Ruediger Pluem from comment #25)
> >
> > (In reply to Jim Jagielski from comment #22)
> > > Of course exposing resources unintentionally is EVIL. That is why when there
> > > is the possibility, it is logged so allow the admin, who is the final
> > > arbitrator, to determine if they are exposed or not.
> >
> > Unfortunately, the log is only advisory. httpd continues to start up in what
> > I would describe generously as a "degraded" condition... one where a
> > (likely) larger URL space will be proxied than initially intended.
>
> I would completely agree if this would be the case, but it isn't. The error
> messages tells you that the worker name was truncated. This is unrelated to
> the request URL that is forwarded. The truncated worker name only means that
> if you have multiple ProxyPass directives and if the truncation of the
> worker name leads to the same worker name they use the same connection pool
> for the backend connections. This is no issue at all, contrawise: It saves
> resources on the backend.

Thanks for that clarification. To re-phrase, the worker name is being truncated, but incoming URLs will not be truncated for comparison, yes?

Could this become a problem is if the hostname of the origin server is something insanely long? For example, if I want two workers like http://super-long-hostname:80/ and http://super-long-hostname:81/ and those "super-long-hostname" names exceed the worker-name limit, they will be considered the same worker, yes? That means that requests that should go to e.g. port 81 might end up instead going to port 80.

Changed in apache2:
status: Confirmed → Fix Released
Revision history for this message
In , Rpluem (rpluem) wrote :

(In reply to Christopher Schultz from comment #26)
> (In reply to Ruediger Pluem from comment #25)
> > >
> > > (In reply to Jim Jagielski from comment #22)
> > > > Of course exposing resources unintentionally is EVIL. That is why when there
> > > > is the possibility, it is logged so allow the admin, who is the final
> > > > arbitrator, to determine if they are exposed or not.
> > >
> > > Unfortunately, the log is only advisory. httpd continues to start up in what
> > > I would describe generously as a "degraded" condition... one where a
> > > (likely) larger URL space will be proxied than initially intended.
> >
> > I would completely agree if this would be the case, but it isn't. The error
> > messages tells you that the worker name was truncated. This is unrelated to
> > the request URL that is forwarded. The truncated worker name only means that
> > if you have multiple ProxyPass directives and if the truncation of the
> > worker name leads to the same worker name they use the same connection pool
> > for the backend connections. This is no issue at all, contrawise: It saves
> > resources on the backend.
>
> Thanks for that clarification. To re-phrase, the worker name is being
> truncated, but incoming URLs will not be truncated for comparison, yes?

Correct.

>
> Could this become a problem is if the hostname of the origin server is
> something insanely long? For example, if I want two workers like
> http://super-long-hostname:80/ and http://super-long-hostname:81/ and those
> "super-long-hostname" names exceed the worker-name limit, they will be
> considered the same worker, yes? That means that requests that should go to
> e.g. port 81 might end up instead going to port 80.

Yes, this would be a problem. There is a similar PR (PR62085) on this.
But due to ABI constraints this is not easy to fix in a stable version.

Revision history for this message
In , Chris-christopherschultz (chris-christopherschultz) wrote :

(In reply to Ruediger Pluem from comment #27)
> (In reply to Christopher Schultz from comment #26)
> > Could this become a problem is if the hostname of the origin server is
> > something insanely long? For example, if I want two workers like
> > http://super-long-hostname:80/ and http://super-long-hostname:81/ and those
> > "super-long-hostname" names exceed the worker-name limit, they will be
> > considered the same worker, yes? That means that requests that should go to
> > e.g. port 81 might end up instead going to port 80.
>
> Yes, this would be a problem. There is a similar PR (PR62085) on this.
> But due to ABI constraints this is not easy to fix in a stable version.

How about adding a new ProxyPass parameter called "workername" where you can explicitly set the worker name to something both short AND unique? Or is that best requested as a new enhancement?

Revision history for this message
In , Chris-christopherschultz (chris-christopherschultz) wrote :

*** This bug has been confirmed by popular vote. ***

Bryce Harrington (bryce)
Changed in apache2 (Ubuntu Yakkety):
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.