as a reverse proxy, a 100 continue response is sent prematurely when a request contains expects: 100-continue

Bug #1641238 reported by Jay R. Wren on 2016-11-11
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Apache2 Web Server
Confirmed
Medium
apache2 (Ubuntu)
Undecided
Unassigned

Bug Description

This effects trusty, xenial and current httpd trunk.

https://bz.apache.org/bugzilla/show_bug.cgi?id=60330

As a reverse proxy, a 100 continue response is sent prematurely when a request contains expects: 100-continue. This causes the requesting client to send a body. The apache httpd proxy will then read the body and attempt to send it to the backend, but the backend already sent an error and should be allowed to NOT read the remaining request body, which never should have existed. When the backend does not read the request body mod_proxy_pass errors and returns a 500 error to the client. The client never receives the correct error message.

Reverse proxy of 100-continue aware backend, sends 100 continue even when backend does not. This causes a client to think it should write a request body, while the backend may still respond with a 400 and not read the request body. mod_proxy_http then responds with 502 as a result of AH01097: pass request body failed

The backend is doing the right thing: it did not send a 100 continue so it should not be required to read a request body, regardless of transfer encoding or content-length.

Expected:

mod_proxy_http reverse proxy should not send 100-continue to a client unless the backend does.

Created attachment 34438
in the reverse proxy case, if request had Expects: 100-continue, delay writing 100 continue response until backend has sent 100-continue response

Created attachment 34451
Forward 100-continue (and minimize race when reusing backend connections)

I proposed this patch a while ago on the dev@ list ([1]), this is an update for latest trunk, with more (though incomplete) testing.

Could you please give it a try?

[1]. https://lists.apache.org/thread.html/4e541e032b8a77ebec8248534637b47cdcd4f38af79baa5259845db0@1430360070@%3Cdev.httpd.apache.org%3E

*** Bug 55433 has been marked as a duplicate of this bug. ***

Robie Basak (racb) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better.

I think we should wait until upstream commits a fix for this before we do anything in Ubuntu. Or is there a specific need that requires this to fixed sooner in Ubuntu? Under what circumstances are Ubuntu users impacted by this problem?

Jay R. Wren (evarlast) wrote :

Hi Robie,

The specific need is that www.jujucharms.com uses apache2 as a reverse proxy to charmstore. The upload of resources and charms is suboptimal without 100-continue support. While investigating adding 100-continue support, we ran into this bug. As for a general need for this to be fixed in Ubuntu, I know of none, other than correctly supporting 100-continue in http reverse proxies.

Only Ubuntu users using mod_proxy_http to reverse proxy a service which implements 100-continue are impacted by this problem. There must not be any. I'm honestly surprised that this bug has existed for so long.

I am working with upstream to get this or a similar patch applied. I'll continue doing so.

Thanks,
--
Jay

Yann,

I tried that patch, but I still get 503 error when expecting a 100 Continue response.

Any chance that this will be fixed? Have the very same problem from a backend Tomcat. I guess I need to drop mod_proxy and try mod_ajp or drop Apache HTTPd altogether for this.

*** Bug 57853 has been marked as a duplicate of this bug. ***

Sorry it's been a long time, I think we need more informations here as to the exact issue.

What exactly isn't working with the proposed patch?
Where are 100-continue or request bodies lacking or sent inappropriately, on which side (client/backend)?
IOW, can we please have a description/scenario of what is supposed to work and how, possibly with the expected request/response on both sides?
What is the configuration being tested?

Changed in apache2:
importance: Unknown → Medium
status: Unknown → Confirmed

(In reply to Yann Ylavic from comment #7)
> Sorry it's been a long time, I think we need more informations here as to
> the exact issue.
>
> What exactly isn't working with the proposed patch?
> Where are 100-continue or request bodies lacking or sent inappropriately, on
> which side (client/backend)?
> IOW, can we please have a description/scenario of what is supposed to work
> and how, possibly with the expected request/response on both sides?
> What is the configuration being tested?

Hi Yann,

I can provide a full verbose log of curl(1) for Tomcat behind HTTPd with faulty behavior and direct Tomcat communication. Moreover, I can expore the httpd.conf for that offending behavior.

Yes please do, along with the httpd error_log with LogLevel trace7.
Thanks!

Created attachment 36015
curl(1) to Tomcat directly

Created attachment 36016
curl(1) to Tomcat via HTTPd

The error log has been sent privately due to sensitive data. Looking forward to an analysis.

FYI, I have tried mod_proxy_{http,ajp} and mod_jk.

Thanks Michael, at first glance the error_log is with mod_proxy_ajp, while attachment 34451 is about mod_proxy_http (and I'd like to keep the scope there for now).
I agree that unpatched mod_proxy_http sends "100 continue" too soon (actually independently on the client and backend side).
The patch is precisely to avoid that (hop by hop 100-continue handling), did you give it a try? If yes, could I have the error_log with mod_proxy_http?

(In reply to Yann Ylavic from comment #14)
> Thanks Michael, at first glance the error_log is with mod_proxy_ajp, while
> attachment 34451 [details] is about mod_proxy_http (and I'd like to keep the
> scope there for now).
> I agree that unpatched mod_proxy_http sends "100 continue" too soon
> (actually independently on the client and backend side).
> The patch is precisely to avoid that (hop by hop 100-continue handling), did
> you give it a try? If yes, could I have the error_log with mod_proxy_http?

I agree, I have tried all possible modules with the same negative result. I will redo for you. Moreover, I will compile from trunk along with your patch and try to reproduce. Does it still apply cleanly to trunk?

Please be patient, I won't be able to test anything before 2018-07-23.

Thank you.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.