302 response with a header size of 4k+ triggers 502 error
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Nginx |
New
|
Undecided
|
Unassigned |
Bug Description
We experienced a great deal of 502 errors when we added App Dynamics user monitoring to our application.
The workaround for us was to change the proxy_buffer_size to 8k, but the behavior we experienced seems like a bug.
If the header is too large, I'd expect to see a 431 Request Header Fields Too Large error, or something of the sort. Troubleshooting the 502 was quite painful. It seems like the cause was ultimately the Content-Length being set to 0.
Here's the debug output:
==> edm.test-
10.1.68.131 - - [23/Aug/
==> edm.test-
2017/08/23 19:57:47 [debug] 24833#24833: *57 http upstream request: "/signin?
2017/08/23 19:57:47 [debug] 24833#24833: *57 http upstream process header
2017/08/23 19:57:47 [debug] 24833#24833: *57 malloc: 00007FF5DAEA219
2017/08/23 19:57:47 [debug] 24833#24833: *57 recv: eof:1, avail:1
2017/08/23 19:57:47 [debug] 24833#24833: *57 recv: fd:41 4096 of 4096
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy status 302 "302 Found"
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Server: Apache-Coyote/1.1"
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: ADRUM_BTa=
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: ADRUM_BTa=
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "X-XSS-Protection: 1; mode=block"
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "X-FRAME-OPTIONS: SAMEORIGIN"
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: ADRUM_BT1=
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: ADRUM_BT1=
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: JSESSIONID=
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: sessionToken=
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: sessionToken=; Domain=.company.ca; Secure"
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: app_login_
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: app_login_
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: sessionToken=""; Domain=.company.ca; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=sessionToken"
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: sessionToken=
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: qbn.ptc.
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: qbn.ptc.
2017/08/23 19:57:47 [debug] 24833#24833: *57 posix_memalign: 00007FF5DAFC6A5
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: qbn.ptc.
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: qbn.ptc.
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: qbn.ptc.
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: qbn.ptc.
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: app_login_
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: app_login_
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: sessionToken=""; Domain=.company.ca; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=sessionToken"
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: sessionToken=
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: application_
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: application_
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: app_login_
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: app_login_
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: sessionToken=""; Domain=.company.ca; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=sessionToken"
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Set-Cookie: sessionToken=
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Location: https:/
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Content-Length: 0"
2017/08/23 19:57:47 [debug] 24833#24833: *57 http proxy header: "Date: Wed, 23 Aug 2017 19:57:47 GMT"
2017/08/23 19:57:47 [error] 24833#24833: *57 upstream sent too big header while reading response header from upstream, client: 10.1.68.131, server: edm.test-
2017/08/23 19:57:47 [debug] 24833#24833: *57 http next upstream, 8
2017/08/23 19:57:47 [debug] 24833#24833: *57 free rr peer 1 4
2017/08/23 19:57:47 [debug] 24833#24833: *57 finalize http upstream request: 502
2017/08/23 19:57:47 [debug] 24833#24833: *57 finalize http proxy request
2017/08/23 19:57:47 [debug] 24833#24833: *57 close http upstream connection: 41
2017/08/23 19:57:47 [debug] 24833#24833: *57 free: 00007FF5DAE9AC70, unused: 48
2017/08/23 19:57:47 [debug] 24833#24833: *57 event timer del: 41: 1503518387034
2017/08/23 19:57:47 [debug] 24833#24833: *57 reusable connection: 0
2017/08/23 19:57:47 [debug] 24833#24833: *57 http finalize request: 502, "/signin?
2017/08/23 19:57:47 [debug] 24833#24833: *57 http special response: 502, "/signin?
2017/08/23 19:57:47 [debug] 24833#24833: *57 HTTP/1.1 502 Bad Gateway
Server: nginx
Date: Wed, 23 Aug 2017 19:57:47 GMT
Content-Type: text/html
Content-Length: 568
Connection: keep-alive