Comment 2 for bug 574697

I believe Apache's mod_proxy is stripping the TE header, and I don't see an easy way to fix it. Here's what I believe is happening:

1. launchpadlib sends a request to https://api.launchpad.dev/ that includes the TE header.
2. Apache prepares to handle the request by proxying it to http://localhost:8086/.
3. The TE header is a hop-by-hop header, and the HTTP proxy connection between https://api.launchpad.dev/ and http://localhost:8086/ counts as a "hop". So mod_proxy removes the TE header before proxying the request.

My skills with mod_proxy are admittedly poor, but I don't see any way to say "pass through the TE header". My skills with mod_rewrite are a little better, so I investigated the possibility of ensuring TE's survival by hacking the header value into the query string,

RewriteCond %{HTTP:TE} .+
RewriteRule (.*) $1?HTTP_TE=%{HTTP:TE} [QSA]

Strangely enough, requests that triggered this condition and were subject to this rewrite rule were *not* subject to the "ProxyPass" directive immediately below. The requests were not proxied and I got Apache 404 errors for my trouble. I don't know why, and this is where I gave up.

If we do get Apache to pass through TE, there's a real possibility that mod_proxy will just strip the Transfer-Encoding _response_ header and decompress the data before sending it over the network, since Transfer-Encoding is also a hop-by-hop header. Basically, I suspect we have this problem in both request and response, and I see even less hope for fixing this on the response side. (I ran a hacky experiment to see if this was true, but the results were inconclusive.)

If you haven't followed this whole drama you might ask: "why not use Accept-Encoding and Content-Encoding like a normal web app?" Well, we did that at one time. The problem is that we were using Content-Encoding with mod_compress, which
changes the ETags of the representations it serves from "foo" to "foo-gzip". (The Apache bug https://issues.apache.org/bugzilla/show_bug.cgi?id=39727 explains why.) When launchpadlib sent a conditional request with If-None-Match: "foo-gzip", lazr.restful had no idea what the ETag meant, and the conditional request didn't work, even if it would have otherwise.

I can think of two solutions that don't involve Apache magic:

1. My understanding of HTTP intermediaries such as proxies is pretty poor. There may be something in RFC2616 that allows a client to ask an intermediary to pass on a hop-by-hop header if that's possible. I'll investigate this.

2. We've already implemented the equivalent of mod_compress in lazr.restful; we just made it use TE and Transfer-Encoding instead of Accept-Encoding and Content-Encoding, so we could use the same ETags. But this means lazr.restful now controls every relevant portion of the system. We can go back to using Accept-Encoding and Content-Encoding, and lazr.restful can be programmed to serve and understand ETags like "foo-gzip". The downside of this solution is that it won't benefit existing clients, who will keep uselessly sending TE headers forever.