Glance uses underscores in header names - stripped by many proxies

[Daniel Speichert]

These headers:
x-image-meta-container_format: bare
x-image-meta-disk_format: qcow2
x-image-meta-is_public: True

are that contain underscores are stripped by many webservers that are widely used to terminate SSL in front of OpenStack.

HTTP proxies such as nginx (which are used e.g. for SSL termination) strip these headers by default. While by RFC definition, the headers are not incorrect, it would be a major improvement to change them into hyphens. No other OpenStack project uses underscores in header names currently.

It would be easier for everyone not to use underscores.

[Flavio Percoco (flaper87)]

Hi Daniel,

Thanks a lot for the report and for debugging the issue.

I agree this is a real issue and I'm marking it as high since I believe most of the deployments use / would use HTTP proxies on top of glance.

[Stuart McLaren (stuart-mclaren)]

We need to be careful with backwards compatability here.

[Ryan O'Hara (rohara)]

Appears this behavior is configurable in nginx.

http://wiki.nginx.org/HttpCoreModule#underscores_in_headers

[Lon Hohberger (lhh)]

Someone else might want to check, but field-names in HTTP 1.1 are according to RFC822 section 3.1: ASCII values 33-126 decimal & inclusive, except the colon character.

This could be a bug in nginx.

Here's RFC822:
http://www.ietf.org/rfc/rfc0822.txt

Here's HTTP 1.1:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2

The ASCII value for underscore is 95.

[Lon Hohberger (lhh)]

Looks like the current draft HTTP 1.1 update clarifies this somewhat.

http://www.ietf.org/id/draft-ietf-httpbis-p1-messaging-26.txt

Section 3.2 defines field-name as a token; section 3.2.6 defines token as the following:

     token = 1*tchar

     tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*"
                    / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
                    / DIGIT / ALPHA
                    ; any VCHAR, except delimiters

I'm reasonably sure underscore is allowed (and furthermore, any unrecognized header must -not- be stripped, according to section 3.2.1).

[Daniel Speichert (dasp)]

Even if it was a bug in nginx, Apache, PHP and Perl and underscores were permitted, what's the use of them if nobody likes them?

http://stackoverflow.com/questions/17440564/all-caps-http-headers-with-underscores-dropped-in-apache-2-4
http://www.perlmonks.org/?node_id=756540
https://devcentral.f5.com/questions/httpheader-insert-cannot-insert-header-with-an-underscore-in-the-name
http://librelist.com/browser/rest.client/2010/4/29/rest-client-converting-underscores-to-hyphens-in-user-headers/
https://issues.apache.org/bugzilla/show_bug.cgi?id=38895
http://stackoverflow.com/questions/18185366/header-names-with-underscores-ignored-in-php-5-5-1-apache-2-4-6

[Alan Pevec (apevec)]

So apache 2.4 new feature is explained as:
"Translation of headers to environment variables is more strict than before to mitigate some possible cross-site-scripting attacks via header injection. Headers containing invalid characters (including underscores) are now silently dropped. Environment Variables in Apache has some pointers on how to work around broken legacy clients which require such headers. (This affects all modules which use these environment variables.)"

"attacks via header injection" is explained in http://kuza55.blogspot.com/2007/07/exploiting-reflected-xss.html
afaict e.g. @request.user_agent can be faked with custom User_Agent: header because web-framework doesn't detect collision with the proper User-Agent: header when translating. But it's bad webapp anyway where all input is not sanitized!

[Erno Kuvaja (jokke)]

"""
These headers:
x-image-meta-container_format: bare
x-image-meta-disk_format: qcow2
x-image-meta-is_public: True

are not valid HTTP headers - these should not contain underscores.
"""

This is not accurate statement.
RFC 822:
3.1.2. STRUCTURE OF HEADER FIELDS
"""
The field-name must be composed of printable ASCII characters (i.e., characters that have values between 33. and 126., decimal, except colon).
"""
and rfc 2616 sec 4.2 refers the format to rfc 822.

So it's nginx breaking the protocol by stripping those, not us by using them and as that 'underscores_in_headers on|off' has been on nginx at least from 2011 already I don't see reason why this should be bug in glance.

[Daniel Speichert (dasp)]

You are right, the header name is not incorrect by RFC definition.

However, due to the fact that most software (not only nginx, Apache and others as well) does not handle underscores in header names well, it would be a major improvement to change them into hyphens. No other OpenStack project uses underscores in header names currently.

It would be easier for everyone not to use underscores.

[Jordan Pittier (jordan-pittier)]

I agree with Daniel. It's simpler and easy for us to just change glance than to change all webservers that doesn't support underscore in header name.

We faced this issue month ago and wasted aprox 1 day to track it... I'm sure many have too.

[Daniel Speichert (dasp)]

It took me a week to track it down with a help of people from RDO mailing list (https://www.redhat.com/archives/rdo-list/2014-February/msg00032.html).
Let's save others from it.