Glance uses underscores in header names - stripped by many proxies

Bug #1276887 reported by Daniel Speichert on 2014-02-06
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Glance
High
Erno Kuvaja

Bug Description

These headers:
x-image-meta-container_format: bare
x-image-meta-disk_format: qcow2
x-image-meta-is_public: True

are that contain underscores are stripped by many webservers that are widely used to terminate SSL in front of OpenStack.

HTTP proxies such as nginx (which are used e.g. for SSL termination) strip these headers by default. While by RFC definition, the headers are not incorrect, it would be a major improvement to change them into hyphens. No other OpenStack project uses underscores in header names currently.

It would be easier for everyone not to use underscores.

Daniel Speichert (dasp) on 2014-02-06
tags: added: glance-api
Flavio Percoco (flaper87) wrote :

Hi Daniel,

Thanks a lot for the report and for debugging the issue.

I agree this is a real issue and I'm marking it as high since I believe most of the deployments use / would use HTTP proxies on top of glance.

Changed in glance:
status: New → Confirmed
importance: Undecided → High
Stuart McLaren (stuart-mclaren) wrote :

We need to be careful with backwards compatability here.

Ryan O'Hara (rohara) wrote :

Appears this behavior is configurable in nginx.

http://wiki.nginx.org/HttpCoreModule#underscores_in_headers

Lon Hohberger (lhh) wrote :

Someone else might want to check, but field-names in HTTP 1.1 are according to RFC822 section 3.1: ASCII values 33-126 decimal & inclusive, except the colon character.

This could be a bug in nginx.

Here's RFC822:
http://www.ietf.org/rfc/rfc0822.txt

Here's HTTP 1.1:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2

The ASCII value for underscore is 95.

Lon Hohberger (lhh) wrote :

Looks like the current draft HTTP 1.1 update clarifies this somewhat.

http://www.ietf.org/id/draft-ietf-httpbis-p1-messaging-26.txt

Section 3.2 defines field-name as a token; section 3.2.6 defines token as the following:

     token = 1*tchar

     tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*"
                    / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
                    / DIGIT / ALPHA
                    ; any VCHAR, except delimiters

I'm reasonably sure underscore is allowed (and furthermore, any unrecognized header must -not- be stripped, according to section 3.2.1).

Alan Pevec (apevec) wrote :

So apache 2.4 new feature is explained as:
"Translation of headers to environment variables is more strict than before to mitigate some possible cross-site-scripting attacks via header injection. Headers containing invalid characters (including underscores) are now silently dropped. Environment Variables in Apache has some pointers on how to work around broken legacy clients which require such headers. (This affects all modules which use these environment variables.)"

"attacks via header injection" is explained in http://kuza55.blogspot.com/2007/07/exploiting-reflected-xss.html
afaict e.g. @request.user_agent can be faked with custom User_Agent: header because web-framework doesn't detect collision with the proper User-Agent: header when translating. But it's bad webapp anyway where all input is not sanitized!

Erno Kuvaja (jokke) on 2014-02-12
Changed in glance:
assignee: nobody → Erno Kuvaja (jokke)
Erno Kuvaja (jokke) wrote :

"""
These headers:
x-image-meta-container_format: bare
x-image-meta-disk_format: qcow2
x-image-meta-is_public: True

are not valid HTTP headers - these should not contain underscores.
"""

This is not accurate statement.
RFC 822:
3.1.2. STRUCTURE OF HEADER FIELDS
"""
The field-name must be composed of printable ASCII characters (i.e., characters that have values between 33. and 126., decimal, except colon).
"""
and rfc 2616 sec 4.2 refers the format to rfc 822.

So it's nginx breaking the protocol by stripping those, not us by using them and as that 'underscores_in_headers on|off' has been on nginx at least from 2011 already I don't see reason why this should be bug in glance.

Changed in glance:
status: Confirmed → Invalid
Daniel Speichert (dasp) wrote :

You are right, the header name is not incorrect by RFC definition.

However, due to the fact that most software (not only nginx, Apache and others as well) does not handle underscores in header names well, it would be a major improvement to change them into hyphens. No other OpenStack project uses underscores in header names currently.

It would be easier for everyone not to use underscores.

description: updated
summary: - Glance uses INVALID HTTP header names - with underscore in their name
+ Glance uses underscores in header names - stripped by many proxies
Changed in glance:
status: Invalid → Opinion

I agree with Daniel. It's simpler and easy for us to just change glance than to change all webservers that doesn't support underscore in header name.

We faced this issue month ago and wasted aprox 1 day to track it... I'm sure many have too.

Daniel Speichert (dasp) wrote :

It took me a week to track it down with a help of people from RDO mailing list (https://www.redhat.com/archives/rdo-list/2014-February/msg00032.html).
Let's save others from it.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.