Large Objects/Manifests ignore allowed_headers (Content-Encoding, Content-Disposition, ...)

Bug #829587 reported by Malte S. Stretz on 2011-08-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
High
John Dickinson

Bug Description

This was originally Rackspace Cloud Ticket 167610.

It looks like custom headers like Content-Encoding and Content-Disposition and the CORS header are ignored for Large Objects which are glued together with a X-Object-Manifest.

You can find a testcase in my manifest-testcase branch of my fork of php-cloudfiles at https://github.com/mss/php-cloudfiles/tree/manifest-testcase (sorry for using PHP but thats what I currently have to work with :). The test script is called manifest-testcase.php, you have to enter your credentials in manifest-testcase.ini first. There are other settings in that file which you might want to change as well.

It will create two objects in the container "test": manifest-test/data and manifest-test/link. The latter has a X-Object-Manifest which points to the first. Additionally, it will set the custom header Content-Disposition (and optionally Content-Encoding, see below). It will also set the container to public and print the public URI for the link in the end.

If you request that URI, you will notice that the Content-Disposition header is not included. If you request the data file instead, it is.

It gets worse if the Large Object is stored gzip compressed (enable the gzip option in the ini file to test this) since the Content-Encoding will not be sent to the user agent.

A quick glance at the Code shows that the swift.proxy.server copies the X-Object-Meta-* headers but not the stuff stored in allowed_headers:
https://github.com/openstack/swift/blob/f96efd9b5113251e08c7/swift/proxy/server.py#L826

It probably needs the functionality implemented in swift.obj.server:
https://github.com/openstack/swift/blob/f96efd9b5113251e08c7/swift/obj/server.py#L365

Here is a sample test run:

mss@TreeHouse:~/Projects/php-cloudfiles/code$ ./manifest-test.sh
++ dirname ./manifest-test.sh
+ cd .
++ basename ./manifest-test.sh .sh
+ exec php manifest-test.php
* About to connect() to storage101.lon3.clouddrive.com port 443 (#0)
* Trying 94.236.56.96... * connected
* Connected to storage101.lon3.clouddrive.com (94.236.56.96) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSL connection using RC4-SHA
* Server certificate:
* subject: O=storage101.lon3.clouddrive.com; OU=Go to https://www.thawte.com/repository/index.html; OU=Thawte SSL123 certificate; OU=Domain Validated; CN=storage101.lon3.clouddrive.com
* start date: 2010-09-24 00:00:00 GMT
* expire date: 2012-09-23 23:59:59 GMT
* common name: storage101.lon3.clouddrive.com (matched)
* issuer: C=US; O=Thawte, Inc.; OU=Domain Validated SSL; CN=Thawte DV SSL CA
* SSL certificate verify ok.
> PUT /v1/MossoCloudFS_fd53b750-c875-4e40-afd3-3f57a36ae152/test HTTP/1.1
Host: storage101.lon3.clouddrive.com
Accept: */*
X-Auth-Token: <hidden>
User-Agent: PHP-CloudFiles/1.8.0

< HTTP/1.1 202 Accepted
< Content-Length: 58
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: tx2b94482108754e699020b8145422f67c
< Date: Fri, 19 Aug 2011 15:39:25 GMT
<
* Connection #0 to host storage101.lon3.clouddrive.com left intact
* About to connect() to cdn3.clouddrive.com port 443 (#0)
* Trying 94.236.56.97... * connected
* Connected to cdn3.clouddrive.com (94.236.56.97) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
* subject: O=cdn3.clouddrive.com; OU=Go to https://www.thawte.com/repository/index.html; OU=Thawte SSL123 certificate; OU=Domain Validated; CN=cdn3.clouddrive.com
* start date: 2010-10-06 00:00:00 GMT
* expire date: 2012-10-05 23:59:59 GMT
* common name: cdn3.clouddrive.com (matched)
* issuer: C=US; O=Thawte, Inc.; OU=Domain Validated SSL; CN=Thawte DV SSL CA
* SSL certificate verify ok.
> HEAD /v1/MossoCloudFS_fd53b750-c875-4e40-afd3-3f57a36ae152/test HTTP/1.1
Host: cdn3.clouddrive.com
Accept: */*
X-Auth-Token: <hidden>
User-Agent: PHP-CloudFiles/1.8.0

< HTTP/1.1 204 No Content
< Date: Fri, 19 Aug 2011 15:39:25 GMT
< Server: Apache
< X-CDN-URI: http://c96969.r69.cf3.rackcdn.com
< X-CDN-SSL-URI: https://c96969.ssl.cf3.rackcdn.com
< X-CDN-Enabled: True
< X-TTL: 86400
< X-Log-Retention: True
< Connection: close
< Content-Type: text/plain; charset=UTF-8
<
* Closing connection #0
* About to connect() to cdn3.clouddrive.com port 443 (#0)
* Trying 94.236.56.97... * connected
* Connected to cdn3.clouddrive.com (94.236.56.97) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
* subject: O=cdn3.clouddrive.com; OU=Go to https://www.thawte.com/repository/index.html; OU=Thawte SSL123 certificate; OU=Domain Validated; CN=cdn3.clouddrive.com
* start date: 2010-10-06 00:00:00 GMT
* expire date: 2012-10-05 23:59:59 GMT
* common name: cdn3.clouddrive.com (matched)
* issuer: C=US; O=Thawte, Inc.; OU=Domain Validated SSL; CN=Thawte DV SSL CA
* SSL certificate verify ok.
> POST /v1/MossoCloudFS_fd53b750-c875-4e40-afd3-3f57a36ae152/test HTTP/1.1
Host: cdn3.clouddrive.com
Accept: */*
X-CDN-Enabled: True
X-TTL: 86400
X-Log-Retention: True
X-Auth-Token: <hidden>
User-Agent: PHP-CloudFiles/1.8.0

< HTTP/1.1 202 Accepted
< Date: Fri, 19 Aug 2011 15:39:26 GMT
< Server: Apache
< X-CDN-URI: http://c96969.r69.cf3.rackcdn.com
< X-CDN-SSL-URI: https://c96969.ssl.cf3.rackcdn.com
< Content-Length: 0
< Connection: close
< Content-Type: text/plain; charset=UTF-8
<
* Closing connection #0
* About to connect() to storage101.lon3.clouddrive.com port 443 (#0)
* Trying 94.236.56.96... * connected
* Connected to storage101.lon3.clouddrive.com (94.236.56.96) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSL connection using RC4-SHA
* Server certificate:
* subject: O=storage101.lon3.clouddrive.com; OU=Go to https://www.thawte.com/repository/index.html; OU=Thawte SSL123 certificate; OU=Domain Validated; CN=storage101.lon3.clouddrive.com
* start date: 2010-09-24 00:00:00 GMT
* expire date: 2012-09-23 23:59:59 GMT
* common name: storage101.lon3.clouddrive.com (matched)
* issuer: C=US; O=Thawte, Inc.; OU=Domain Validated SSL; CN=Thawte DV SSL CA
* SSL certificate verify ok.
> HEAD /v1/MossoCloudFS_fd53b750-c875-4e40-afd3-3f57a36ae152/test/manifest-test/data HTTP/1.1
Host: storage101.lon3.clouddrive.com
Accept: */*
X-Auth-Token: <hidden>
User-Agent: PHP-CloudFiles/1.8.0

< HTTP/1.1 404 Not Found
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: tx2d2b1e1cf3324cb28d4570b07f00657d
< Date: Fri, 19 Aug 2011 15:39:26 GMT
<
* Connection #0 to host storage101.lon3.clouddrive.com left intact
* About to connect() to storage101.lon3.clouddrive.com port 443 (#0)
* Trying 94.236.56.96... * connected
* Connected to storage101.lon3.clouddrive.com (94.236.56.96) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSL connection using RC4-SHA
* Server certificate:
* subject: O=storage101.lon3.clouddrive.com; OU=Go to https://www.thawte.com/repository/index.html; OU=Thawte SSL123 certificate; OU=Domain Validated; CN=storage101.lon3.clouddrive.com
* start date: 2010-09-24 00:00:00 GMT
* expire date: 2012-09-23 23:59:59 GMT
* common name: storage101.lon3.clouddrive.com (matched)
* issuer: C=US; O=Thawte, Inc.; OU=Domain Validated SSL; CN=Thawte DV SSL CA
* SSL certificate verify ok.
> PUT /v1/MossoCloudFS_fd53b750-c875-4e40-afd3-3f57a36ae152/test/manifest-test/data HTTP/1.1
Host: storage101.lon3.clouddrive.com
Accept: */*
X-Object-Meta-Data: test
Content-Disposition: inline
Content-Encoding: gzip
ETag: 38962e22d2a9e97ece48c2019f3e9c6e
Content-Type: application/x-gzip
X-Auth-Token: <hidden>
User-Agent: PHP-CloudFiles/1.8.0
Content-Length: 24
Expect: 100-continue

< HTTP/1.1 100 Continue
< HTTP/1.1 201 Created
< Content-Length: 118
< Content-Type: text/html; charset=UTF-8
< Etag: 38962e22d2a9e97ece48c2019f3e9c6e
< Last-Modified: Fri, 19 Aug 2011 15:39:26 GMT
< X-Trans-Id: txeb60af8276ab432db2ad3a49b308903e
< Date: Fri, 19 Aug 2011 15:39:26 GMT
<
* Connection #0 to host storage101.lon3.clouddrive.com left intact
* Re-using existing connection! (#0) with host storage101.lon3.clouddrive.com
* Connected to storage101.lon3.clouddrive.com (94.236.56.96) port 443 (#0)
> HEAD /v1/MossoCloudFS_fd53b750-c875-4e40-afd3-3f57a36ae152/test/manifest-test/link HTTP/1.1
Host: storage101.lon3.clouddrive.com
Accept: */*
X-Auth-Token: <hidden>
User-Agent: PHP-CloudFiles/1.8.0

< HTTP/1.1 404 Not Found
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: tx10edbc85d8fb447c95d5f956ab32df48
< Date: Fri, 19 Aug 2011 15:39:26 GMT
<
* Connection #0 to host storage101.lon3.clouddrive.com left intact
* Re-using existing connection! (#0) with host storage101.lon3.clouddrive.com
* Connected to storage101.lon3.clouddrive.com (94.236.56.96) port 443 (#0)
> PUT /v1/MossoCloudFS_fd53b750-c875-4e40-afd3-3f57a36ae152/test/manifest-test/link HTTP/1.1
Host: storage101.lon3.clouddrive.com
Accept: */*
X-Object-Meta-Link: test
Content-Disposition: attachment
Content-Encoding: gzip
X-Object-Manifest: test/manifest-test/data
ETag: 5058f1af8388633f609cadb75a75dc9d
Content-Type: text/plain
X-Auth-Token: <hidden>
User-Agent: PHP-CloudFiles/1.8.0
Content-Length: 1
Expect: 100-continue

< HTTP/1.1 100 Continue
< HTTP/1.1 201 Created
< Content-Length: 118
< Content-Type: text/html; charset=UTF-8
< Etag: 5058f1af8388633f609cadb75a75dc9d
< Last-Modified: Fri, 19 Aug 2011 15:39:26 GMT
< X-Trans-Id: txf18c33510a3d48f9b31b80d1f1dc2b8e
< Date: Fri, 19 Aug 2011 15:39:26 GMT
<
* Connection #0 to host storage101.lon3.clouddrive.com left intact
* Re-using existing connection! (#0) with host storage101.lon3.clouddrive.com
* Connected to storage101.lon3.clouddrive.com (94.236.56.96) port 443 (#0)
> HEAD /v1/MossoCloudFS_fd53b750-c875-4e40-afd3-3f57a36ae152/test/manifest-test/data HTTP/1.1
Host: storage101.lon3.clouddrive.com
Accept: */*
X-Auth-Token: <hidden>
User-Agent: PHP-CloudFiles/1.8.0

< HTTP/1.1 200 OK
< X-Object-Meta-Data: test
< Content-Disposition: inline
< Content-Encoding: gzip
< Last-Modified: Fri, 19 Aug 2011 15:39:26 GMT
< Etag: 38962e22d2a9e97ece48c2019f3e9c6e
< Accept-Ranges: bytes
< Content-Length: 24
< Content-Type: application/x-gzip
< X-Trans-Id: tx3ee2495962b04cf1aa9729dde3f75cd2
< Date: Fri, 19 Aug 2011 15:39:26 GMT
<
* Connection #0 to host storage101.lon3.clouddrive.com left intact
* Re-using existing connection! (#0) with host storage101.lon3.clouddrive.com
* Connected to storage101.lon3.clouddrive.com (94.236.56.96) port 443 (#0)
> HEAD /v1/MossoCloudFS_fd53b750-c875-4e40-afd3-3f57a36ae152/test/manifest-test/link HTTP/1.1
Host: storage101.lon3.clouddrive.com
Accept: */*
X-Auth-Token: <hidden>
User-Agent: PHP-CloudFiles/1.8.0

< HTTP/1.1 200 OK
< X-Object-Manifest: test/manifest-test/data
< Content-Type: text/plain
< X-Object-Meta-Link: test
< Etag: 8bfb9e5e3baa089de0a603445bc0fa1c
< Content-Length: 24
< Last-Modified: Fri, 19 Aug 2011 15:39:26 GMT
< Accept-Ranges: bytes
< X-Trans-Id: tx8b2a98052fd8441aae2e1e2101408987
< Date: Fri, 19 Aug 2011 15:39:27 GMT
<
* Connection #0 to host storage101.lon3.clouddrive.com left intact

-------------------------

http://c96969.r69.cf3.rackcdn.com/manifest-test/link

-------------------------
* Closing connection #0
* Closing connection #0
* Closing connection #0
mss@TreeHouse:~/Projects/php-cloudfiles/code$ GET -USe http://test.cdn.lorentz.de/manifest-test/data
GET http://test.cdn.lorentz.de/manifest-test/data
User-Agent: lwp-request/5.834 libwww-perl/5.837

GET http://test.cdn.lorentz.de/manifest-test/data --> 200 OK
Cache-Control: public, max-age=86400
Connection: close
Date: Fri, 19 Aug 2011 15:42:12 GMT
ETag: 38962e22d2a9e97ece48c2019f3e9c6e
Server: nginx/0.7.65
Content-Length: 4
Content-Type: application/x-gzip
Expires: Sat, 20 Aug 2011 15:42:12 GMT
Last-Modified: Fri, 19 Aug 2011 15:39:26 GMT
Client-Date: Fri, 19 Aug 2011 15:42:12 GMT
Client-Peer: 195.95.193.9:80
Client-Response-Num: 1
Content-Disposition: inline
X-Object-Meta-Data: test
X-Trans-Id: txa5ecc909568f47af8600b58e3567aae4

foo
mss@TreeHouse:~/Projects/php-cloudfiles/code$ GET -USe http://test.cdn.lorentz.de/manifest-test/link
GET http://test.cdn.lorentz.de/manifest-test/link
User-Agent: lwp-request/5.834 libwww-perl/5.837

GET http://test.cdn.lorentz.de/manifest-test/link --> 200 OK
Cache-Control: public, max-age=85217
Connection: close
Date: Fri, 19 Aug 2011 15:42:16 GMT
Accept-Ranges: bytes
ETag: 8bfb9e5e3baa089de0a603445bc0fa1c
Server: nginx/0.7.65
Content-Length: 24
Content-Type: text/plain; charset=UTF-8
Expires: Sat, 20 Aug 2011 15:22:33 GMT
Last-Modified: Fri, 19 Aug 2011 15:09:14 GMT
Client-Date: Fri, 19 Aug 2011 15:42:15 GMT
Client-Peer: 195.95.193.16:80
Client-Response-Num: 1
X-Object-Manifest: test/manifest-test/data
X-Object-Meta-Link: test
X-Trans-Id: tx403f9813ea77453ba94c385aca8a2b50

<binary data omitted>

Related branches

Changed in swift:
status: New → Confirmed
importance: Undecided → High
Changed in swift:
assignee: nobody → John Dickinson (notmyname)
milestone: none → 1.4.3
status: Confirmed → In Progress
Changed in swift:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2011-09-12
Changed in swift:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers