md5 sum is incorrect from proxy

[Josh Pasqualetto]

Currently on my Swift cluster I am getting improper content lengths back from newly created objects.

Here is an example of how I am reproducing, Please note that the content length coming back from the object servers are right, it's the proxy that is mangling the content length, also note that a length of 13 seems to be the standard for what it returns -- I think it might be giving the content length of the request itself.

josh@vm-10-128-46-8:~$ dd if=/dev/urandom of=swift_md5_test bs=4k count=5096
5096+0 records in
5096+0 records out
20873216 bytes (21 MB) copied, 3.16561 s, 6.6 MB/s
josh@vm-10-128-46-8:~$ swift -A https://xxx:777/auth/v1.0 -U xxx -K xxx upload data.dev.sa2s swift_md5_test
swift_md5_test

josh@vm-10-128-46-8:~$ swift -A https://xxx:777/auth/v1.0 -U xxx -K xxx stat data.dev.sa2s swift_md5_test
       Account: AUTH_e46592f7-6864-4ce6-a554-11520ee2dxxx
     Container: data.dev.sa2s
        Object: swift_md5_test
  Content Type: text/plain
Content Length: 13
 Last Modified: Mon, 02 Apr 2012 17:00:24 GMT
          ETag: 37389c3e6d1940dc810b1ea943d71e49
    Meta Mtime: 1333385918.0
 Accept-Ranges: bytes
    X-Trans-Id: txb2be4170b438421a81c4800129b70239

josh@vm-10-128-46-8:~$ curl -I -XHEAD "http://xxx:6000/vdd/159308/AUTH_e46592f7-6864-4ce6-a554-11520ee2dxxx/data.dev.sa2s/swift_md5_test"
HTTP/1.1 200 OK
Content-Type: application/octet-stream
X-Object-Meta-Mtime: 1333385918.0
Etag: "37389c3e6d1940dc810b1ea943d71e49"
Last-Modified: Mon, 02 Apr 2012 17:00:24 GMT
Content-Length: 20873216
X-Timestamp: 1333386024.44127
Date: Mon, 02 Apr 2012 17:07:11 GMT

[Doug Weimer (dweimer)]

Hi Josh,

I'm seeing the exact same thing with a swift 1.4.8 installation. My test environment is running CentOS 6. Are you running the standard Ubuntu packages or a custom install as well?

The software versions that I have installed:

python - 2.6.6 (CentOS 6 OS packages)
boto-2.3.0-py2.6.egg
configobj-4.7.2-py2.6.egg
coverage-3.5.1-py2.6-linux-x86_64.egg
decorator-3.3.2-py2.6.egg
distribute-0.6.10-py2.6.egg-info
eventlet-0.9.16-py2.6.egg
greenlet-0.3.4-py2.6-linux-x86_64.egg
iniparse-0.3.1-py2.6.egg-info
netifaces-0.8-py2.6-linux-x86_64.egg
nose-1.1.2-py2.6.eggPasteDeploy-1.5.0-py2.6.eggpip-0.8-py2.6.egg-infosetuptools-0.6c11-py2.6.egg-info
simplejson-2.5.0-py2.6-linux-x86_64.egg
swauth-1.0.4-py2.6.egg (From http://github.com/gholt/swauth, git tag 1.0.4)
swift-1.4.8-py2.6.egg (From http://github.com/openstack/swift, git tag 1.4.8)
urlgrabber-3.9.1-py2.6.egg-info
WebOb-1.1.1-py2.6.egg
WsgiService-0.2.5-py2.6.egg
xattr-0.6.4-py2.6-linux-x86_64.egg

An example of the issue:
# Object server view with content-length 1000000, this is the same for all copies of the object

curl -I -XHEAD "http://xxxxxx:6000/0/510874/DURACLOUD_7aa09e4f-83e5-4667-8809-8855f56052a9/api-test-space/file-999.txt"
HTTP/1.1 200 OK
Content-Type: text/plain
X-Object-Meta-Content-Mimetype: text%2Fplain
Etag: "c13dbb31ea312057db38a9787386dd07"
Last-Modified: Fri, 06 Apr 2012 18:44:43 GMT
Content-Length: 1000000
X-Timestamp: 1333737883.55361
Date: Fri, 06 Apr 2012 21:36:42 GMT

# Proxy view of the same file-999.txt object
swift stat api-test-space file-999.txt
       Account: DURACLOUD_7aa09e4f-83e5-4667-8809-8855f56052a9
     Container: api-test-space
        Object: file-999.txt
  Content Type: text/plain
Content Length: 13
 Last Modified: Fri, 06 Apr 2012 18:44:43 GMT
          ETag: c13dbb31ea312057db38a9787386dd07
Meta Content-Mimetype: text%2Fplain
 Accept-Ranges: bytes
    X-Trans-Id: tx4e9802d5d3674195916368b119683433

[Doug Weimer (dweimer)]

This looks like a WebOb issue. When I downgrade to WebOb 1.0 it resolves the problem.

       Account: DURACLOUD_7aa09e4f-83e5-4667-8809-8855f56052a9
     Container: api-test-space
        Object: file-999.txt
  Content Type: text/plain
Content Length: 1000000
 Last Modified: Fri, 06 Apr 2012 18:44:43 GMT
          ETag: c13dbb31ea312057db38a9787386dd07
Meta Content-Mimetype: text%2Fplain
 Accept-Ranges: bytes
    X-Trans-Id: txaee60bb9523944079d7251a599e05333

[Josh Pasqualetto (josh-pasqualetto)]

Hey Doug,

I downgraded my cluster to Webob 1.0 and that fixed my issue as well, Thank you for the help it is much appreciated.

[Doug Weimer (dweimer)]

I think that this issue is related to the WebOb problem identified in Bug #920197 (https://bugs.launchpad.net/swift/+bug/920197).

[Doug Weimer (dweimer)]

A patch has been proposed to the upstream WebOb project as part of Bug #920197 .
If you want to avoid the bug use WebOb 1.08.
If you want to track the progress of the fix for WebOb 1.1+ follow Bug #920197.