ACL tests fails on fresh Ubuntu 14.04

[Alexey Khivin]

some of the unit tests fails on my SAIO installation

======================================================================
FAIL: test_acl_syntax_verification (test.unit.common.middleware.test_tempauth.TestAccountAcls)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/swift/test/unit/common/middleware/test_tempauth.py", line 1199, in test_acl_syntax_verification
    self.assertEquals(resp.status_int, 204)
AssertionError: 400 != 204

======================================================================
FAIL: test_acls_propagate_to_sysmeta (test.unit.common.middleware.test_tempauth.TestAccountAcls)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/swift/test/unit/common/middleware/test_tempauth.py", line 1264, in test_acls_propagate_to_sysmeta
    self.assertEquals(resp.status_int, 204)
AssertionError: 400 != 204

======================================================================
FAIL: test_acl_syntax_verification (test.unit.common.middleware.test_tempauth.TestAuthMultiplePrefixes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/swift/test/unit/common/middleware/test_tempauth.py", line 1199, in test_acl_syntax_verification
    self.assertEquals(resp.status_int, 204)
AssertionError: 400 != 204

======================================================================
FAIL: test_acls_propagate_to_sysmeta (test.unit.common.middleware.test_tempauth.TestAuthMultiplePrefixes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/swift/test/unit/common/middleware/test_tempauth.py", line 1264, in test_acls_propagate_to_sysmeta
    self.assertEquals(resp.status_int, 204)
AssertionError: 400 != 204

after simplejson version had been changed from 3.6.5 to 3.5.3 tests passed successfully

[Matthew Oliver (matt-0)]

Thanks for raising this Alex, in channel and as this bug.

I ran the unit tests on the current state of master and couldn't replicate, i built a SAIO but again couldn't replicate, I made sure I had simplejson 3.6.5 and again failed to recreate.

To help recreate this issue I think we need more information. what Linux Distro where you using, what branch of the code (master [default]) and just in case it makes a difference what type of SAIO did you build? Loop back or partition?

Using the later version of simplejson makes no difference for me, could you have had a corrupted version of 3.6.5 installed? (Just a thought).

I'll try running the unit tests on a few more distro's to see if I can replicate.

[Matthew Oliver (matt-0)]

OK managed to recreate in OSX, by running the unit tests on the base code. Annoyingly when running the tests via PyCharms (as to debug) they run fine.

So far its been one of those days.

[Christian Schwede (cschwede)]

I wasn't able to reproduce this - neither on my linux SAIO nor on OSX, and I used simplejson 3.6.5 in both cases.

Alex, can you update back to simplejson 3.6.5 and recheck? Just to make sure the previous version was not corrupted, as Matthew suspected?

[Alistair Coles (alistair-coles)]

Ditto, cannot reproduce with simplejson 3.6.5 on SAIO on precise and trusty

[Alexey Khivin (akhivin)]

I read yesterday's chat again

torgomatic> except that these are unicodes, which happens when either (a) you do not have simplejson installed, or (b) you do have simplejson installed but without the C bindings

So after I had reinstall simplejson (3.5.3 or 3.6.5 doesn't matter) it became with c bindings and tests passed successfully

In Ubuntu 14.04 simplejson installed without speedup extension by default

So It would be nice to mention this thing in the SAIO tutorial

thanks all for your help in resolving this issue!

[Alexey Khivin (akhivin)]

My offer is to add an additional test to check getattr(simplejson, '_speedups', None) and raise understandable exception if speedup extension is not being used

[Christian Schwede (cschwede)]

OK, I was able to reproduce this by manually removing /usr/lib/python2.7/dist-packages/simplejson/_speedups.so to some other place. Interesting that only two tests are affected by this

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/159478

[Alistair Coles (alistair-coles)]

Some diagnosis: tempauth account acl validation checks that values are type str, but without the speedups extension the json parser is returning unicode objects.

[Alistair Coles (alistair-coles)]

Its not just those unit tests that fail without the simplejson speedups extension - functional tests also fail - see example below with and without speedups:

swift@saio-1:~/swift$ sudo apt-get remove python-simplejson
Removing python-simplejson ...

swift@saio-1:~/swift$ sudo apt-get install python-simplejson
Setting up python-simplejson (2.3.2-1) ...

swift@saio-1:~/swift$ ls /usr/lib/python2.7/dist-packages/simplejson
decoder.py encoder.py __init__.py ordered_dict.py scanner.py _speedups.so tool.py
decoder.pyc encoder.pyc __init__.pyc ordered_dict.pyc scanner.pyc tests tool.pyc

swift@saio-1:~/swift$ swift-init restart main >> /dev/null
swift@saio-1:~/swift$ nosetests ./test/functional/test_account.py:TestAccount.test_unicode_metadata
SKIPPING FUNCTIONAL TESTS SPECIFIC TO AUTH VERSION 3
SKIPPING FUNCTIONAL TESTS SPECIFIC TO SERVICE TOKENS
test_unicode_metadata (test.functional.test_account.TestAccount) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.332s

OK

swift@saio-1:~/swift$ sudo rm /usr/lib/python2.7/dist-packages/simplejson/_speedups.so

swift@saio-1:~/swift$ swift-init restart main >> /dev/null

swift@saio-1:~/swift$ nosetests ./test/functional/test_account.py:TestAccount.test_unicode_metadata
SKIPPING FUNCTIONAL TESTS SPECIFIC TO AUTH VERSION 3
SKIPPING FUNCTIONAL TESTS SPECIFIC TO SERVICE TOKENS
test_unicode_metadata (test.functional.test_account.TestAccount) ... ERROR

======================================================================
ERROR: test_unicode_metadata (test.functional.test_account.TestAccount)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/swift/swift/test/functional/test_account.py", line 689, in test_unicode_metadata
    resp = retry(post, uni_key, '1')
  File "/home/swift/swift/test/functional/__init__.py", line 665, in retry
    raise Exception('No result after %s retries.' % retries)
Exception: No result after 5 retries.
-------------------- >> begin captured logging << --------------------
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1
requests.packages.urllib3.connectionpool: DEBUG: "GET /auth/v1.0 HTTP/1.1" 200 0
swiftclient: DEBUG: REQ: curl -i http://127.0.0.1:8080/auth/v1.0 -X GET
swiftclient: DEBUG: RESP STATUS: 200 OK
swiftclient: DEBUG: RESP HEADERS: [('content-length', '0'), ('x-trans-id', 'txd606fa2bb6e046c584ece-0054ef44bf'), ('x-auth-token', 'AUTH_tk2d23edbc174c47e1a9198d450b2f4ec3'), ('x-storage-token', 'AUTH_tk2d23edbc174c47e1a9198d450b2f4ec3'), ('date', 'Thu, 26 Feb 2015 16:07:27 GMT'), ('x-storage-url', 'http://127.0.0.1:8080/v1/AUTH_test'), ('content-type', 'text/html; charset=UTF-8')]
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1
requests.packages.urllib3.connectionpool: DEBUG: "GET /auth/v1.0 HTTP/1.1" 200 0
swiftclient: DEBUG: REQ: curl -i http://127.0.0.1:8080/auth/v1.0 -X GET
swiftclient: DEBUG: RESP STATUS: 200 OK
swiftclient: DEBUG: RESP HEADERS: [('content-length', '0'), ('x-trans-id', 'tx07424b24bd4344099666c-0054ef44bf'), ('x-auth-token', 'AUTH_tk2...

[Alistair Coles (alistair-coles)]

More detail on failing func test - saw exception in proxy log for account server PUT:

swift@saio-1:~/swift$ git log --pretty=oneline -n 1
37fb2cef0462a4934b91c2bf8353e0425b741454 Merge "Enable override of container update headers"

swift@saio-1:~/swift$ ls /usr/lib/python2.7/dist-packages/simplejson
decoder.py encoder.py __init__.py ordered_dict.py scanner.py tests tool.pyc
decoder.pyc encoder.pyc __init__.pyc ordered_dict.pyc scanner.pyc tool.py

swift@saio-1:~/swift$ swift-init restart main >> /dev/null

swift@saio-1:~/swift$ nosetests ./test/functional/test_account.py:TestAccount.test_unicode_metadata
SKIPPING FUNCTIONAL TESTS SPECIFIC TO AUTH VERSION 3
SKIPPING FUNCTIONAL TESTS SPECIFIC TO SERVICE TOKENS
test_unicode_metadata (test.functional.test_account.TestAccount) ... ERROR

======================================================================
ERROR: test_unicode_metadata (test.functional.test_account.TestAccount)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/swift/swift/test/functional/test_account.py", line 689, in test_unicode_metadata
    resp = retry(post, uni_key, '1')
  File "/home/swift/swift/test/functional/__init__.py", line 665, in retry
    raise Exception('No result after %s retries.' % retries)
Exception: No result after 5 retries.
-------------------- >> begin captured logging << --------------------
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1
requests.packages.urllib3.connectionpool: DEBUG: "GET /auth/v1.0 HTTP/1.1" 200 0
swiftclient: DEBUG: REQ: curl -i http://127.0.0.1:8080/auth/v1.0 -X GET
swiftclient: DEBUG: RESP STATUS: 200 OK
swiftclient: DEBUG: RESP HEADERS: [('content-length', '0'), ('x-trans-id', 'txa17f72089a0d40618b147-0054ef4a67'), ('x-auth-token', 'AUTH_tk2d23edbc174c47e1a9198d450b2f4ec3'), ('x-storage-token', 'AUTH_tk2d23edbc174c47e1a9198d450b2f4ec3'), ('date', 'Thu, 26 Feb 2015 16:31:35 GMT'), ('x-storage-url', 'http://127.0.0.1:8080/v1/AUTH_test'), ('content-type', 'text/html; charset=UTF-8')]
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1
requests.packages.urllib3.connectionpool: DEBUG: "GET /auth/v1.0 HTTP/1.1" 200 0
swiftclient: DEBUG: REQ: curl -i http://127.0.0.1:8080/auth/v1.0 -X GET
swiftclient: DEBUG: RESP STATUS: 200 OK
swiftclient: DEBUG: RESP HEADERS: [('content-length', '0'), ('x-trans-id', 'txab49107119d149ce85c3a-0054ef4a67'), ('x-auth-token', 'AUTH_tk2d23edbc174c47e1a9198d450b2f4ec3'), ('x-storage-token', 'AUTH_tk2d23edbc174c47e1a9198d450b2f4ec3'), ('date', 'Thu, 26 Feb 2015 16:31:35 GMT'), ('x-storage-url', 'http://127.0.0.1:8080/v1/AUTH_test'), ('content-type', 'text/html; charset=UTF-8')]
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1
requests.packages.urllib3.connectionpool: DEBUG: "HEAD /v1/AUTH_test HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (2): 127.0.0.1
requests.packages.urllib3.connectionpool: DEBUG: "POST /v1/AUTH_test HTTP/1.1" 503 118
requests.packages.urllib3.connectionpool: DEBUG: "POST /v1/AUTH_...

[Samuel Merritt (torgomatic)]

Basically, simplejson is the devil and as soon as Swift drops support for Python 2.6, we should carefully go module-by-module and swap it out for stdlib's json, fixing bugs as we go.

There's some hyperbole there, obviously, but simplejson's sometimes-a-str-sometimes-a-unicode behavior has led to literally dozens of bugs in Swift, and every time we end up hacking around it with "if isinstance(thingy, unicode): ..."

[OpenStack Infra (hudson-openstack)]

Change abandoned by Christian Schwede (<email address hidden>) on branch: master
Review: https://review.openstack.org/159478
Reason: Let's fix the cause and not the effect.

[Alistair Coles (alistair-coles)]

Diagnosis of exception trace show in #11:

- test sends a header with key uni_key = u'X-Account-Meta-uni\u0E12'
- header key gets utf8 encoded
- proxy sends this in POST request to account server
- down in httplib, header *str* is added to a buffer list
- buffer list also has url path etc.
- url path container device name
- device name is from account ring data
- ring data I deserialized json
- when simplejson has no speedups, device name is unicode object
- when httplib send_ouput does a join of the buffer list, the join will attempt to coerce the header str to unicode
- bang - the coercion will fail with the UnicodeDecodeError because of non-ascii characters

Happy - path:
- with simplejson speedups the account ring data will have str not unicode values
- the device name will be a str
- the join in httplib send_output does not need to coerce the header str to unicode, so all ok.

[Alistair Coles (alistair-coles)]

+1 for Sam's comment at #12

So how do we resolve this bug report? Options:

1. document that simplejson speedups is required
   - I attempted to encode this in requirements.txt using "simplejson[speedups]>=2.0.9" but pip did not recognize speedups as a feature of simplejson
  - Another idea: wherever simplejson is imported, apply Christian's check for _speedups attr, default to json if not found

2. Find every place that the str->unicode coercion might fail and do some type-morphing

I'd vote for something along the lines of 1. My experience is that (s) fresh Ubuntu 14.04 (reported bug environment) did not have simplejson installed, sudo apt-get install simplejson DID install speedups, so I am not yet sure how common the reported bug scenario is.

[Alistair Coles (alistair-coles)]

Without simplejson speedups, I am getting a functional test *hang*:

anc@u128:~/0dev/swift {p-tempauth-acl-json}$ nosetests ./test/functional/tests.py:TestAccountUTF8.testAccountHead
SKIPPING FUNCTIONAL TESTS SPECIFIC TO AUTH VERSION 3
SKIPPING FUNCTIONAL TESTS SPECIFIC TO SERVICE TOKENS
testAccountHead (test.functional.tests.TestAccountUTF8) ...

In proxy.error log I see an exception trace:

Feb 27 14:00:15 localhost proxy-server: STDOUT: Traceback (most recent call last):
Feb 27 14:00:15 localhost proxy-server: STDOUT: File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
Feb 27 14:00:15 localhost proxy-server: STDOUT: timer()
Feb 27 14:00:15 localhost proxy-server: STDOUT: File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
Feb 27 14:00:15 localhost proxy-server: STDOUT: cb(*args, **kw)
Feb 27 14:00:15 localhost proxy-server: STDOUT: File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
Feb 27 14:00:15 localhost proxy-server: STDOUT: result = function(*args, **kwargs)
Feb 27 14:00:15 localhost proxy-server: STDOUT: File "/home/swift/swift/swift/common/utils.py", line 2180, in _run_func
Feb 27 14:00:15 localhost proxy-server: STDOUT: self._responses.put(func(*args, **kwargs))
Feb 27 14:00:15 localhost proxy-server: STDOUT: File "/home/swift/swift/swift/proxy/controllers/base.py", line 1051, in _make_request
Feb 27 14:00:15 localhost proxy-server: STDOUT: {'method': method, 'path': path})
Feb 27 14:00:15 localhost proxy-server: STDOUT: File "/home/swift/swift/swift/proxy/server.py", line 543, in exception_occurred
Feb 27 14:00:15 localhost proxy-server: STDOUT: 'device': node['device'], 'info': additional_info})
Feb 27 14:00:15 localhost proxy-server: STDOUT: File "/home/swift/swift/swift/common/utils.py", line 1261, in exception
Feb 27 14:00:15 localhost proxy-server: STDOUT: call('%s: %s' % (msg, emsg), *args, **kwargs)
Feb 27 14:00:15 localhost proxy-server: STDOUT: File "/home/swift/swift/swift/common/utils.py", line 1232, in _exception
Feb 27 14:00:15 localhost proxy-server: STDOUT: logging.LoggerAdapter.exception(self, msg, *args, **kwargs)
Feb 27 14:00:15 localhost proxy-server: STDOUT: File "/usr/lib/python2.7/logging/__init__.py", line 1445, in exception
Feb 27 14:00:15 localhost proxy-server: STDOUT: self.logger.error(msg, *args, **kwargs)
Feb 27 14:00:15 localhost proxy-server: STDOUT: File "/usr/lib/python2.7/logging/__init__.py", line 1166, in error
Feb 27 14:00:15 localhost proxy-server: STDOUT: self._log(ERROR, msg, args, **kwargs)
Feb 27 14:00:15 localhost proxy-server: STDOUT: File "/usr/lib/python2.7/logging/__init__.py", line 1258, in _log
Feb 27 14:00:15 localhost proxy-server: STDOUT: self.handle(record)
Feb 27 14:00:15 localhost proxy-server: STDOUT: File "/usr/lib/python2.7/logging/__init__.py", line 1268, in handle
Feb 27 14:00:15 localhost proxy-server: STDOUT: self.callHandlers(record)
Feb 27 14:00:15 localhost proxy-server: STDOUT: File "/usr/lib/python2.7/logging/__init__.py", line 1308, in callHandlers
Feb 27 14:00:15 localhost proxy-server: STDOUT: hdlr.handle(record)
Feb 27 14:00:15 localhost proxy-s...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/159530
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=2080f7dbd897d6130542dbf88e37641a41625eb5
Submitter: Jenkins
Branch: master

commit 2080f7dbd897d6130542dbf88e37641a41625eb5
Author: Alistair Coles <email address hidden>
Date: Thu Feb 26 15:16:22 2015 +0000

    Fix tempauth acl checks when simplejson has no speedups

    As documented in linked bug report, tempauth unit tests
    were seen to fail on a system where simplejson was
    installed but without the speedups extension. This
    is because the tempauth account acl validation checks
    that values are type str, but without the speedups
    extension the json parser is returning unicode objects.

    Fix is to have the acl validator tolerate those objects
    being unicode or str.

    Also change common/bufferedhttp.py to coerce ring device
    to type str when constructing a path, in order to avoid
    a UnicodeDecodeError when httplib sends a message that
    has non-ascii header values.

    Change-Id: I01524282cbaa25dc4b6dfa09f3f4723516cdba99
    Closes-Bug: 1425776

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/hummingbird
Review: https://review.openstack.org/202227

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Barton (<email address hidden>) on branch: feature/hummingbird
Review: https://review.openstack.org/202227
Reason: Apparently I did this wrong.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/hummingbird
Review: https://review.openstack.org/202230

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/202230
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=f7cb1777e1b514b3345b9e516ed8f89ad1a4ae87
Submitter: Jenkins
Branch: feature/hummingbird

commit 51f806d3e3d3a1fcbc80d2f7d7ddbe5cc4c024c9
Author: John Dickinson <email address hidden>
Date: Tue Jul 14 20:49:08 2015 -0700

    remove Python 2.6 from the classifier

    Change-Id: I67233e9c7b69826242546bd6bd98c24b81070579

commit 278adf5c20101a191979ce1e4d6277e5f209149e
Author: Hisashi Osanai <email address hidden>
Date: Tue Jul 14 15:33:45 2015 +0900

    Make logic of unit tests responsive to the method names

    The two methods, test_authorize_succeeds_for_tenant_name_in_roles and
    test_authorize_succeeds_for_tenant_id_in_roles, have names that don't
    match what they are testing. tenant_name and tenant_id need to be
    switched.

    Change-Id: I7cb0a7d2b2111127fd5d6b55f2da6a3eadf2235d

commit 1cc3eff958fdd4fb07c2b74c52df7829d3125466
Author: Victor Stinner <email address hidden>
Date: Fri Jul 10 13:04:44 2015 +0200

    Fixes for mock 1.1

    The new release of mock 1.1 is more strict. It helped to find bugs in
    tests.

    Closes-Bug: #1473369
    Change-Id: Id179513c6010d827cbcbdda7692a920e29213bcb

commit ff192cfe5705324497a389aa2f22227d75dc0f8e
Author: janonymous <email address hidden>
Date: Wed Jul 8 18:38:22 2015 +0530

    Replace reduce and unichr , these are no longer available in py3

    * Replace reduce() with six.moves.reduce()
    * Replace unichr with six.unichr

    Change-Id: I2038e47e0a6522dd992fd2a4aeff981cf7750fe0

commit 4beceab4f4be99f14025815cf7ed4510ea77f460
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Jul 9 06:14:56 2015 +0000

    Imported Translations from Transifex

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I9ff1dde06be45fc7d6c441a1e1c07221f839a9a1

commit 56ee39a7e13417203c5e1816d7a3184a07f85826
Author: Matthew Oliver <email address hidden>
Date: Thu Jul 9 15:19:32 2015 +1000

    Ring builder code clean up follow up patch

    This is a simple change that cleans up a NIT from Sam's 'stop moving
    partitions unnecessarily when overload is on' patch.

    Change-Id: I9d9f1cc23e2bb625d8e158f4d3f64e10973176a1

commit 6cafd0a4c0bb8f311fc59df580b42e801214effd
Author: Oshrit Feder <email address hidden>
Date: Wed Jul 8 15:18:22 2015 +0300

    Fix Container Sync example

    Container-sync realm uses cluster_ as a prefix to specify clusters'
    names. At use, the prefix should not be included. Fixing the examples
    and sample conf to make it clearer that only the name of the cluster
    should be passed.

    Change-Id: I2e521d86faffb59e1b45d3f039987ee023c5e939

commit 125238612f58481316db68d7087252bb7729f447
Author: Janie Richling <email address hidden>
Date: Sat Jul 4 17:08:32 2015 -0500

    Add CORS unit tests to base

    In earlier versions of swift when a request was made with an
    existing origin, but without any CORS settings in the container,
    it was possible to get an u...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/crypto
Review: https://review.openstack.org/205579

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/205579
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=8ab46b64365b8eab80680f2562f81e8adb3032a3
Submitter: Jenkins
Branch: feature/crypto

commit 89f705e8aab144092d40a13fc4ef19ffef5f3eba
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Jul 23 06:11:27 2015 +0000

    Imported Translations from Transifex

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I94cf347564cb33977f33b1e64259bcb39a8cf809

commit a65e9db8752793ec37b594dc9eca5066171825db
Author: Christian Schwede <email address hidden>
Date: Wed Jul 22 10:43:17 2015 +0000

    Removing commented out code in test/unit/account/test_backend.py

    Noticed this while reviewing another change. Looks like the test itself already
    ensures correct functionality of the reclaim() method in AccountBroker without
    the commented code, thus removing this stale code.

    Change-Id: I6a26a7591adef9fd794ca68a4e9c493d1127f93c

commit 99d052772a9585e0befdfd292fd03aefde77180a
Author: Kota Tsuyuzaki <email address hidden>
Date: Mon Jul 13 01:12:43 2015 -0700

    Fix 499 client disconnected on COPY EC object

    Currently, a COPY request for an EC object might go to fail as 499 Client
    disconnected because of the difference between destination request content
    length and actual transferred bytes.

    That is because the conditional response status and content length for
    an EC object range GET is handled at calling the response instance on
    proxy server. Therefore the calling response instance (resp()) will change
    the conditional status from 200 (HTTP_OK) to 206 (PartialContent) and will
    change the content length for the range GET.

    In EC case, sometimes Swift needs whole stored contents to decode a segment.
    It will make 200 HTTP OK response from object-server and proxy-server
    will unfortunately set whole content length to the destination content
    length and it makes the bug 1467677.

    This patch introduces a new method "fix_conditional_response" for
    swift.common.swob.Response that calling _response_iter() and cached the
    iter in the Response instance. By calling it, Swift can set correct condtional
    response any time after setting whole content_length to the response
    instance like EC case.

    Change-Id: If85826243f955d2f03c6ad395215c73daab509b1
    Closes-Bug: #1467677

commit 62ed4f81ef80440550633eaaaa962a4f9383c2d3
Author: Timur Alperovich <email address hidden>
Date: Tue Jul 14 16:56:44 2015 -0700

    Add two functional tests for delimiter.

    The first test verifies that a delimiter will trim entries beyond the
    first matching instance of delimiter (after the given matching prefix,
    if any) and squash duplicates. So, when setting the delimiter
    to "-", given blobs "test", "test-foo" and "test-bar-baz", we expect
    only "test" (no matching delim) and "test-" (trim all characters after
    the first "-", and squash duplicates).

    The second test verifies that when a prefix is provid...