Bug #1466138 “EC reconstructor (ssync_sender) got exceptions whi...” : Bugs : OpenStack Object Storage (swift)

Revision history for this message

clayg (clay-gerrard) wrote on 2015-06-17:

#1

It would be interesting to turn on DEBUG logging on .14 and .13 and see if they're logging the 507 response to the REPLICATE request.

Also there's a change on master post-kilo that may help with this issue [1] - can you verify if this is reproducible with the latest code?

1. https://review.openstack.org/#/c/177836/

Revision history for this message

Charles Hsu (charles0126) wrote on 2015-06-17:

#2

@clayg,

I'll turn it on to see what I get.

Should I cherry pick this patch(https://review.openstack.org/#/c/177836/), or I should move to latest code?

description:

updated

John Dickinson (notmyname) on 2015-06-17

tags:

added: ec

clayg (clay-gerrard) on 2015-06-25

Changed in swift:
status:	New → Confirmed

Revision history for this message

clayg (clay-gerrard) wrote on 2015-06-25:

#3

read the response from the server Edit (584 bytes, text/plain)

this is very closely related to lp bug #1466138

On the latest code here's an example conversation

Jun 25 08:17:15 saio object-6030: STDERR: (26055) accepted ('127.0.0.1', 42375)
Jun 25 08:17:15 saio object-6030: 127.0.0.1 - - [25/Jun/2015:08:17:15 +0000] "SSYNC /sdb7/65" 507 - "-" "-" "-" 0.0008 "-" 26055 1

Jun 25 08:17:15 saio object-6020: 127.0.0.1:6030/sdb7/65 Expected status 200; got 507

    Jun 25 08:17:15 saio object-6030: STDERR: Traceback (most recent call last):
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
    Jun 25 08:17:15 saio object-6030: STDERR: func(*args, **kwargs)
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 686, in process_request
    Jun 25 08:17:15 saio object-6030: STDERR: proto.__init__(sock, address, self)
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__
    Jun 25 08:17:15 saio object-6030: STDERR: self.handle()
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    Jun 25 08:17:15 saio object-6030: STDERR: self.handle_one_request()
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 358, in handle_one_request
    Jun 25 08:17:15 saio object-6030: STDERR: self.handle_one_response()
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 507, in handle_one_response
    Jun 25 08:17:15 saio object-6030: STDERR: while self.environ['eventlet.input'].read(MINIMUM_CHUNK_SIZE):
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 189, in read
    Jun 25 08:17:15 saio object-6030: STDERR: return self._chunked_read(self.rfile, length)
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 179, in _chunked_read
    Jun 25 08:17:15 saio object-6030: STDERR: self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)
    Jun 25 08:17:15 saio object-6030: STDERR: ValueError: invalid literal for int() with base 16: ''
    Jun 25 08:17:15 saio object-6030: STDERR: (26055) accepted ('127.0.0.1', 42376)
    Jun 25 08:17:15 saio object-6030: 127.0.0.1 - - [25/Jun/2015:08:17:15 +0000] "REPLICATE /sdb7/65/852" 507 - "-" "-" "obj-reconstructor 26062" 0.0003 "-" 26055 1

object-6030 is blowing up *after* it responds 507 and *after* object-6020 logs the 507 response.

This seems to indicate that object-6020 isn't closing down the connection well.

this is very closely related to lp bug #1466138

On the latest code here's an example conversation

Jun 25 08:17:15 saio object-6030: STDERR: (26055) accepted ('127.0.0.1', 42375)
    Jun 25 08:17:15 saio object-6030: 127.0.0.1 - - [25/Jun/2015:08:17:15 +0000] "SSYNC /sdb7/65" 507 - "-" "-" "-" 0.0008 "-" 26055 1

Jun 25 08:17:15 saio object-6020: 127.0.0.1:6030/sdb7/65 Expected status 200; got 507

Jun 25 08:17:15 saio object-6030: STDERR: Traceback (most recent call last):
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
    Jun 25 08:17:15 saio object-6030: STDERR: func(*args, **kwargs)
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 686, in process_request
    Jun 25 08:17:15 saio object-6030: STDERR: proto.__init__(sock, address, self)
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__
    Jun 25 08:17:15 saio object-6030: STDERR: self.handle()
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    Jun 25 08:17:15 saio object-6030: STDERR: self.handle_one_request()
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 358, in handle_one_request
    Jun 25 08:17:15 saio object-6030: STDERR: self.handle_one_response()
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 507, in handle_one_response
    Jun 25 08:17:15 saio object-6030: STDERR: while self.environ['eventlet.input'].read(MINIMUM_CHUNK_SIZE):
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 189, in read
    Jun 25 08:17:15 saio object-6030: STDERR: return self._chunked_read(self.rfile, length)
    Jun 25 08:17:15 saio object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 179, in _chunked_read
    Jun 25 08:17:15 saio object-6030: STDERR: self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)
    Jun 25 08:17:15 saio object-6030: STDERR: ValueError: invalid literal for int() with base 16: ''
    Jun 25 08:17:15 saio object-6030: STDERR: (26055) accepted ('127.0.0.1', 42376)
    Jun 25 08:17:15 saio object-6030: 127.0.0.1 - - [25/Jun/2015:08:17:15 +0000] "REPLICATE /sdb7/65/852" 507 - "-" "-" "obj-reconstructor 26062" 0.0003 "-" 26055 1

object-6030 is blowing up *after* it responds 507 and *after* object-6020 logs the 507 response.

This seems to indicate that object-6020 isn't closing down the connection well.

Revision history for this message

clayg (clay-gerrard) wrote on 2015-06-26:

#4

patch is

https://review.openstack.org/#/c/195457/

Changed in swift:
importance:	Undecided → High
assignee:	nobody → clayg (clay-gerrard)
status:	Confirmed → In Progress

Revision history for this message

clayg (clay-gerrard) wrote on 2015-07-07:

#5

https://review.openstack.org/#/c/195457/ merged

Changed in swift:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-15: Fix proposed to swift (feature/hummingbird)

#6

Fix proposed to branch: feature/hummingbird
Review: https://review.openstack.org/202227

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-15: Change abandoned on swift (feature/hummingbird)

#7

Change abandoned by Michael Barton (<email address hidden>) on branch: feature/hummingbird
Review: https://review.openstack.org/202227
Reason: Apparently I did this wrong.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-15: Fix proposed to swift (feature/hummingbird)

#8

Fix proposed to branch: feature/hummingbird
Review: https://review.openstack.org/202230

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-15: Fix merged to swift (feature/hummingbird)

#9

Download full text (72.8 KiB)

Reviewed: https://review.openstack.org/202230
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=f7cb1777e1b514b3345b9e516ed8f89ad1a4ae87
Submitter: Jenkins
Branch: feature/hummingbird

commit 51f806d3e3d3a1fcbc80d2f7d7ddbe5cc4c024c9
Author: John Dickinson <email address hidden>
Date: Tue Jul 14 20:49:08 2015 -0700

remove Python 2.6 from the classifier

Change-Id: I67233e9c7b69826242546bd6bd98c24b81070579

commit 278adf5c20101a191979ce1e4d6277e5f209149e
Author: Hisashi Osanai <email address hidden>
Date: Tue Jul 14 15:33:45 2015 +0900

Make logic of unit tests responsive to the method names

    The two methods, test_authorize_succeeds_for_tenant_name_in_roles and
    test_authorize_succeeds_for_tenant_id_in_roles, have names that don't
    match what they are testing. tenant_name and tenant_id need to be
    switched.

Change-Id: I7cb0a7d2b2111127fd5d6b55f2da6a3eadf2235d

commit 1cc3eff958fdd4fb07c2b74c52df7829d3125466
Author: Victor Stinner <email address hidden>
Date: Fri Jul 10 13:04:44 2015 +0200

Fixes for mock 1.1

The new release of mock 1.1 is more strict. It helped to find bugs in
tests.

Closes-Bug: #1473369
Change-Id: Id179513c6010d827cbcbdda7692a920e29213bcb

commit ff192cfe5705324497a389aa2f22227d75dc0f8e
Author: janonymous <email address hidden>
Date: Wed Jul 8 18:38:22 2015 +0530

Replace reduce and unichr , these are no longer available in py3

* Replace reduce() with six.moves.reduce()
* Replace unichr with six.unichr

Change-Id: I2038e47e0a6522dd992fd2a4aeff981cf7750fe0

commit 4beceab4f4be99f14025815cf7ed4510ea77f460
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Jul 9 06:14:56 2015 +0000

Imported Translations from Transifex

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I9ff1dde06be45fc7d6c441a1e1c07221f839a9a1

commit 56ee39a7e13417203c5e1816d7a3184a07f85826
Author: Matthew Oliver <email address hidden>
Date: Thu Jul 9 15:19:32 2015 +1000

Ring builder code clean up follow up patch

This is a simple change that cleans up a NIT from Sam's 'stop moving
partitions unnecessarily when overload is on' patch.

Change-Id: I9d9f1cc23e2bb625d8e158f4d3f64e10973176a1

commit 6cafd0a4c0bb8f311fc59df580b42e801214effd
Author: Oshrit Feder <email address hidden>
Date: Wed Jul 8 15:18:22 2015 +0300

Fix Container Sync example

    Container-sync realm uses cluster_ as a prefix to specify clusters'
    names. At use, the prefix should not be included. Fixing the examples
    and sample conf to make it clearer that only the name of the cluster
    should be passed.

Change-Id: I2e521d86faffb59e1b45d3f039987ee023c5e939

commit 125238612f58481316db68d7087252bb7729f447
Author: Janie Richling <email address hidden>
Date: Sat Jul 4 17:08:32 2015 -0500

Add CORS unit tests to base

    In earlier versions of swift when a request was made with an
    existing origin, but without any CORS settings in the container,
    it was possible to get an u...

Reviewed:  https://review.openstack.org/202230
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=f7cb1777e1b514b3345b9e516ed8f89ad1a4ae87
Submitter: Jenkins
Branch:    feature/hummingbird

commit 51f806d3e3d3a1fcbc80d2f7d7ddbe5cc4c024c9
Author: John Dickinson <me@not.mn>
Date:   Tue Jul 14 20:49:08 2015 -0700

remove Python 2.6 from the classifier
    
    Change-Id: I67233e9c7b69826242546bd6bd98c24b81070579

commit 278adf5c20101a191979ce1e4d6277e5f209149e
Author: Hisashi Osanai <osanai.hisashi@jp.fujitsu.com>
Date:   Tue Jul 14 15:33:45 2015 +0900

Make logic of unit tests responsive to the method names
    
    The two methods, test_authorize_succeeds_for_tenant_name_in_roles and
    test_authorize_succeeds_for_tenant_id_in_roles, have names that don't
    match what they are testing. tenant_name and tenant_id need to be
    switched.
    
    Change-Id: I7cb0a7d2b2111127fd5d6b55f2da6a3eadf2235d

commit 1cc3eff958fdd4fb07c2b74c52df7829d3125466
Author: Victor Stinner <vstinner@redhat.com>
Date:   Fri Jul 10 13:04:44 2015 +0200

Fixes for mock 1.1
    
    The new release of mock 1.1 is more strict. It helped to find bugs in
    tests.
    
    Closes-Bug: #1473369
    Change-Id: Id179513c6010d827cbcbdda7692a920e29213bcb

commit ff192cfe5705324497a389aa2f22227d75dc0f8e
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Wed Jul 8 18:38:22 2015 +0530

Replace reduce and unichr , these are no longer available in py3
    
    * Replace reduce() with six.moves.reduce()
    * Replace unichr with six.unichr
    
    Change-Id: I2038e47e0a6522dd992fd2a4aeff981cf7750fe0

commit 4beceab4f4be99f14025815cf7ed4510ea77f460
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu Jul 9 06:14:56 2015 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I9ff1dde06be45fc7d6c441a1e1c07221f839a9a1

commit 56ee39a7e13417203c5e1816d7a3184a07f85826
Author: Matthew Oliver <matt@oliver.net.au>
Date:   Thu Jul 9 15:19:32 2015 +1000

Ring builder code clean up follow up patch
    
    This is a simple change that cleans up a NIT from Sam's 'stop moving
    partitions unnecessarily when overload is on' patch.
    
    Change-Id: I9d9f1cc23e2bb625d8e158f4d3f64e10973176a1

commit 6cafd0a4c0bb8f311fc59df580b42e801214effd
Author: Oshrit Feder <oshritf@il.ibm.com>
Date:   Wed Jul 8 15:18:22 2015 +0300

Fix Container Sync example
    
    Container-sync realm uses cluster_ as a prefix to specify clusters'
    names. At use, the prefix should not be included. Fixing the examples
    and sample conf to make it clearer that only the name of the cluster
    should be passed.
    
    Change-Id: I2e521d86faffb59e1b45d3f039987ee023c5e939

commit 125238612f58481316db68d7087252bb7729f447
Author: Janie Richling <jrichli@us.ibm.com>
Date:   Sat Jul 4 17:08:32 2015 -0500

Add CORS unit tests to base
    
    In earlier versions of swift when a request was made with an
    existing origin, but without any CORS settings in the container,
    it was possible to get an unhandled exception due to a method call
    on the "None" return of cors.get('allow_origin', '').
    
    Unit tests have been added to assert that this problem cannot go
    undetected again.
    
    Change-Id: Ia74896dabe1cf5a307c551b15a43ab1fd789c213
    Fixes: bug 1468782

commit e87a74695e2f55dbb0e01721de9d46e230e36008
Author: John Dickinson <me@not.mn>
Date:   Sat Jun 6 13:03:15 2015 -0700

add domain_remap to /info
    
    Also added the value of default_reseller_prefix to /info. I did not
    add the reseller_prefixes value to /info because deployers may not
    want to expose all of the resellers that are available.
    
    Change-Id: I9ca2f002dc395913bb646390d6908dfb8f554df1

commit 12473104f005b76ebe4b6dbe118a5ac37518eb52
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Jul 7 18:02:53 2015 -0700

Cleanup error messages in ECDiskFileWriter.commit
    
    I think the error messages make better sense and it's more obvious what the
    error handling code was trying to bring to the surface for operators.
    
    If you run this test with nosetests --verbose --nocapture
    
    test_diskfile.py:TestECDiskFile.test_commit_fsync_dir_raises_DiskFileErrors
    
    ... you can review most of the potential log output
    
    Change-Id: I40c9d77f44e087ee61d9642e924b0a4039d6ca9a

commit bd3717df65f57da9e7f06292ba349442d17fc939
Author: John Dickinson <me@not.mn>
Date:   Thu Jul 2 09:54:30 2015 -0700

update AUTHORS file
    
    Change-Id: I509ceeb522bd83c407beb40c22c2924a1d92b8ff

commit cd20961abd20f01a14554c14d4f61a7f0d8deb88
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Sun Jul 5 10:59:50 2015 +0530

Replace dict.iteritems() with dict.items() in bin directory of swift.
    
    The iteritems() of Python 2 dictionaries has been renamed to items() on
    Python 3.
    
    Change-Id: I4bdc064c90bab56cd60f2dca2a5a78426ffbb31c

commit 8eca02d3a1db1e0ed68b15ed41bc62e6c89870c7
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Fri Jul 3 00:45:58 2015 +0530

Replace xrange() with six.moves.range()
    
    xrange is no longer available in py3.
    replacing xrange with range from six.moves
    
    Change-Id: Ib863c316a0724bd9c4f53c2e5a8d1bcd42c4dc5a

commit 10f367224d5f30c3398169ffe3087f225f3343fc
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Fri Jul 3 00:05:36 2015 +0530

Fix Python 3 issues
    
    * Replace long with six.integer_type
    
    six.integer_type :In Python 2, this is long and int, and in Python 3,
    just int.
    
    'long' is an undefined variable in py3.
    
    Change-Id: Ibe4c5d0d81fe883f4fe33be93a68d814a228ad28

commit 2328983b7535d24d08942e62095b4346f2626566
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Wed Jun 24 17:07:32 2015 -0700

Stop moving partitions unnecessarily when overload is on.
    
    When overload was on and in use, the ring builder was unnecessarily
    moving partitions. It would converge on a good solution and settle
    down eventually, but it moved more partitions than necessary along the
    way.
    
    There are three partition gatherers used in the ring builder:
    dev-removed, dispersion, and weight, in that order. The dev-removed
    gatherer will pick up all partitions on removed devices. The
    dispersion gatherer picks up replicas of partitions that are
    suboptimally dispersed. The weight gatherer picks up partitions on
    devices which are overweight.
    
    The weight gatherer was not overload-aware, so it would pick up
    partitions that did not need to move. Consider a device that would
    normally have 100 partitions assigned, assume we set overload to 0.1
    so that this device will hold up to 110 (10 extra) for the sake of
    dispersion, and assume the device actually has 104 partitions assigned
    to it. The correct behavior is to gather nothing from this device
    because it has fewer than the maximum. Prior to this commit, the
    weight gatherer would remove 4 partitions from this device; they would
    subsequently be reassigned by the overload-aware partition placer
    (_reassign_parts()). In a ring with multiple overloaded devices, the
    builder would pick up some partitions from each, shuffle them, and
    then assign them back to those same devices. Obviously, this just
    created extra replication work for no benefit.
    
    Now, the weight gatherer takes overload into account, and will no
    longer needlessly gather partitions.
    
    That's not all, though; this change worsened the behavior of a ring
    with more overload than necessary. Before, the ring would balance as
    best it could, using the minimal amount of overload. With the
    weight-gatherer change, the ring builder will stop gathering
    partitions once a device reaches its maximum-permissible assignment
    including overload.
    
    For example, imagine a 3-replica, 4-zone ring with overload=0.2 and
    weights:
    
      z1: 100
      z2: 60
      z3: 60
      z4: 60
    
    Since z1 has more than 1/3 of the weight, z2, z3, and z4 must take
    more than their fair share for the sake of dispersion.
    
    Now, turn up the weights some:
    
      z1: 100
      z2: 80
      z3: 80
      z4: 80
    
    Overload is no longer needed; this ring can balance. However, z2, z3,
    and z4 would end up keeping more than their fair share since (a) they
    already had extra due to earlier conditions, and (b) the weight
    gatherer won't pick up partitions from them since they're not
    overburdened once it takes overload into account.
    
    To fix this, we compute the minimum overload factor required for
    optimal dispersion and then use min(user-input-overload,
    minimum-overload) during rebalance. This way, we don't overload
    devices more than the user says, but if the user sets overload higher
    than necessary, we'll still give the best balance possible.
    
    Change-Id: If5666ba654ee25da54f9144f3b78840273a49627

commit ccf0758ef1f362312a6197f86455f680ebd990d3
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Jun 11 15:40:28 2015 -0700

Add ring-builder analyzer.
    
    This is a tool to help developers quantify changes to the ring
    builder. It takes a scenario (JSON file) describing the builder's
    basic parameters (part_power, replicas, etc.) and a number of
    "rounds", where each round is a set of operations to perform on the
    builder. For each round, the operations are applied, and then the
    builder is rebalanced until it reaches a steady state.
    
    The idea is that a developer observes the ring builder behaving
    suboptimally, writes a scenario to reproduce the behavior, modifies
    the ring builder to fix it, and references the scenario with the
    commit so that others can see that things have improved.
    
    I decided to write this after writing my fourth or fifth hacky one-off
    script to reproduce some bad behavior in the ring builder.
    
    Change-Id: I114242748368f142304aab90a6d99c1337bced4c

commit e860ec138913dc4c450725766a77e76a7101d06e
Author: Christian Schwede <cschwede@redhat.com>
Date:   Fri May 29 10:11:44 2015 +0000

Ignore invalid PIDs in swift-init
    
    While reviewing ab9f6340 I recognized that invalid pids in the pid files raise
    an exceptions (in case it is not a number), and also accept a 0 that is
    impossible to use as a pid for any Swift process. In the latter case the process
    was recognized as runnning, even if it did not - and thus restarting didn't
    worked.
    
    This patch ensures a better handling of any non-numeric value and also removes a
    pid file if it uses 0 as the pid.
    
    Change-Id: I09c415c7a75e4b9a0574a08f46f48fff5bb11d02

commit 0a6f2ab870a868e3036dbb212c0e7dd8375a39eb
Author: asettle <alexandra.settle@rackspace.com>
Date:   Fri Jun 26 10:57:35 2015 +1000

Correcting minor grammatical errors
    
    1. Removing unnecessary semi colon
    2. Removing unnecessary content
    
    Change-Id: Ie95403a4d96db2b8465e75495061fc059098c922

commit 4d4db054d35c7e9de9ed28ac24965e8408dd9f82
Author: Minwoo Bae <minwoob@us.ibm.com>
Date:   Mon May 18 14:08:25 2015 -0500

After the .durable has been written, fsync the directory.
    
    Added try-except statements in _finalize_durable() to fsync the directory
    after a successful fsync of the .durable file.
    
    Added test_commit_fsync_dir_raises_DiskFileErrors() for testing whether
    certain assertions hold for the change to include fsync_dir().
    
    Some more error details have been included in the logger.
    
    Closes-Bug: #1470651
    Change-Id: I4791d75ade8542678369ba0811ef39af6e955cc6

commit d4d7c0e1b11af8f20f965f410901a56e1f8ee69e
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Jun 30 17:24:13 2015 -0700

Fix race in test_server.TestObjectController.test_PUT_ssync_multi_frag
    
    If the test ran across a one second boundary it would fail because while
    the timestamp normalization was doing some rounding it was making no
    attempt to reuse the same timestamp on subsequent requests.
    
    Change-Id: Ic560032bcfacd6f0d10cfc0f4f10e5d6c2bc8dd5

commit a88bd6e6b9253b6488a2bf1a354ce29b51f37fe1
Author: ZhiQiang Fan <aji.zqfan@gmail.com>
Date:   Sun Jun 28 22:04:46 2015 -0700

fix api overview document
    
    The content about Object Storage HTTP requests constraints seems to
    be a table but broken, this patch fixes it into correct rst table format.
    
    Change-Id: I1b4c62da3e6d91add3ee0218707c1628c8f04b33

commit c95a0efe7973a660f4abcd13f4ea6211d72c4baa
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu Jun 25 01:44:10 2015 -0700

Make ssync_sender a better HTTP client
    
    When a server responses with an error - if that error includes a body - the
    client should read the body.  This cleans up some ugly eventlet/wsgi.server log
    output related to chunked transfer disconnect (invalid literal for int() with
    base 16).
    
    Change-Id: Ibd06ddee9f216fce07fa33c3a7d8306b59eb6d77
    Closes-Bug: #1466138

commit 44b76a1b1b83ac9563010f1ddfca5fca76e567bf
Author: Minwoo Bae <minwoob@us.ibm.com>
Date:   Thu Jun 18 14:21:06 2015 -0500

EC Reconstructor: Do not reconstruct existing fragments.
    
    The EC reconstructor needs to verify that the fragment needing to
    be reconstructed does not reside in the collection of node responses.
    Otherwise, resources will be spent unnecessarily reconstructing
    the fragment. Moreover, this could cause a segfault on some backends.
    
    This change adds the necessary verification steps to make sure
    that a fragment will only be rebuilt in the case it is missing from
    the other fragment archives.
    
    Added some tests to provide coverage for these scenarios.
    
    Change-Id: I91f3d4af52cbc66c9f7ce00726f247b5462e66f9
    Closes-Bug: #1452553

commit d124ce5792d93e1efcdb083211ecef381f7b7173
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu Jun 25 01:35:07 2015 -0700

Fix ValueError in ssync_receiver
    
    httplib's putheader method will cast whatever you give it to a string.
    where we allow the default dict.get default of None to be passed to
    putheader unmodified ssync_receiver is surpised that the non-empty
    string isn't able to be converted to an integer.
    
    We can avoid surprising the ssync_receiver in this way by sending the
    empty string as a better default.
    
    Change-Id: Ie9df9927ff4d3dd3f334647f883b2937d0d81030

commit edfca861b6fa39972df276fb1f37aa81583a605d
Author: Christian Schwede <cschwede@redhat.com>
Date:   Fri Jun 26 08:54:22 2015 +0000

Increase httplib._MAXHEADERS
    
    Python 2.7.9+ and 3.2.6+ limits the number of maximum headers in httplib to 100
    [1,2,3]. This setting is too low for Swift.
    
    By default the maximum number of allowed headers depends on the number of max
    allowed metadata settings plus a default value of 32 for regular http headers.
    If for some reason this is not enough (custom middleware for example) it can be
    increased with the extra_header_count constraint.
    
    [1] https://bugs.python.org/issue16037
    [2] https://hg.python.org/cpython/raw-file/15c95b7d81dc/Misc/NEWS
    [3] https://hg.python.org/cpython/raw-file/v3.2.6/Misc/NEWS
    
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
    Co-Authored-By: Thomas Herve <therve@redhat.com>
    
    Change-Id: I388fd697ec88476024b0e9f1ae75ba35ff765282

commit 81ceee056de604d81a921280f453731eadf7be09
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Mon Jun 22 23:24:19 2015 -0700

Add one more test for ssync_receiver
    
    To prevent 409 conflict on a primary node during ssyncing,
    ssync-receiver should add x-ssync-backend-frag-index generated from
    x-ssync-backend-node-index of the SSYNC replication request header.
    The change is done by previous work[1], but we need more test for that.
    
    This patch addes one more assertion if the x-ssync-backend-frag-index
    is in the ssync subrequest correctly.
    
    *BONUS*
    Fix some weird mock and add some sanities.
    
    1: https://review.openstack.org/#/c/191521/
    
    Change-Id: I27aff713c69017c0bc4c60b4833184e1285595d7

commit c2c1366b0794bcdbbbaef7fedaef2cab19cd8db8
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Wed Jun 24 21:07:16 2015 +0530

Fixed Errors in Swift-Form Post middleware section rst.
    
    Inline markup cannot be nested in RST.
    Fixing markups by using a single option for the whole line and making
    some minor error corrections.
    
    Change-Id: I42bfd7dfe5c93a69436ecc5d154f2e61ca83fa82

commit 514daea6317edc02587f7abfaae8770e1ef9cc4b
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Fri Jun 12 23:40:51 2015 +0530

Fixed Formatting Error in Swift -Form Post middleware section.
    
    Fixed Formatting error in ``action=”SWIFT_URL``” => action=”``SWIFT_URL``”
    
    Change-Id: Id461129ef7f5b6412f94d36920c942a4181c0eb7
    Closes-Bug: #1464740

commit e70b66586e13718e7ac0078c2f2cd1135c61a1f1
Author: Victor Stinner <vstinner@redhat.com>
Date:   Wed Jun 24 09:36:37 2015 +0200

Replace dict.iteritems() with dict.items()
    
    The iteritems() of Python 2 dictionaries has been renamed to items() on
    Python 3. According to a discussion on the openstack-dev mailing list,
    the overhead of creating a temporary list using dict.items() on Python 2
    is very low because most dictionaries are small:
    
    http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
    
    Patch generated by the following command:
    
        sed -i 's,iteritems,items,g' \
          $(find swift -name "*.py") \
          $(find test -name "*.py")
    
    Change-Id: I6070bb6c684be76e8e77222a7d280ec6edd43496

commit e6165a7879d796efd3992260ef23a7f95ceeab32
Author: paul luse <paul.e.luse@intel.com>
Date:   Mon May 25 14:41:42 2015 -0700

Add policy support to dispersion tools
    
    Doesn't work for anything other than policy 0. updated to allow user
    to specify policy name on cmd line (as with object-info) which
    then makes populate/report work with 3x, 2x, or EC style policies
    
    Change-Id: Ib7c298f0f6d666b1ecca25315b88539f45cf9f95
    Closes-Bug: 1458688

commit e5c962a28ce99f01c6cd0b68b7434ddcd0e55f62
Author: Victor Stinner <vstinner@redhat.com>
Date:   Mon May 25 18:28:02 2015 +0200

Replace xrange() with six.moves.range()
    
    Patch generated by the xrange operation of the sixer tool:
    https://pypi.python.org/pypi/sixer
    
    Manual changes:
    
    * Fix indentation for pep8 checks
    * Fix TestGreenthreadSafeIterator.test_access_is_serialized of
      test.unit.common.test_utils:
      replace range(1, 11) with list(range(1, 11))
    * Fix UnsafeXrange docstring, revert change
    
    Change-Id: Icb7e26135c5e57b5302b8bfe066b33cafe69fe4d

commit 4a5b851207c98a380a32e27a183dd98022be9d16
Author: Christian Schwede <cschwede@redhat.com>
Date:   Mon Jun 22 14:48:41 2015 +0200

Add note about updatedb to the docs
    
    Change-Id: Ia1aa0bb1f93ee487e2f7ddf76a7a08efa8f3ba41

commit ac8a76958510646c369df9ffc2bf22005e43d153
Author: paul luse <paul.e.luse@intel.com>
Date:   Sat Jun 13 11:03:56 2015 -0700

EC Ssync:  Update parms to include node and frag indices
    
    Previously we sent the ssync backend frag index based on the node
    index.  We need to be more specific for ssync to handle both sync
    and revert cases so now we send the frag index based on the job
    contents (as determined by the ec recon)) and the node index
    as a new header based on, well, the node index.
    
    The rcvr can now validate the incoming pair to reject (400) when
    a primary node is being asked to accept fragments that don't
    belong to it.  Additionally, by having the frag index the
    rcvr can reject (409) an attempt to accept a fragment when its
    a handoff and already has one that needs to be reverted.
    
    Fixes-bug: #1452619
    Change-Id: I8287b274bbbd00903c1975fe49375590af697be4

commit 12d8a53fffea6e4bed8ba3d502ce625f5c6710b9
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Jun 18 12:58:03 2015 -0700

Get better at closing WSGI iterables.
    
    PEP 333 (WSGI) says: "If the iterable returned by the application has
    a close() method, the server or gateway must call that method upon
    completion of the current request[.]"
    
    There's a bunch of places where we weren't doing that; some of them
    matter more than others. Calling .close() can prevent a connection
    leak in some cases. In others, it just provides a certain pedantic
    smugness. Either way, we should do what WSGI requires.
    
    Noteworthy goofs include:
    
      * If a client is downloading a large object and disconnects halfway
        through, a proxy -> obj connection may be leaked. In this case,
        the WSGI iterable is a SegmentedIterable, which lacked a close()
        method. Thus, when the WSGI server noticed the client disconnect,
        it had no way of telling the SegmentedIterable about it, and so
        the underlying iterable for the segment's data didn't get
        closed.
    
        Here, it seems likely (though unproven) that the object server
        would time out and kill the connection, or that a
        ChunkWriteTimeout would fire down in the proxy server, so the
        leaked connection would eventually go away. However, a flurry of
        client disconnects could leave a big pile of useless connections.
    
      * If a conditional request receives a 304 or 412, the underlying
        app_iter is not closed. This mostly affects conditional requests
        for large objects.
    
    The leaked connections were noticed by this patch's co-author, who
    made the changes to SegmentedIterable. Those changes helped, but did
    not completely fix, the issue. The rest of the patch is an attempt to
    plug the rest of the holes.
    
    Co-Authored-By: Romain LE DISEZ <romain.ledisez@ovh.net>
    
    Change-Id: I168e147aae7c1728e7e3fdabb7fba6f2d747d937
    Closes-Bug: #1466549

commit df134df901a13c2261a8205826ea1aa8d75dc283
Author: Darrell Bishop <darrell@swiftstack.com>
Date:   Thu May 14 22:14:15 2015 -0700

Allow 1+ object-servers-per-disk deployment
    
    Enabled by a new > 0 integer config value, "servers_per_port" in the
    [DEFAULT] config section for object-server and/or replication server
    configs.  The setting's integer value determines how many different
    object-server workers handle requests for any single unique local port
    in the ring.  In this mode, the parent swift-object-server process
    continues to run as the original user (i.e. root if low-port binding
    is required), binds to all ports as defined in the ring, and forks off
    the specified number of workers per listen socket.  The child, per-port
    servers drop privileges and behave pretty much how object-server workers
    always have, except that because the ring has unique ports per disk, the
    object-servers will only be handling requests for a single disk.  The
    parent process detects dead servers and restarts them (with the correct
    listen socket), starts missing servers when an updated ring file is
    found with a device on the server with a new port, and kills extraneous
    servers when their port is found to no longer be in the ring.  The ring
    files are stat'ed at most every "ring_check_interval" seconds, as
    configured in the object-server config (same default of 15s).
    
    Immediately stopping all swift-object-worker processes still works by
    sending the parent a SIGTERM.  Likewise, a SIGHUP to the parent process
    still causes the parent process to close all listen sockets and exit,
    allowing existing children to finish serving their existing requests.
    The drop_privileges helper function now has an optional param to
    suppress the setsid() call, which otherwise screws up the child workers'
    process management.
    
    The class method RingData.load() can be told to only load the ring
    metadata (i.e. everything except replica2part2dev_id) with the optional
    kwarg, header_only=True.  This is used to keep the parent and all
    forked off workers from unnecessarily having full copies of all storage
    policy rings in memory.
    
    A new helper class, swift.common.storage_policy.BindPortsCache,
    provides a method to return a set of all device ports in all rings for
    the server on which it is instantiated (identified by its set of IP
    addresses).  The BindPortsCache instance will track mtimes of ring
    files, so they are not opened more frequently than necessary.
    
    This patch includes enhancements to the probe tests and
    object-replicator/object-reconstructor config plumbing to allow the
    probe tests to work correctly both in the "normal" config (same IP but
    unique ports for each SAIO "server") and a server-per-port setup where
    each SAIO "server" must have a unique IP address and unique port per
    disk within each "server".  The main probe tests only work with 4
    servers and 4 disks, but you can see the difference in the rings for the
    EC probe tests where there are 2 disks per server for a total of 8
    disks.  Specifically, swift.common.ring.utils.is_local_device() will
    ignore the ports when the "my_port" argument is None.  Then,
    object-replicator and object-reconstructor both set self.bind_port to
    None if server_per_port is enabled.  Bonus improvement for IPv6
    addresses in is_local_device().
    
    This PR for vagrant-swift-all-in-one will aid in testing this patch:
    https://github.com/swiftstack/vagrant-swift-all-in-one/pull/16/
    
    Also allow SAIO to answer is_local_device() better; common SAIO setups
    have multiple "servers" all on the same host with different ports for
    the different "servers" (which happen to match the IPs specified in the
    rings for the devices on each of those "servers").
    
    However, you can configure the SAIO to have different localhost IP
    addresses (e.g. 127.0.0.1, 127.0.0.2, etc.) in the ring and in the
    servers' config files' bind_ip setting.
    
    This new whataremyips() implementation combined with a little plumbing
    allows is_local_device() to accurately answer, even on an SAIO.
    
    In the default case (an unspecified bind_ip defaults to '0.0.0.0') as
    well as an explict "bind to everything" like '0.0.0.0' or '::',
    whataremyips() behaves as it always has, returning all IP addresses for
    the server.
    
    Also updated probe tests to handle each "server" in the SAIO having a
    unique IP address.
    
    For some (noisy) benchmarks that show servers_per_port=X is at least as
    good as the same number of "normal" workers:
    https://gist.github.com/dbishop/c214f89ca708a6b1624a#file-summary-md
    
    Benchmarks showing the benefits of I/O isolation with a small number of
    slow disks:
    https://gist.github.com/dbishop/fd0ab067babdecfb07ca#file-results-md
    
    If you were wondering what the overhead of threads_per_disk looks like:
    https://gist.github.com/dbishop/1d14755fedc86a161718#file-tabular_results-md
    
    DocImpact
    
    Change-Id: I2239a4000b41a7e7cc53465ce794af49d44796c6

commit af72881d1dcc46486d8d652b92a0ec9aa9ca2bfa
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon Jun 15 13:36:36 2015 -0700

Use just IP, not port, when determining partition placement
    
    In the ring builder, we place partitions with maximum possible
    dispersion across tiers, where a "tier" is region, then zone, then
    IP/port,then device. Now, instead of IP/port, just use IP. The port
    wasn't really getting us anything; two different object servers on two
    different ports on one machine aren't separate failure
    domains. However, if someone has only a few machines and is using one
    object server on its own port per disk, then the ring builder would
    end up with every disk in its own IP/port tier, resulting in bad (with
    respect to durability) partition placement.
    
    For example: assume 1 region, 1 zone, 4 machines, 48 total disks (12
    per machine), and one object server (and hence one port) per
    disk. With the old behavior, partition replicas will all go in the one
    region, then the one zone, then pick one of 48 IP/port pairs, then
    pick the one disk therein. This gives the same result as randomly
    picking 3 disks (without replacement) to store data on; it completely
    ignores machine boundaries.
    
    With the new behavior, the replica placer will pick the one region,
    then the one zone, then one of 4 IPs, then one of 12 disks
    therein. This gives the optimal placement with respect to durability.
    
    The same applies to Ring.get_more_nodes().
    
    Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
    
    Change-Id: Ibbd740c51296b7e360845b5309d276d7383a3742

commit 09e7477a391e7425af62a6516f43c69bc98b7080
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Mon Jun 15 22:10:45 2015 +0530

Replace it.next() with next(it) for py3 compat
    
    The Python 2 next() method of iterators was renamed to __next__() on
    Python 3. Use the builtin next() function instead which works on Python
    2 and Python 3.
    
    Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d

commit 5ad369c1cb0f480b164a5b5e7eb194081835c266
Author: Victor Stinner <vstinner@redhat.com>
Date:   Tue Jun 9 00:22:12 2015 +0200

Add six requirement
    
    The six module is needed to add Python 3 support to Swift.
    
    Change-Id: Ie637ed3458c7ff56c26834bca73203ed55604d74

commit 037a0c5dbe01a57b741d34988e21f4f763b6204b
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Sat Jun 6 17:42:41 2015 +0530

Object are sorted and concatenated by swift in cardinal order.
    
    Modified values in example in decimal to be more precise,
    added a small description on Object are sorted and concatenated
    by swift in cardinal order as a small phrase would be sufficient.
    
    Change-Id: I914ded4e5726e50bb93b05759c3bfb76edda53ab
    backport: none
    Closes-Bug: #1383893

commit 11e5c4adf0e02c8fdab0d8584fc943d96e1165cd
Author: Koert van der Veer <koert@cloudvps.com>
Date:   Tue Dec 16 11:15:19 2014 +0100

Allow default reseller prefix in domain_remap middleware
    
    Previously, the reseller prefix needed to be provided in the host name
    even when the domain was unique to that reseller. With the
    default_reseller_prefix, any domain which matches in this middleware,
    will will be passed on with a reseller prefix, whether or not it was
    provided.
    
    Change-Id: I5aa5ce78ad1ee2e3660cce4c3e07306f8999f02a
    Implements: blueprint domainremap-reseller-domains

commit c953e8cc9049e94cd01239d5e4a30a56f26a97d3
Author: Minwoo Bae <minwoob@us.ibm.com>
Date:   Tue Apr 28 13:18:32 2015 -0500

The hash_cleanup_listdir function should only be called when necessary.
    
    Introduced a conditional that will specify to _finalize_put() whether
    the put() method is being called within the context of ECDiskFileWriter.
    
    Included some unit tests that will test the number of calls to
    hash_cleanup_listdir() during put(), commit(), and delete()
    operations.
    
    Change-Id: I287b2648591fdbf686898dc9047beda80a64da0c

commit 93de6c73e587a672663fa0997f439b73b336769c
Author: Hisashi Osanai <osanai.hisashi@jp.fujitsu.com>
Date:   Tue Jun 2 17:00:42 2015 +0900

Add metadata size tests on the border
    
    This patch adds a test for a 204 when a single metadata item in a POST causes
    the backend aggregate constraints check to be on the border.
    
    Background:
    Overall metadata size constraint is enforced in the container and
    account backends as well as in the proxy controllers.
    Whereas the proxy controller can check that constraints are not exceeded
    by a single PUT or POST request, the backend checks that constraints are not
    exceeded by the aggregate of all PUTs and POSTs.
    
    The change [1] added a test for a 400 when a single metadata item in
    a POST causes the backend aggregate constraints check to go over limit.
    
    [1] I1489e29686013cbd3d70283d8756b548aea3c2e1
    
    Change-Id: Iac86ea71240ddde177e625c279c21aef67659d10

commit af734b3fb6fdbc11b8a0b775256e7642eb485b8d
Author: Mark Seger <Mark.Seger@hp.com>
Date:   Wed Jun 3 08:16:06 2015 -0400

Change usage help and Attention messages to warnings
    
    Change-Id: I1396aaffe36e739606f15f7fef37b11bd83f1fc1

commit 4f2ed8bcd0468f3b69d5fded274d8d6b02ac3d10
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Mar 20 09:56:30 2015 -0700

EC: support multiple ranges for GET requests
    
    This commit lets clients receive multipart/byteranges responses (see
    RFC 7233, Appendix A) for erasure-coded objects. Clients can already
    do this for replicated objects, so this brings EC closer to feature
    parity (ha!).
    
    GetOrHeadHandler got a base class extracted from it that treats an
    HTTP response as a sequence of byte-range responses. This way, it can
    continue to yield whole fragments, not just N-byte pieces of the raw
    HTTP response, since an N-byte piece of a multipart/byteranges
    response is pretty much useless.
    
    There are a couple of bonus fixes in here, too. For starters, download
    resuming now works on multipart/byteranges responses. Before, it only
    worked on 200 responses or 206 responses for a single byte
    range. Also, BufferedHTTPResponse grew a readline() method.
    
    Also, the MIME response for replicated objects got tightened up a
    little. Before, it had some leading and trailing CRLFs which, while
    allowed by RFC 7233, provide no benefit. Now, both replicated and EC
    multipart/byteranges avoid extraneous bytes. This let me re-use the
    Content-Length calculation in swob instead of having to either hack
    around it or add extraneous whitespace to match.
    
    Change-Id: I16fc65e0ec4e356706d327bdb02a3741e36330a0

commit f1f4bb30cd8ad930ddb7a232b2744b48e35a0480
Author: Christian Schwede <cschwede@redhat.com>
Date:   Tue Jun 2 19:51:39 2015 +0000

Fix testing issues
    
    When functional tests are run in tox and an exception is raised when connecting
    to Swift (for example: Swift not running, missing python-keystoneclient package
    used by python-swiftclient) 0 tests are executed, but tox returns a success.
    
    An exception is raised during tests, caused by a missing python-keystoneclient
    in python-swiftclient. Instead of adding python-keystoneclient as a dependency
    in python-swiftclient the package is added to the test-requirements.txt in Swift
    itself. Note that adding python-keystoneclient to the test-requirements in
    python-swiftclient is not sufficient (it's already in there).
    
    The exception in setup_package() is catched by the openstack.nose_plugin, thus
    disabling this plugin for now as well.
    
    Also fixing two test errors seen on the gate regarding the tempurl middleware.
    
    There was also an update to tox, environment variables were no longer passed
    with versions >= 2.0 (http://tox.readthedocs.org/en/latest/changelog.html).
    Swift test environment variables have been added to the passenv to re-enable the
    former behavior, as well as environment variables required to pass proxy
    settings.
    
    This also led to skipped tempauth tests, and together with the missing
    python-keystoneclient no tests were executed.
    
    Related-Bug: 1461440
    Related-Bug: 1455102
    
    Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
    
    Change-Id: Ideea071017d04912c60ed0bc76532adbb446c31d

commit 2759a6f159d4a90e4ad4c8e9e479085d195f8d14
Author: Pradeep Kumar Singh <pradeep.singh@nectechnologies.in>
Date:   Wed May 13 21:18:02 2015 +0530

Handle Disk IO error Exception in swift account auditor
    
    Swift account auditor fails to quarantine corrupt db due to
    Disk IO error. This patch fixes that by handling Disk IO Error
    Exception.
    
    Closes-Bug:1169189
    Change-Id: I031ee2a5775e4a88d4fb00d972d553936147c42e

commit 1bef06eec8f5f780914ac701d63f9c498b29119b
Author: Michael Barton <mike@weirdlooking.com>
Date:   Sun May 31 23:10:15 2015 +0000

Don't quarantine on read_metadata ENOENT
    
    An operation that removes an existing .ts or .meta out from under another
    concurrent operation at the right point can cause the whole object to be
    needlessly quarantined.
    
    Closes-Bug: #1451520
    
    Change-Id: I37d660199e54411d0610889f9ee230b13747244b

commit 8b4af92dac813d707d124b3be37df586260fe74a
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Mon Jun 1 17:57:25 2015 +0100

Tighten up overall metadata size tests
    
    Overall metadata size constraint is enforced in the container and account
    backends as well as in the proxy controllers. Whereas the proxy controller
    can check that constraints are not exceeded by a single PUT or POST request,
    the backend checks that constraints are not exceeded by the aggregate of all
    PUTs and POSTs.
    
    The current functional tests only exercise the proxy controller checks, since
    they test for a 400 when sending excessive metadata in a single POST. This patch
    adds a test for a 400 when a single metadata item in a POST causes the backend
    aggregate constraints check to go over limit.
    
    The extra coverage of the new assertions can be seen by modifying
    swift/common/db.DatabasBroker.validate_metadata() to always return None
    immediately - only the new assertions fail when functests are run.
    
    Change-Id: I1489e29686013cbd3d70283d8756b548aea3c2e1

commit b4c1d73ad56e0705d1c5dd1fc4f1d89b09fceff1
Author: Christian Schwede <cschwede@redhat.com>
Date:   Mon Jun 1 06:50:33 2015 +0000

Make swift-recon compatible for servers without storage policies
    
    Swift recon introduced a new key for storage policies, and the CLI expected this
    key in the server response. However, if one updates the CLI but not yet the
    server an exception will be raised, because there is no default value and no
    check if the key is included in the response.
    
    This change checks if the policies key is included in the response and updates
    one test to ensure backward compability.
    
    Closes-Bug: 1453599
    
    Change-Id: I7c7a90f9933bec2ab45595df9dc600a6cba65666

commit 38787d0fb5102e41b153e4629d2ef374a02965e9
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Tue May 26 16:19:54 2015 -0700

Remove simplejson from staticweb
    
    Since we're dropping Python 2.6 support, we can rely on stdlib's json
    and get rid of our dependency on simplejson.
    
    This lets us get rid of some redundant Unicode encoding. Before, we
    would take the container-listing response off the wire,
    JSON-deserialize it (str -> unicode), then pass each of several fields
    from each entry to get_valid_utf8_str(), which would encode it,
    (unicode -> str), decode it (str -> unicode), and then encode it again
    (unicode -> str) for good measure.
    
    The net effect was that each object's name would, in the proxy server,
    go str -> unicode -> str -> unicode -> str.
    
    By replacing simplejson with stdlib json, we get a guarantee that each
    container-listing entry's name, hash, content_type, and last_modified
    are unicodes, so we can stop worrying about them being valid UTF-8 or
    not. This takes an encode and decode out of the path, so we just have
    str -> unicode -> str. While it'd be ideal to avoid this, the first
    transform (str -> unicode) happens when we decode the
    container-listing response body (json.loads()), so there's no way out.
    
    Change-Id: I00aedf952d691a809c23025b89131ea0f02b6431

commit 736cf54adf3ee85d2f473e5e5374f9833422967c
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu May 28 15:30:47 2015 -0700

Remove simplejson from tests
    
    Since we're dropping Python 2.6 support, we can rely on stdlib's json
    and get rid of our dependency on simplejson.
    
    This commit just takes simplejson out of the unit and functional
    tests. They still pass.
    
    Change-Id: I96f17df81fa5d265395a938b19213d2638682106

commit 68c30b80b47c281c549d101c43d79e718e91d21d
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu May 28 06:08:12 2015 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I0c7b2bdb0edcc3bb3fa06903ec8457ca5e2dd688

commit 5374ba3a80a5b895542196502eac4d9300ba53d2
Author: John Dickinson <me@not.mn>
Date:   Wed May 27 12:28:04 2015 -0700

drop Python 2.6 testing support
    
    Change-Id: I78f21e5794e8ba7a095f03d279247516a241f555

commit 191f2a00bd9121fddc03d8f07f15e5e34790541e
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Mon Apr 27 16:39:23 2015 +0100

Remove _ensure_flush() from SSYNC receiver
    
    The Receiver._ensure_flush() method in ssync_receiver.py has
    the following comment:
    
        Sends a blank line sufficient to flush buffers.
    
        This is to ensure Eventlet versions that don't support
        eventlet.minimum_write_chunk_size will send any previous data
        buffered.
    
        If https://bitbucket.org/eventlet/eventlet/pull-request/37
        ever gets released in an Eventlet version, we should make
        this yield only for versions older than that.
    
    The reference pull request was included with eventlet 0.14 [1] and
    swift now requires >=0.16.1 so it is safe to remove _ensure_flush()
    and save > 8k bytes per SSYNC response.
    
    [1] https://bitbucket.org/eventlet/eventlet/commits/4bd654205a4217970a57a7c4802fed7ff2c8b770
    
    Change-Id: I367e9a6e92b7ea75fe7e5795cded212657de57ed

commit 3aa06f185ac4256a8883c565bdc90b1ffbd519ca
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Mon Apr 27 09:17:46 2015 +0100

Make SSYNC receiver return a reponse when initial checks fail
    
    The ssync Receiver performs some checks on request parameters
    in initialize_request() before starting the exchange of missing
    hashes and updates e.g. the destination device must be available;
    the policy must be valid. Currently if any of these checks fails
    then the receiver just closes the connection, so the Sender gets
    no useful response code and noise is generated in logs by httplib
    and wsgi Exceptions.
    
    This change moves the request parameter checks to the Receiver
    constructor so that the HTTPExceptions raised are actually sent
    as responses. (The 'connection close' exception handling still
    applies once the 'missing_check' and 'updates' handshakes are in
    progress.)
    
    Moving initialize_request() revealed the following lurking bug:
     * initialize_request() sets
           req.environ['eventlet.minimum_write_chunk_size'] = 0
     * this was previously ineffective because the Response environ
       had already been copied from Request environ before this value
       was set, so the Response never used the value :/
     * Now that it is effective (a good thing) it causes the empty string
       yielded by the receiver when there are no missing hashes in
       missing_checks() to be sent to the sender immediately. This makes
       the Sender.readline() think there has been an early disconnect
       and raise an Exception (a bad thing), as revealed by
       test/unit/obj/test_ssync_sender.py:TestSsync.test_nothing_to_sync
    
    The fix for this is to simply make the receiver skip sending the empty
    string if there are no missing object_hashes.
    
    Change-Id: I036a6919fead6e970505dccbb0da7bfbdf8cecc3

commit bb716573ab5c8455348ec013feb894421e0e1f1c
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed May 20 00:39:41 2015 -0700

Allow SLO PUTs to forgo per-segment integrity checks
    
    While manifests still require 'etag' and 'size_bytes' fields for each
    segment (to catch user errors like 'etaf' or 'size_btyes'), an explicit
    null for either will skip that particular integrity check and instead
    use whatever value is retrieved when HEADing the segment. So, if a user
    uploads a manifest like:
    
        [{"path": "/con/obj_seg_1", "etag": null, "size_bytes": 1048576},
         {"path": "/con/obj_seg_2", "etag": "etag2", "size_bytes": null},
         {"path": "/con/obj_seg_3", "etag": null, "size_bytes": null}]
    
    then the etag will only be verified for the /con/obj_seg_2 segment,
    and the segment size will only be verified for the /con/obj_seg_1
    segment. However, the manifest that's ultimately stored (and can be
    retrieved with a ?multipart-manifest=get query-string) will still look
    like:
    
        [{"name": "/con/obj_seg_1", "hash": "etag1", "bytes": 1048576, ...},
         {"name": "/con/obj_seg_2", "hash": "etag2", "bytes": 1048576, ...},
         {"name": "/con/obj_seg_3", "hash": "etag3", "bytes": 1234, ...}]
    
    This allows the middleware to continue performing integrity checks on
    object GET.
    
    Change-Id: I2c4e585221387dd02a8679a50398d6b614407b12
    DocImpact

commit a1c327022c70907ccc159d6203de26b37c3a4586
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Tue May 26 16:43:55 2015 -0700

Remove simplejson from swift-recon
    
    Since we're dropping Python 2.6 support, we can rely on stdlib's json
    and get rid of our dependency on simplejson.
    
    All swift-recon was doing with json was decoding a JSON response (from
    the recon middleware) and printing it to the terminal. This still
    works just fine.
    
    Change-Id: I28cf25a7c2856f230d4642c62fb8bf9c4d37e9e5

commit a3559edc2342e2cf92a5188336ab263ffd038554
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Apr 17 16:30:30 2015 -0700

Exclude local_dev from sync partners on failure
    
    If the primary left or right hand partners are down, the next best thing
    is to validate the rest of the primary nodes.  Where the rest should
    exclude not just the left and right hand partners - but ourself as well.
    
    This fixes a accidental noop when partner node is unavailable and
    another node is missing data.
    
    Validation:
    
    Add probetests to cover ssync failures for the primary sync_to nodes for
    sync jobs.
    
    Drive-by:
    
    Make additional plumbing for the check_mount and check_dir constraints into
    the remaining daemons.
    
    Change-Id: I4d1c047106c242bca85c94b569d98fd59bb255f4

commit 666bf06c26bc9e0d6256d054835386e50e67b8a2
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Wed May 6 16:29:06 2015 -0700

EC: don't 503 on marginally-successful PUT
    
    On EC PUT in an M+K scheme, we require M+1 fragment archives to
    durably land on disk. If we get that, then we go ahead and ask the
    object servers to "commit" the object by writing out .durable
    files. We only require 2 of those.
    
    When we got exactly M+1 fragment archives on disk, and then one
    connection timed out while writing .durable files, we should still be
    okay (provided M is at least 3). However, we'd take our M > 2
    remaining successful responses and pass that off to best_response()
    with a quorum size of M+1, thus getting a 503 even though everything
    worked well enough.
    
    Now we pass 2 to best_response() to avoid that false negative.
    
    There was also a spot where we were getting the quorum size wrong. If
    we wrote out 3 fragment archives for a 2+1 policy, we were only
    requiring 2 successful backend PUTs. That's wrong; the right number is
    3, which is what the policy's .quorum() method says. There was a spot
    where the right number wasn't getting plumbed through, but it is now.
    
    Change-Id: Ic658a199e952558db329268f4d7b4009f47c6d03
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Closes-Bug: 1452468

commit e7c8c578d9e5b0aa7e56b02bd9c39baa99d2d6ae
Author: Michael MATUR <michael.matur@gmail.com>
Date:   Mon May 25 15:13:01 2015 +0200

fixup!Patch of "parse_content_disposition" method to meet RFC2183
    
    The spec of Content-Disposition does not require a space character after
    comma: http://www.ietf.org/rfc/rfc2183.txt
    
    Change-Id: Iff438dc36ce78c6a79bb66ab3d889a8dae7c0e1f
    Closes-Bug: #1458497

commit af8d842076ba269fed7f4128d0c7503ab5d1a94a
Author: Joanna H. Huang <joanna.huitzu.huang@gmail.com>
Date:   Tue Oct 21 09:24:25 2014 +0000

Replaced setting run_pause with standard interval
    
    The deprecated directive `run_pause` should be replaced with the more
    standard one `interval`. The `run_pause` should be still supported for
    backward compatibility. This patch updates object replicator to use
    `interval` and support `run_pause`. It also updates its sample config
    and documentation.
    
    Co-Authored-By: Joanna H. Huang <joanna.huitzu.huang@gmail.com>
    Co-Authored-By: Kamil Rykowski <kamil.rykowski@intel.com>
    
    Change-Id: Ie2a3414a96a94efb9273ff53a80b9d90c74fff09
    Closes-Bug: #1364735

commit f11d92d566757a54ff1e3800ec0bfac098347a68
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Fri May 22 16:58:04 2015 -0700

Add swift-durability-calculator line to docs
    
    This commits add a line (link and small doc) for
    swift-durability-calculator which provides a browser based
    durability calculation tool to docs as an associated project.
    
    Change-Id: I4ea8015f512616dc25072080bef79b8734971ccf

commit f864092455ebcd40b3568633c3524cc5c64d3309
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed May 20 17:50:07 2015 -0700

Add Swift Inspector to assoicated projects
    
    Change-Id: I5b5448674ea455119a51509ab5e7cd11a764b5a7

commit e54781a2aa29808aff654861d8a0aafd24b6620c
Author: Thiago da Silva <thiago@redhat.com>
Date:   Tue May 19 20:27:06 2015 +0000

add object post and delete methods to BaseObjectController
    
    Adding post and delete methods to BaseObjectController that can
    be overriden by ObjectCOntroller subclasses. These methods are
    similar to the PUT and GET methods that were introduced as part
    of the EC work
    
    Change-Id: I197364bc3e2f2287c0afc8948863e3cdeab90383
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit ab9f63402de6d554528699a02955854ac28264c5
Author: Takashi Kajinami <kajinamit@nttdata.co.jp>
Date:   Mon Nov 24 21:44:03 2014 +0900

Add process name checking into swift-init
    
    Swift-init uses pid files to detect existing swift processes by pid.
    However, it mistakes an unrelated process for a swift process and makes
    a wrong decision, when the unrelated process is running with pid written
    in swift pid files.
    This patch adds process name checking into swift-init and enable it to remove
    invalid pid files in such situation.
    
    Change-Id: Ibca026bdfbdacdd92c8763e1eb15d98293c70656
    Closes-Bug: #1327106

commit aa4866eb6014d7fbb8531eb5d927d5611ebfd06d
Author: Thiago da Silva <thiago@redhat.com>
Date:   Wed May 13 20:10:59 2015 +0000

move replication code to ReplicatedObjectController
    
    Moving _connect_put_node, send_file, _transfer_data and _store_object
    methods to ReplicatedObjectController. Each one of these methods are
    specific to replication policy.  EC policy implements their own
    version of them.
    
    Of these four methods, only _store_object and _connect_put_node are
    required to be implemented by the policy specific Object Controllers.
    
    Change-Id: Ifc72461b77dbfdaae9d63417f1286e8b5da3ca4e
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit 025c4c4339e7ace2f5be8cb3a3cddf6c38ceff37
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Fri Apr 24 02:15:36 2015 -0700

Remove confusable query string on post as copy
    
    Current post as copy routine (i.e. POST object with post_as_copy option
    turned on) on Object Controller uses "multipart-manifest" query string
    which is feeded to env['copy_hook'] to decide which data (the manifest or
    object pointed by the manifest) should be copied.
    
    However, the way using the query string will confuse operators looking at
    logging system (or analyzing the log) because whole POST object requests
    have 'multipart-manifest=get' like as:
    
    POST /v1/AUTH_test/d4c816b24d38489082f5118599a67920/manifest-abcde%3Fmultipart-manifest%3Dget
    
    We cannot know whether the query string was added by hand
    (from user) or not. In addition, the query isn't needed by the
    backend conversation between proxy-server and object-server.
    (Just needed by "copy_hook" on the proxy controller!)
    
    To remove the confusable query string and to keep the log to be clean,
    this patch introduces new environment variable "swift.post_as_copy"
    and changes proxy controller and the copy_hook to use the new env.
    
    This item was originally discussed at
    https://review.openstack.org/#/c/177132/
    
    Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
    
    Change-Id: I0cd37520eea1825a10ebd27ccdc7e9162647233e

commit 98b725fec639c5501c645ce4e4dc9d12c686f91d
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Fri May 1 13:02:29 2015 +0100

Cleanup and extend end to end ssync tests
    
    Extends the existing end to end ssync tests with
    a test using replication policy.
    
    Also some cleanup and improvements to the test framework e.g. rather
    than faking the connection between sender and receiver, use a real
    connection and wrap it to capture traffic for verification.
    
    Change-Id: Id71d2eb3fb8fa15c016ef151aacf95f97196a902

commit 518262ab6ecd8faa2b915df118ffc70a30112a18
Author: paul luse <paul.e.luse@intel.com>
Date:   Tue May 12 15:21:13 2015 -0700

Remove 1 line of dead code from EC reconstructor
    
    Assuming nobody intentionally left this in here for some reason...
    
    Change-Id: I4bf43bb3828e062c0342557243076ed62d6790f4

commit 29f4393d88426fd6c34f2cfe43a8c434bfad8d47
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri May 8 15:55:14 2015 -0700

Remove workaround for old eventlet version
    
    Swift now requires eventlet >= 0.16.1, so we can get rid of this
    workaround for a bug in eventlet 0.9.16.
    
    Change-Id: I4a1200b9bd9266896a704a840fda0d1b720bc86d

commit 90b84d3a699811a99c97ebbe4f71a14d2f76a0e5
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri May 8 11:45:12 2015 -0700

Properly re-raise exceptions in proxy_logging
    
    Previously, this could encounter TypeErrors, presumably because
    sys.exc_clear() was called somewhere in the block of code between
    catching the exception and re-raising.
    
    Related-Bug: 1181146
    Change-Id: Iadeea3f61e70bf83dc0eb063fdb27edd16f3ca32

commit 664a632c01f8c5c80826b223b9ade774bfe2ed9a
Author: Christian Schwede <cschwede@redhat.com>
Date:   Fri May 8 08:41:39 2015 +0200

Update my mailmap entry
    
    Change-Id: I5d21a55d0fa4cab6eaa6ff426819aa1dc997de2f

commit 0b20a18e5216b991855c461e41e9ef32e17fadb4
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Thu May 7 13:16:15 2015 -0600

Spell "rebalance" right in swift-ring-builder.1
    
    See Red Hat bug #1218269.
    
    Change-Id: I814eb4b3c0821f5a8df5feea2bda3a964aace536

commit 1faad248f833735585aa8f6135babceb46fbb6f8
Author: Emmanuel Cazenave <contact@emcaz.fr>
Date:   Tue May 5 12:31:22 2015 +0200

X-Auth-Token should be a bytestring.
    
    Change-Id: I2aa941d74883e17e9548b0144a4a2e2db33aba95
    Closes-Bug: 1451773

commit 55dd705a863c4500330cbd2b8c2fec46d618dc71
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Wed May 6 19:53:09 2015 +0200

Add missing statsd metrics section for object-reconstructor
    
    Change-Id: Id3f98e5f637ff537a387262b40f21c05876fca91

commit c77c79b2c9b9ca7790c29577341cf36c5e9012cf
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Tue May 5 06:08:09 2015 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I9b0156c7fc315182d80604bf353586455bbc34d1

commit 94215049fd37f810ddf4e6b8122cce02aea3e6e3
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon May 4 15:08:51 2015 -0700

Bump up a timeout in a test
    
    Got a slow crappy VM like I do? You might see this fail
    occasionally. Bump up the timeout a little to help it out.
    
    Change-Id: I8c0e5b99012830ea3525fa55b0811268db3da2a2

commit 03536dbb55c219b94389e635babec2df2e2759fa
Author: Prashanth Pai <ppai@redhat.com>
Date:   Wed Apr 29 12:11:59 2015 +0530

Fix incorrect passing of file object to fsync()
    
    swift.common.utils.fsync() requires a file descriptor as argument
    but file object handle was being passed.
    
    Change-Id: I316b58f6bc37de0945eff551e4e50565653664f5
    Signed-off-by: Prashanth Pai <ppai@redhat.com>

commit bfbc94c3cb34eb9ff288fb817dee667cc870d9eb
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Tue Apr 28 10:45:50 2015 +0100

Fix intermittent container replicator test failure
    
    Intermittent failure of this test could be due to
    insufficient time elapsing between either the local
    and remote db's being created or during the
    debug_timing calls. This patch forces greater timestamp
    separation and forces debug_timing to always log timings.
    
    Also add message to the failign assertion so if this does
    fail again we get some clue as to why.
    
    Closes-Bug: 1369663
    
    Change-Id: I4b69b2e759d586a14abd0931a68dbdf256d57c32

commit dbb9d4b7938f24ee588d260c9b51ca65d5095749
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Mon Apr 27 08:52:18 2015 +0200

Add missing docstring in direct_client
    
    Added a missing docstring to the direct_delete_container method.
    
    Also checked other docstrings in the same file and fixed a wrong docstring
    element in direct_delete_object. Added raises: docstring entry to all methods
    that raise an Exception.
    
    Change-Id: If463a0f9ddff3fe2d13f6d97fcfa955e91d0f01f

commit 8cdf0fdebe9eb782322fccfc11253dc959cf321d
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Mon Apr 27 13:29:50 2015 -0700

Fix account replication during pre-storage-policy upgrade
    
    Old account schemas don't send the storage_policy_index key for container rows
    during replication, and if the recieving end is already running an upgraded
    server it is surprised with a KeyError.  Normally this would work itself out
    if the old schema recieved any updates from container layer, or a new
    container is created, or requires a row sync from another account database -
    but if the account databases have rows out of sync and there's no activity in
    the account otherwise, there's nothing to force the old schemas to be
    upgraded.
    
    Rather than force the old schema that already has a complete set of container
    rows to migrate even in the absense of activity we can just fill in default
    legacy value for the storage policy index and allow the accounts to get back
    in sync and migrate the next time a container update occurs.
    
    FWIW, I never able to get a cluster upgrade to get stuck in this state without
    some sort of account failure that forced them to get their rows out of sync
    (in my cause I just unlinked a pending and then made sure to force all my
    account datbases to commit pending files before upgrading - leading to an
    upgraded cluster that absolutly needed account-replication to solve a row
    mismatch for inactive accounts with old schemas)
    
    Closes-Bug #1424108
    
    Change-Id: Iaf4ef834eb24f0e11a52cc22b93a864574fabf83

commit 03380380efb5d5c0b2a3acfaa64b486ee2cb6e64
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu Apr 23 19:39:16 2015 -0700

Simplify ring.builder.RingBuilder.__deepcopy__
    
    Only container classes (lists, sets, dicts, graphs, collections,
    etc) need to track objects they deepcopy in the memo dict -
    particularly when they may contain other containers!  As they
    recreate a new container with the same items as themselves, they'll
    reference the memo for each item they contain before making a
    deepcopy of it, and place a reference to the copied item into memo
    after they do.  Trying to help out some other container class in
    this endeavor by attempting to add ourselves to the memo dict in
    some useful manor on their behalf however; is not helpful.
    
    All we need to do to make sure we're being a good __deepcopy__
    implementation is make sure we pass on memo to our calls of deepcopy
    so that other container classes can avoid making additional
    deepcopy's of our containers if they already have a memorized copy
    (which would be odd since unique instances of RingBuilders aren't
    expected to share state, but hey - python doesn't have private
    attributes so you never know!)
    
    Change-Id: Ifac444dffbf79d650b2d858f6282e05d8ea741a0

commit 43ace3c62893364b6e3c130df56438995627598d
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon Apr 20 12:17:56 2015 -0700

Make RingBuilders deep-copy-able
    
    We used to be able to deep-copy RingBuilder objects, but the addition
    of debug logging (8d3b3b2) broke that since you can't deep-copy a
    Python logger. This commit fixes that.
    
    Swift doesn't really deep-copy RingBuilders anywhere, but third-party
    code might.
    
    Change-Id: If8bdadd93d9980db3d8a093f32d76ca604de9301

commit 215cd551df8be066edafd2a1e16d0bd143ec214b
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Tue Apr 21 17:38:04 2015 -0700

Bulk upload: treat user xattrs as object metadata
    
    Currently, if you PUT a single object, then you can also associate
    metadata with it by putting it in the request headers, prefixed with
    "X-Object-Meta". However, if you're bulk-uploading objects, then you
    have no way to assign any metadata.
    
    The tar file format* allows for arbitrary UTF-8 key/value pairs to be
    associated with each file in an archive (as well as with the archive
    itself, but we don't care about that here). If a file has extended
    attributes, then tar will store those as key/value pairs.
    
    This commit makes bulk upload read those extended attributes, if
    present, and convert those to Swift object metadata. Attributes
    starting with "user.meta" are converted to object metadata, and
    "user.mime_type"** is converted to Content-Type.
    
    For example, if you have a file "setup.py":
    
        $ setfattr -n user.mime_type -v "application/python-setup" setup.py
        $ setfattr -n user.meta.lunch -v "burger and fries" setup.py
        $ setfattr -n user.meta.dinner -v "baked ziti" setup.py
        $ setfattr -n user.stuff -v "whee" setup.py
    
    This will get translated to headers:
    
        Content-Type: application/python-setup
        X-Object-Meta-Lunch: burger and fries
        X-Object-Meta-Dinner: baked ziti
    
    Swift will handle xattrs stored by both GNU and BSD tar***. Only
    xattrs user.mime_type and user.meta.* are processed; others are
    ignored.
    
    This brings bulk upload much closer to feature-parity with non-bulk upload.
    
    * The POSIX 1003.1-2001 (pax) format, at least. There are a few
      different, mutually-incompatible tar formats out there, because of
      course there are. This is the default format on GNU tar 1.27.1 or
      later.
    
    ** http://standards.freedesktop.org/shared-mime-info-spec/latest/ar01s02.html#idm140622087713936
    
    *** Even with pax-format tarballs, different encoders store xattrs
        slightly differently; for example, GNU tar stores the xattr
        "user.rubberducky" as pax header "SCHILY.xattr.user.rubberducky",
        while BSD tar (which uses libarchive) stores it as
        "LIBARCHIVE.xattr.user.rubberducky". One might wonder if this is
        some programmer's attempt at job security.
    
    Change-Id: I5e3ce87d31054f5239e86d47c45adbde2bb93640

commit f8dee761bd36f857aa1288c27e095907032fad68
Author: Andreas Jaeger <aj@suse.de>
Date:   Mon Apr 20 11:15:35 2015 +0200

Release Import of Translations from Transifex
    
    Manual import of Translations from Transifex. This change also removes
    all po files that are less than 66 per cent translated since such
    partially translated files will not help users.
    
    This updates also recreates all pot (translation source files) to
    reflect the state of the repository.
    
    This change needs to be done manually since the automatic import does
    not handle the proposed branches and we need to sync with latest
    translations.
    
    Note: This is part of importing of translations, there are no new
    translations for this project, thus only the pot file gets updated.
    
    Change-Id: I0cbfdae3bd1662da54c58e91a13f49419eba9b2d

commit cd7c58e93690fd25f5266754d0593c656dd51e2e
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Mon Apr 20 00:18:25 2015 -0700

Use reconstruct insetad of decode/encode
    
    With bumping PyECLib up to 1.0.7 on global requirements,
    we can use the "reconstruct" function directly instead
    of the current hack doing decode/encode on reconstructor.
    That is because the hack was for treating PyECLib < 1.0.7
    (strictly jearsure scheme) reconstruction bug so we don't
    have to do decode/encode anymore.
    
    Closes-Bug: 1446801
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Change-Id: I69aae495670e3d0bdebe665f73915547a4d56f99

commit 281cb1c210f0292bad190cabaae447145fa5eade
Author: Tushar Gohad <tushar.gohad@intel.com>
Date:   Wed Apr 15 17:34:48 2015 -0700

Bump PyECLib version from 1.0.3 to 1.0.7
    
    In addition to fixing several bugs, 1.0.7 eliminates the need for a
    few work-around code in Swift.  This code was only to hide issues in
    the current version, but it also ends up breaking some third-party
    integration.  In order to enable expected functionality and to avoid
    dealing with deprecation issues right from the beginning, we need to
    bump the minium PyECLib requirement to 1.0.7.
    
    Closes-Bug: 1446727
    Change-Id: I03e059e7335656c22be28ffd6157b56e13bdfc1b

commit 51e31c5c7147f3ba61437e132b12b491ca718ce4
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed Apr 15 15:31:06 2015 -0700

Don't apply the wrong Etag validation to rebuilt fragments
    
    Because of the object-server's interaction with ssync sender's
    X-Backend-Replication-Headers when a object (or fragment archive) is
    pushed unmodified to another node it's ETag value is duped into the
    recieving ends metadata as Etag.  This interacts poorly with the
    reconstructor's RebuildingECDiskFileStream which can not know ahead of
    time the ETag of the fragment archive being rebuilt.
    
    Don't send the Etag from the local source fragment archive being used as
    the basis for the rebuilt fragent archive's metadata along to ssync.
    
    Closes-Bug: 1446800
    Change-Id: Ie59ad93a67a7f439c9a84cd9cff31540f97f334a

commit 2080f7dbd897d6130542dbf88e37641a41625eb5
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Thu Feb 26 15:16:22 2015 +0000

Fix tempauth acl checks when simplejson has no speedups
    
    As documented in linked bug report, tempauth unit tests
    were seen to fail on a system where simplejson was
    installed but without the speedups extension. This
    is because the tempauth account acl validation checks
    that values are type str, but without the speedups
    extension the json parser is returning unicode objects.
    
    Fix is to have the acl validator tolerate those objects
    being unicode or str.
    
    Also change common/bufferedhttp.py to coerce ring device
    to type str when constructing a path, in order to avoid
    a UnicodeDecodeError when httplib sends a message that
    has non-ascii header values.
    
    Change-Id: I01524282cbaa25dc4b6dfa09f3f4723516cdba99
    Closes-Bug: 1425776

commit 27f6fba5c3b9e0461d52c1deffe32130e7e9af51
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Mon Apr 20 00:18:25 2015 -0700

Use reconstruct insetad of decode/encode
    
    With bumping PyECLib up to 1.0.7 on global requirements,
    we can use the "reconstruct" function directly instead
    of the current hack doing decode/encode on reconstructor.
    That is because the hack was for treating PyECLib < 1.0.7
    (strictly jearsure scheme) reconstruction bug so we don't
    have to do decode/encode anymore.
    
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Change-Id: I69aae495670e3d0bdebe665f73915547a4d56f99

commit 0c391d6daffe1943f6def803db42e08e6d6846d2
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon Apr 20 16:47:10 2015 -0700

SAIO instructions: ensure ~/bin exists before copying into it
    
    Change-Id: I16cd211b00b529ccc4b46f6b10497c32b6741896

commit c5c281ba6dd97a301ba80511e3356d6ca536d701
Author: Minwoo B <minwoob@us.ibm.com>
Date:   Mon Apr 20 17:03:25 2015 -0500

Included step in development_saio.rst for installing dependencies in requirements.txt.
    
    Change-Id: I6ed1704148e5ae1e3164d10080c350d81856f7a9

commit 2203b46e3f14ef68a090aaea284f0a0442bbb86f
Author: Tushar Gohad <tushar.gohad@intel.com>
Date:   Wed Apr 15 17:34:48 2015 -0700

Bump PyECLib version from 1.0.3 to 1.0.7
    
    In addition to fixing several bugs, 1.0.7 eliminates the need for a
    few work-around code in Swift.  This code was only to hide issues in
    the current version, but it also ends up breaking some third-party
    integration.  In order to enable expected functionality and to avoid
    dealing with deprecation issues right from the beginning, we need to
    bump the minium PyECLib requirement to 1.0.7.
    
    Change-Id: I03e059e7335656c22be28ffd6157b56e13bdfc1b

commit f6482bdece27144ee083a53d696469528d7940c2
Author: Thierry Carrez <thierry@openstack.org>
Date:   Thu Apr 16 22:08:47 2015 +0200

Set default branch to stable/kilo
    
    Open stable/kilo branch by setting defaultbranch for git-review.
    
    Change-Id: I81bcda30f99173416eaaa3f1d42da32f3ab5b6d2

commit a2a5b6aa6664b61fa31d25f329426ff089372f17
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Apr 16 11:42:12 2015 -0700

Functional test for SLO PUT overwriting one of its own segments
    
    Change-Id: I4855816848f4fdb148d0b82735cf79bc68429617

commit e4d326b5a7dd186d762726faa45733ff2900343d
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Thu Feb 19 17:38:10 2015 +0900

Fix conflict SLO reponse
    
    This patch fixes Swift to respond "409 Conflict"
    when a segment object path of the manifest on PUT SLO
    is same as requested object path. It is because
    the request will overwrite the segment and then it
    will absolutely cause "409 Conflict" on GET SLO.
    
    e.g.:
    
    request:
    PUT "http://hostname/v1/AUTH_account/container/segment_object_00?multipart-manifest=put"
    
    manifest file:
    [{"path" : "container/segment_object_00", "etag" : "<etag of segment_object_00>", "size_bytes" : <size of segment_object_00>},
    {"path" : "container/segment_object_01", "etag" : "<etag of segment_object_01>", "size_bytes" : <size of segment_object_01>},
    {"path" : "container/segment_object_02", "etag" : "<etag of segment_object_02>", "size_bytes" : <size of segment_object_02>}]
    
    Change-Id: I4f4f7b9dbeb6a7c355b801c7e0ae560aa19a70b4
    Closes-Bug: 1417936

commit 38ae7bb89c95292c171cb89c1d62859cee0cc24f
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Wed Apr 15 23:03:21 2015 +0100

Make the reaper use same timestamp for replica deletes
    
    The account reaper is using a unique timestamp when deleting
    replicas of the same resource. This will result in unnecessary
    replication traffic after reaping.
    
    This patch makes the reaper use a single timestamp per resource.
    
    Probe test is modified to check that delete times are equal
    across replicas before replicators run.
    
    test_direct_client.py is modified to check that it uses explicit
    timestamp when passed to direct_delete_[object|container]
    methods.
    
    Drive-by bug fixes in the probe test e.g. it was not sending
    X-Backend-Storage-Policy-Index when doing a direct GET to check
    object state, so the 404s being verified could in fact be due
    to diskfile not existing rather than diskfile being deleted.
    
    Closes-Bug: 1442879
    Change-Id: I8bab22d66308bb9d3294e1e0def017c784228423

commit 9c33bbde6923b26f111572ae967a3b97a8ab12f2
Author: Prashanth Pai <ppai@redhat.com>
Date:   Tue Jan 20 12:14:32 2015 +0530

Allow rsync to use compression
    
    From rsync's man page:
    -z, --compress
    With this option, rsync compresses the file data as it is sent to the
    destination machine, which reduces the amount of data being transmitted --
    something that is useful over a slow connection.
    
    A configurable option has been added to allow rsync to compress, but only
    if the remote node is in a different region than the local one.
    
    NOTE: Objects that are already compressed (for example: .tar.gz, .mp3)
    might slow down the syncing process.
    
    On wire compression can also be extended to ssync later in a different
    change if required. In case of ssync, we could explore faster
    compression libraries like lz4. rsync uses zlib which is slow but offers
    higher compression ratio.
    
    Change-Id: Ic9b9cbff9b5e68bef8257b522cc352fc3544db3c
    Signed-off-by: Prashanth Pai <ppai@redhat.com>

tags:

added: in-feature-hummingbird

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-24: Fix proposed to swift (feature/crypto)

#10

Fix proposed to branch: feature/crypto
Review: https://review.openstack.org/205579

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-24: Fix merged to swift (feature/crypto)

#11

Download full text (80.5 KiB)

Reviewed: https://review.openstack.org/205579
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=8ab46b64365b8eab80680f2562f81e8adb3032a3
Submitter: Jenkins
Branch: feature/crypto

commit 89f705e8aab144092d40a13fc4ef19ffef5f3eba
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Jul 23 06:11:27 2015 +0000

Imported Translations from Transifex

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I94cf347564cb33977f33b1e64259bcb39a8cf809

commit a65e9db8752793ec37b594dc9eca5066171825db
Author: Christian Schwede <email address hidden>
Date: Wed Jul 22 10:43:17 2015 +0000

Removing commented out code in test/unit/account/test_backend.py

    Noticed this while reviewing another change. Looks like the test itself already
    ensures correct functionality of the reclaim() method in AccountBroker without
    the commented code, thus removing this stale code.

Change-Id: I6a26a7591adef9fd794ca68a4e9c493d1127f93c

commit 99d052772a9585e0befdfd292fd03aefde77180a
Author: Kota Tsuyuzaki <email address hidden>
Date: Mon Jul 13 01:12:43 2015 -0700

Fix 499 client disconnected on COPY EC object

    Currently, a COPY request for an EC object might go to fail as 499 Client
    disconnected because of the difference between destination request content
    length and actual transferred bytes.

    That is because the conditional response status and content length for
    an EC object range GET is handled at calling the response instance on
    proxy server. Therefore the calling response instance (resp()) will change
    the conditional status from 200 (HTTP_OK) to 206 (PartialContent) and will
    change the content length for the range GET.

    In EC case, sometimes Swift needs whole stored contents to decode a segment.
    It will make 200 HTTP OK response from object-server and proxy-server
    will unfortunately set whole content length to the destination content
    length and it makes the bug 1467677.

    This patch introduces a new method "fix_conditional_response" for
    swift.common.swob.Response that calling _response_iter() and cached the
    iter in the Response instance. By calling it, Swift can set correct condtional
    response any time after setting whole content_length to the response
    instance like EC case.

Change-Id: If85826243f955d2f03c6ad395215c73daab509b1
Closes-Bug: #1467677

commit 62ed4f81ef80440550633eaaaa962a4f9383c2d3
Author: Timur Alperovich <email address hidden>
Date: Tue Jul 14 16:56:44 2015 -0700

Add two functional tests for delimiter.

    The first test verifies that a delimiter will trim entries beyond the
    first matching instance of delimiter (after the given matching prefix,
    if any) and squash duplicates. So, when setting the delimiter
    to "-", given blobs "test", "test-foo" and "test-bar-baz", we expect
    only "test" (no matching delim) and "test-" (trim all characters after
    the first "-", and squash duplicates).

The second test verifies that when a prefix is provid...

Reviewed:  https://review.openstack.org/205579
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=8ab46b64365b8eab80680f2562f81e8adb3032a3
Submitter: Jenkins
Branch:    feature/crypto

commit 89f705e8aab144092d40a13fc4ef19ffef5f3eba
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu Jul 23 06:11:27 2015 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I94cf347564cb33977f33b1e64259bcb39a8cf809

commit a65e9db8752793ec37b594dc9eca5066171825db
Author: Christian Schwede <cschwede@redhat.com>
Date:   Wed Jul 22 10:43:17 2015 +0000

Removing commented out code in test/unit/account/test_backend.py
    
    Noticed this while reviewing another change. Looks like the test itself already
    ensures correct functionality of the reclaim() method in AccountBroker without
    the commented code, thus removing this stale code.
    
    Change-Id: I6a26a7591adef9fd794ca68a4e9c493d1127f93c

commit 99d052772a9585e0befdfd292fd03aefde77180a
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Mon Jul 13 01:12:43 2015 -0700

Fix 499 client disconnected on COPY EC object
    
    Currently, a COPY request for an EC object might go to fail as 499 Client
    disconnected because of the difference between destination request content
    length and actual transferred bytes.
    
    That is because the conditional response status and content length for
    an EC object range GET is handled at calling the response instance on
    proxy server. Therefore the calling response instance (resp()) will change
    the conditional status from 200 (HTTP_OK) to 206 (PartialContent) and will
    change the content length for the range GET.
    
    In EC case, sometimes Swift needs whole stored contents to decode a segment.
    It will make 200 HTTP OK response from object-server and proxy-server
    will unfortunately set whole content length to the destination content
    length and it makes the bug 1467677.
    
    This patch introduces a new method "fix_conditional_response" for
    swift.common.swob.Response that calling _response_iter() and cached the
    iter in the Response instance. By calling it, Swift can set correct condtional
    response any time after setting whole content_length to the response
    instance like EC case.
    
    Change-Id: If85826243f955d2f03c6ad395215c73daab509b1
    Closes-Bug: #1467677

commit 62ed4f81ef80440550633eaaaa962a4f9383c2d3
Author: Timur Alperovich <timur.alperovich@gmail.com>
Date:   Tue Jul 14 16:56:44 2015 -0700

Add two functional tests for delimiter.
    
    The first test verifies that a delimiter will trim entries beyond the
    first matching instance of delimiter (after the given matching prefix,
    if any) and squash duplicates.  So, when setting the delimiter
    to "-", given blobs "test", "test-foo" and "test-bar-baz", we expect
    only "test" (no matching delim) and "test-" (trim all characters after
    the first "-", and squash duplicates).
    
    The second test verifies that when a prefix is provided, the delimiter
    will trim entries beyond the first matching instance of the delimiter
    *after the given prefix*.  So "bar, "bazar" which both match the
    prefix "ba" will be returned as "bar" (no delimiter after the matching
    prefix) and "baza" (after matching the prefix the remainder after the
    the *next* matching delimiter "a" is trimmed).
    
    Change-Id: I49a2aa8722f83e87b7d211e5c26827e93963d92a

commit 2d6c5fe2afde382edb8f04c91fca1911601b61d9
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Jul 21 09:56:13 2015 -0700

Fix Associated Projects link on Getting Started
    
    Change-Id: I50be656bc5868242489f97c96a1feaef171933e0

commit cd7b2db550e41412281da0256b468c90ebcf3e8b
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Tue Jul 21 19:23:00 2015 +0530

unit tests: Replace "self.assert_" by "self.assertTrue"
    
    The assert_() method is deprecated and can be safely replaced by assertTrue().
    This patch makes sure that running the tests does not create undesired
    warnings.
    
    Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71

commit c3cc98b2c918476f1b54924e0c54e4c746eba4ac
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon Jul 20 17:03:20 2015 -0700

Add comment about ResumingGetter.used_source_etag
    
    This confused a couple developers and took about ten minutes to
    unravel in IRC; let's leave a clue for the next person.
    
    Change-Id: I356c8c7a44de23f02eaf68d23a39c9eb4c203ff1

commit 06326d74657a8796439988115002298e4a5f4630
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Mon Jul 20 21:41:58 2015 +0530

Metaclass Python 3.x Compatibility
    
    use "six.add_metaclass" instead of "__metaclass__"
    
    Change-Id: I57390fec40afe3a965be3215db8f79b9cf862a70

commit 317fa1af49d72ecaf7a9a499d8cdf00a086b4c6f
Author: Azhagu Selvan SP <tamizhgeek@gmail.com>
Date:   Sat Jul 18 14:37:16 2015 +0530

Minor change in development saio setup doc
    
    Make the test-requirements install command uniform with the rest of the
    setup commands in the page by using the $HOME, instead of hardcoding
    the folder path.
    
    Change-Id: I8da7a191a301601e27f894f8268238d77b150ede

commit faff98efd47fd59e28b1adb3e7ad4bc3dc6dd781
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Thu Jul 9 07:35:47 2015 +0530

Replace python print operator with print function (pep H233, py33)
    
    'print' function is compatible with 2.x and 3.x python versions
    Link : https://www.python.org/dev/peps/pep-3105/
    
    Python 2.6 has a __future__ import that removes print as language syntax,
    letting you use the functional form instead
    
    Change-Id: I9d9ca5766a2773a7b4071e280f00361d16b85383

commit 8753e452b04b0af6d6b444184b628724e14fbbb1
Author: Victor Stinner <vstinner@redhat.com>
Date:   Wed May 27 18:01:37 2015 +0200

Replace StringIO with BytesIO for WSGI input
    
    wsgi.input is a binary stream (bytes), not a text stream (unicode).
    
    * Replace StringIO with BytesIO for WSGI input
    * Replace StringIO('') with StringIO() and replace WsgiStringIO('') with
      WsgiStringIO(): an empty string is already the default value
    
    Change-Id: I09c9527be2265a6847189aeeb74a17261ddc781a

commit 6e70f3fa322da70c2a65fb919d200e52fc5add8a
Author: Victor Stinner <vstinner@redhat.com>
Date:   Wed May 27 17:27:47 2015 +0200

Get StringIO and cStringIO from six.moves
    
    * replace "from cStringIO import StringIO"
      with "from six.moves import cStringIO as StringIO"
    * replace "from StringIO import StringIO"
      with "from six import StringIO"
    * replace "import cStringIO" and "cStringIO.StringIO()"
      with "from six import moves" and "moves.cStringIO()"
    * replace "import StringIO" and "StringIO.StringIO()"
      with "import six" and "six.StringIO()"
    
    This patch was generated by the stringio operation of the sixer tool:
    https://pypi.python.org/pypi/sixer
    
    Change-Id: Iacba77fec3045f96773d1090c0bd48613729a561