OpenStack Object Storage (swift)

Container-sync doesn't timeout when putting/deleting object

Bug #1419916 reported by Takashi Kajinami on 2015-02-09

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Object Storage (swift)	Fix Released	Undecided	Christian Schwede

Bug Description

Container-sync has no timeouts when putting/deleting objects, and a physical down or disconnection of proxy servers in the other swift cluster makes container-sync stuck.

CVE References

2015-1856

Christian Schwede (cschwede) on 2015-02-18

Changed in swift:
status:	New → Confirmed
assignee:	nobody → Christian Schwede (cschwede)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-18: Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.openstack.org/156943

Changed in swift:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-13: Fix merged to swift (master)

Reviewed: https://review.openstack.org/156943
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=16ee994c1ea4ddcfaeae26babada6382daa69bc9
Submitter: Jenkins
Branch: master

commit 16ee994c1ea4ddcfaeae26babada6382daa69bc9
Author: Christian Schwede <email address hidden>
Date: Wed Feb 18 10:17:23 2015 +0000

Set connection timeout in container sync

Container sync might get stuck without a connection timeout if the remote proxy
is not responding.

    This patch sets a default timeout of 5.0 seconds for the connection attempt. The
    value is much higher than other connection timeouts inside Swift (0.5); however
    there might be a much higher latency to the remote peer, thus playing it safe.
    There is also a retry if the attempt timed out.

Note that this setting only applies to the connection request itself. Setting
this timeout does not apply when the remote proxy goes away during a request.

Also added a short test to ensure urlopen is called with the timeout value.

Co-Authored-By: Alistair Coles <email address hidden>

Change-Id: Ic08a55157fa91fe1316653781adf4d66eead61bc
Partial-Bug: 1419916

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-21: Fix proposed to swift (feature/crypto)

Fix proposed to branch: feature/crypto
Review: https://review.openstack.org/175866

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-21: Fix merged to swift (feature/crypto)

Download full text (39.4 KiB)

Reviewed: https://review.openstack.org/175866
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=5bb7c286ebb4a54e4d2bd5a02845644d1c651183
Submitter: Jenkins
Branch: feature/crypto

commit e440d6aed8a40848584767ed36811bf09c738838
Author: Kota Tsuyuzaki <email address hidden>
Date: Wed Apr 15 11:25:13 2015 -0700

Fix best response to return correct status

    Current best response could return "503 Internal Server Error".
    However, "503" means "Service Unavailable". (The status int of
    Internal Server Error is 500)

This patch fixes the response status as "503 Service Unavailable"

Change-Id: I88b8c52c26b19e9e76ba3375f1e16ced555ed54c

commit 57011d5699d49a47ae89073ff27b39140ab4d652
Author: Ricardo Ferreira <email address hidden>
Date: Thu Mar 12 23:18:33 2015 +0000

More user-friendly output for object metadata

    Split out system, user and other metadata in swift-object-info. Print
    every position line by line instead of raw dict representation, so it
    would be easier to parse with tools such as grep.

Co-Authored-By: Ricardo Ferreira <email address hidden>
Co-Authored-By: Kamil Rykowski <email address hidden>

Change-Id: Ia78da518c18f7e26016700aee87efb534fbd2040
Closes-Bug: #1428866

commit a162c2bdd7be12daa29dd07230f84efcaf1cab37
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Apr 16 06:06:35 2015 +0000

Imported Translations from Transifex

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I48ba06f4801ff2d7856d67e74d2b1f76c550fcf4

commit 52b102163e48dc04a6a492a3430efa1f7778d7ec
Author: Clay Gerrard <email address hidden>
Date: Wed Apr 15 15:31:06 2015 -0700

Don't apply the wrong Etag validation to rebuilt fragments

    Because of the object-server's interaction with ssync sender's
    X-Backend-Replication-Headers when a object (or fragment archive) is
    pushed unmodified to another node it's ETag value is duped into the
    recieving ends metadata as Etag. This interacts poorly with the
    reconstructor's RebuildingECDiskFileStream which can not know ahead of
    time the ETag of the fragment archive being rebuilt.

Don't send the Etag from the local source fragment archive being used as
the basis for the rebuilt fragent archive's metadata along to ssync.

Change-Id: Ie59ad93a67a7f439c9a84cd9cff31540f97f334a

commit 46bd6716ffae28aef53f15af170fd2df01b49843
Author: Kota Tsuyuzaki <email address hidden>
Date: Tue Apr 14 23:22:14 2015 -0700

Small minor refactor on ec diskfile

To be more helpful, add an inline comment and remove
unnecessary assignment.

Change-Id: Ia9c6993dfa03c238736955de8b0f5c1a7d5d1b65

commit 193fe9a5f906a2344bb5d328ad55b881e4086caa
Author: Lorcan <email address hidden>
Date: Wed Apr 15 11:32:32 2015 +0100

Update swift-recon doc with more options

The swit-recon tool has had several functional additions
added recently but not all of these have been added to the docs.

This change add...

Reviewed:  https://review.openstack.org/175866
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=5bb7c286ebb4a54e4d2bd5a02845644d1c651183
Submitter: Jenkins
Branch:    feature/crypto

commit e440d6aed8a40848584767ed36811bf09c738838
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Wed Apr 15 11:25:13 2015 -0700

Fix best response to return correct status
    
    Current best response could return "503 Internal Server Error".
    However, "503" means "Service Unavailable". (The status int of
    Internal Server Error is 500)
    
    This patch fixes the response status as "503 Service Unavailable"
    
    Change-Id: I88b8c52c26b19e9e76ba3375f1e16ced555ed54c

commit 57011d5699d49a47ae89073ff27b39140ab4d652
Author: Ricardo Ferreira <ricardo.sff@gmail.com>
Date:   Thu Mar 12 23:18:33 2015 +0000

More user-friendly output for object metadata
    
    Split out system, user and other metadata in swift-object-info. Print
    every position line by line instead of raw dict representation, so it
    would be easier to parse with tools such as grep.
    
    Co-Authored-By: Ricardo Ferreira <ricardo.sff@gmail.com>
    Co-Authored-By: Kamil Rykowski <kamil.rykowski@intel.com>
    
    Change-Id: Ia78da518c18f7e26016700aee87efb534fbd2040
    Closes-Bug: #1428866

commit a162c2bdd7be12daa29dd07230f84efcaf1cab37
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu Apr 16 06:06:35 2015 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I48ba06f4801ff2d7856d67e74d2b1f76c550fcf4

commit 52b102163e48dc04a6a492a3430efa1f7778d7ec
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed Apr 15 15:31:06 2015 -0700

Don't apply the wrong Etag validation to rebuilt fragments
    
    Because of the object-server's interaction with ssync sender's
    X-Backend-Replication-Headers when a object (or fragment archive) is
    pushed unmodified to another node it's ETag value is duped into the
    recieving ends metadata as Etag.  This interacts poorly with the
    reconstructor's RebuildingECDiskFileStream which can not know ahead of
    time the ETag of the fragment archive being rebuilt.
    
    Don't send the Etag from the local source fragment archive being used as
    the basis for the rebuilt fragent archive's metadata along to ssync.
    
    Change-Id: Ie59ad93a67a7f439c9a84cd9cff31540f97f334a

commit 46bd6716ffae28aef53f15af170fd2df01b49843
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Tue Apr 14 23:22:14 2015 -0700

Small minor refactor on ec diskfile
    
    To be more helpful, add an inline comment and remove
    unnecessary assignment.
    
    Change-Id: Ia9c6993dfa03c238736955de8b0f5c1a7d5d1b65

commit 193fe9a5f906a2344bb5d328ad55b881e4086caa
Author: Lorcan <lorcan.browne@hp.com>
Date:   Wed Apr 15 11:32:32 2015 +0100

Update swift-recon doc with more options
    
    The swit-recon tool has had several functional additions
    added recently but not all of these have been added to the docs.
    
    This change adds the following options to the manpages:
    
      --human-readable
      --validate-servers
      --sockstat
      --driveaudit
      --region
      --timeout
    
    Also fixes a typo on line 78 (cop -> copy)
    
    Change-Id: Id083b32a60473ad5a2b9ac1d092528d230521c86

commit 663ccd4e7ad50f67cd2f85b4e7cfe1de6ecbbce6
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Tue Apr 14 10:52:36 2015 +0000

More tests for swift recon
    
    Change-Id: I8d568c0f6fbe1c01d97491740aebf299deb63732

commit c8e943c916c6e2eb6212508bad367e5dc1c64c5c
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Wed Apr 15 06:07:23 2015 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I15b1d475d157bfb74ff7a68400cf06596488aa93

commit 5ef2e9ec00b1498fb411fc918b71ae9c50b1308f
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Apr 14 15:24:58 2015 -0700

fixup random test failure
    
    Change-Id: I053cec441623e7556d6cdfbc1d271c5cdf21aa8b

commit 86920f3dbe43a0389119b7d26b61650997c0fc9f
Author: John Dickinson <me@not.mn>
Date:   Mon Mar 23 12:00:28 2015 -0700

2.3.0 authors and changelog updates
    
    Change-Id: I5a46437a718e2a29fd289bde7bec45db4ee873bd

commit 4aba2fbb25edf8936e00ee9f5736cc2c0c383c32
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Mar 27 15:50:38 2015 -0700

Check if REST API version is valid
    
    Swift doesn't check if the used API version is valid. Currently there
    is only one valid REST API version, but that might change in the
    future.
    
    This patch enforces "v1" or "v1.0" as the version string when accessing
    account, containers and objects.
    
    The list of accepted version strings can be manually overridden using a
    comma-separated list in swift.conf to make this backward-compatible.
    The constraint loader has been modified slightly to accept strings as
    well as integers.
    
    Any request to an account, container, and object which does not
    provide the correct version string will get a 400 BadRequest response.
    
    The allowed api versions are by default excluded from /info.
    
    Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
    Co-Authored-By: John Dickinson <me@not.mn>
    
    Closes Bug #1437442
    
    Change-Id: I5ab6e236544378abf2eab562ab759513d09bc256

commit e16df14a734757fabea5dbc13851050e52c4b4da
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Sun Apr 12 22:35:56 2015 +0200

Add test for swift_recon.disk_usage
    
    Change-Id: I4cab7aa6df3f0e1933e52ee5dbbb829f30604f10

commit 66fb207fcb288889d73d42960b037e3ef7ef3ce2
Author: Lorcan <lorcan.browne@hp.com>
Date:   Tue Mar 24 13:54:00 2015 +0000

Add lowest option to swift-recon disk usage output
    
    Currently there is a "--top" option when running swift-recon for
    disk usage stats. This option lists the x disks with the highest
    disk usage in descending order.
    
    This feature adds a "--lowest" option which does the opposite and
    lists the y disks with lowest disk usage in ascending order.
    
    Have also updated the docs section with --top and --lowest options
    
    Change-Id: Ic15d407fe010a31995c2bdd9fb88548a1057f569

commit b76d8c5156c65e3ae6ead5213e40f70d80f7fa9d
Author: Mike Fedosin <mfedosin@mirantis.com>
Date:   Wed Apr 8 15:00:27 2015 +0300

Fix broken test_policy_IO_override test
    
    Since the existing check depends on how quickly the test
    passes, it causes to the unpredictable behavior and may break
    the test.
    
    This commit removes the unnecessary dependency on the
    current time and leaves only the check that timestamp is
    greater than 0.
    
    Change-Id: I3321e9aba0df52cb78523710b26f310f35801674
    Closes-Bug: #1441599

commit 26bff69cb5c5f86815ea8b4592a1a0119475023b
Author: Andreas Jaeger <aj@suse.de>
Date:   Wed Apr 15 09:43:26 2015 +0200

Fix untranslatable string
    
    The word "number" is not marked for translation, we cannot just add
    it this way to a string.
    Instead create two separate strings.
    
    Change-Id: I0396f1d7405987ed1c57f0344e1c5ebeca30a2e3
    Closes-Bug: #1444327

commit 749bdac1df3571324af6bdcf64e1d283f9ec6cb9
Author: Hisashi Osanai <osanai.hisashi@jp.fujitsu.com>
Date:   Mon Mar 23 13:47:22 2015 +0900

Refactor the getting roles logic in _keystone_identity
    
    The list_from_csv in swift.common.utils is used from the composite
    auth token support and this method has been used in
    _integral_keystone_identity. There is same logic in
    _keystone_identity so the logic would be replaced with list_from_csv.
    
    Change-Id: I3d72a34e6fc21fbe1d7331954695b1e3e1b67816

commit 8a1453e61e95956fd8a87546e068a60a275ae665
Author: Donagh McCabe <donagh.mccabe@hp.com>
Date:   Wed Feb 25 13:54:55 2015 +0000

Support HTTP_X_SERVICE_IDENTITY_STATUS in keystoneauth
    
    Keystone's auth_token now supports delay denial when
    a service token is invalid. This change handles
    the consequences for determining 401 vs 403 responses.
    
    Related-Bug: #1422389
    
    Change-Id: I90ba062e357cccea061c7101463b82b7c5e69293

commit d3213fb1fe00ae649ba198577b7e8b37180d3753
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Tue Mar 31 08:51:18 2015 +0000

Check if device name is valid when adding to the ring
    
    Currently device names can be empty or start and/or end with spaces.
    This can create unexpected results, for example these three commands
    are all valid:
    
    swift-ring-builder account.builder add "r1z1-127.0.0.1:6000/" 1
    swift-ring-builder account.builder add "r1z1-127.0.0.1:6000/sda " 1
    swift-ring-builder account.builder add "r1z1-127.0.0.1:6000/ meta" 1
    
    This patch validates device names and prevents empty names or names
    starting and/or ending with spaces.
    
    Also fixed the test "test_warn_at_risk" - the test passed if the
    exception was not raised.
    
    Closes-Bug: 1438579
    
    Change-Id: I811b0eae7db503279e6429d985275bbab8b29c9f

commit 3c419dfb12f19d31d6685ad76c0cf78d5e28f147
Author: Richard Hawkins <richard.hawkins@rackspace.com>
Date:   Sat Feb 14 10:19:16 2015 -0600

Add additional func tests for TempURLs
    
    Change-Id: I25097c87d0a06c283b8ee5067b840f0a22776369

commit 8f5d4d24557887b4691fc219cefbc30e478bf7ed
Author: Paul Luse <paul.e.luse@intel.com>
Date:   Thu Jul 3 15:59:33 2014 -0700

Erasure Code Documentation
    
    This patch adds all the relevant EC documentation to
    the source tree. Notable additions are:
      - Updated SAIO documentation
      - Updates to existing swift documentation; and
      - Erasure Coding overview
    
    Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
    Co-Authored-By: Thiago da Silva <thiago@redhat.com>
    Co-Authored-By: John Dickinson <me@not.mn>
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
    Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
    Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
    Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
    Change-Id: I0403016a4bb7dad9535891632753b0e5e9d402eb
    Implements: blueprint swift-ec
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit 647b66a2ce4c85c43dcca49776d35c5ebb9cf15e
Author: paul luse <paul.e.luse@intel.com>
Date:   Tue Oct 28 09:51:06 2014 -0700

Erasure Code Reconstructor
    
    This patch adds the erasure code reconstructor. It follows the
    design of the replicator but:
      - There is no notion of update() or update_deleted().
      - There is a single job processor
      - Jobs are processed partition by partition.
      - At the end of processing a rebalanced or handoff partition, the
        reconstructor will remove successfully reverted objects if any.
    
    And various ssync changes such as the addition of reconstruct_fa()
    function called from ssync_sender which performs the actual
    reconstruction while sending the object to the receiver
    
    Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
    Co-Authored-By: Thiago da Silva <thiago@redhat.com>
    Co-Authored-By: John Dickinson <me@not.mn>
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
    Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
    Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
    Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
    blueprint ec-reconstructor
    Change-Id: I7d15620dc66ee646b223bb9fff700796cd6bef51

commit b2189ef47ae08c39c348e7f4c90697ecb9ba64f9
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Apr 3 16:23:14 2015 -0700

Fix account reaper for > 3 replicas
    
    There's a pre-existing IndexError in the pop from the container node
    list in reaper's reap_object method for object rings with a replica
    count greater than the container replica count.  Which is more likely
    on EC storage policies.
    
    When making the backend direct delete requests to the nodes once the
    container node's list is exhausted the generic exception handler logs
    the error and breaks out of any other backend object requests - but
    the reaper marches forward and eventually the tombstones are
    replicated.
    
    This change just cycles the container headers across all the nodes -
    which seems reasonable enough - but could certainly garner
    bikeshedding.
    
    Change-Id: I5897d00b0a8c1e05884945dd93d9ce891b207001

commit 61a9d35fd58381b7c299f125ef01d00f9b0203fe
Author: Yuan Zhou <yuan.zhou@intel.com>
Date:   Wed Dec 24 11:52:34 2014 +0800

Update container sync to use internal client
    
    This patch changes container sync to use Internal Client instead
    of Direct Client.
    
    In the current design, container sync uses direct_get_object to
    get the newest source object(which talks to storage node directly).
    This works fine for replication storage policies however in
    erasure coding policies, direct_get_object would only return part
    of the object(it's encoded as several pieces). Using Internal
    Client can get the original object in EC case.
    
    Note that for the container sync put/delete part, it's working in
    EC since it's using Simple Client.
    
    Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
    
    DocImpact
    Change-Id: I91952bc9337f354ce6024bf8392046a1ecf6ecc9

commit decbcd24d41d6367901db16aaa2578f74870b6b5
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Wed Oct 22 13:18:34 2014 -0700

Foundational support for PUT and GET of erasure-coded objects
    
    This commit makes it possible to PUT an object into Swift and have it
    stored using erasure coding instead of replication, and also to GET
    the object back from Swift at a later time.
    
    This works by splitting the incoming object into a number of segments,
    erasure-coding each segment in turn to get fragments, then
    concatenating the fragments into fragment archives. Segments are 1 MiB
    in size, except the last, which is between 1 B and 1 MiB.
    
    +====================================================================+
    |                             object data                            |
    +====================================================================+
    
                                       |
              +------------------------+----------------------+
              |                        |                      |
              v                        v                      v
    
    +===================+    +===================+         +==============+
    |     segment 1     |    |     segment 2     |   ...   |   segment N  |
    +===================+    +===================+         +==============+
    
              |                       |
              |                       |
              v                       v
    
         /=========\             /=========\
         | pyeclib |             | pyeclib |         ...
         \=========/             \=========/
    
              |                       |
              |                       |
              +--> fragment A-1       +--> fragment A-2
              |                       |
              |                       |
              |                       |
              |                       |
              |                       |
              +--> fragment B-1       +--> fragment B-2
              |                       |
              |                       |
             ...                     ...
    
    Then, object server A gets the concatenation of fragment A-1, A-2,
    ..., A-N, so its .data file looks like this (called a "fragment archive"):
    
    +=====================================================================+
    |     fragment A-1     |     fragment A-2     |  ...  |  fragment A-N |
    +=====================================================================+
    
    Since this means that the object server never sees the object data as
    the client sent it, we have to do a few things to ensure data
    integrity.
    
    First, the proxy has to check the Etag if the client provided it; the
    object server can't do it since the object server doesn't see the raw
    data.
    
    Second, if the client does not provide an Etag, the proxy computes it
    and uses the MIME-PUT mechanism to provide it to the object servers
    after the object body. Otherwise, the object would not have an Etag at
    all.
    
    Third, the proxy computes the MD5 of each fragment archive and sends
    it to the object server using the MIME-PUT mechanism. With replicated
    objects, the proxy checks that the Etags from all the object servers
    match, and if they don't, returns a 500 to the client. This mitigates
    the risk of data corruption in one of the proxy --> object connections,
    and signals to the client when it happens. With EC objects, we can't
    use that same mechanism, so we must send the checksum with each
    fragment archive to get comparable protection.
    
    On the GET path, the inverse happens: the proxy connects to a bunch of
    object servers (M of them, for an M+K scheme), reads one fragment at a
    time from each fragment archive, decodes those fragments into a
    segment, and serves the segment to the client.
    
    When an object server dies partway through a GET response, any
    partially-fetched fragment is discarded, the resumption point is wound
    back to the nearest fragment boundary, and the GET is retried with the
    next object server.
    
    GET requests for a single byterange work; GET requests for multiple
    byteranges do not.
    
    There are a number of things _not_ included in this commit. Some of
    them are listed here:
    
     * multi-range GET
    
     * deferred cleanup of old .data files
    
     * durability (daemon to reconstruct missing archives)
    
    Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
    Co-Authored-By: Thiago da Silva <thiago@redhat.com>
    Co-Authored-By: John Dickinson <me@not.mn>
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
    Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
    Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
    Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
    Change-Id: I9c13c03616489f8eab7dcd7c5f21237ed4cb6fd2

commit b1eda4aef8a228961d5aafe7e4fbd4e812d233ad
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Tue Sep 16 18:40:41 2014 -0700

Allow sending object metadata after data
    
    This lets the proxy server send object metadata to the object server
    after the object data. This is necessary for EC, as it allows us to
    compute the etag of the object in the proxy server and still store it
    with the object.
    
    The wire format is a multipart MIME document. For sanity during a
    rolling upgrade, the multipart MIME document is only sent to the
    object server if it indicates, via 100 Continue header, that it knows
    how to consume it.
    
    Example 1 (new proxy, new obj server):
    
       proxy: PUT /p/a/c/o
              X-Backend-Obj-Metadata-Footer: yes
    
         obj: 100 Continue
            X-Obj-Metadata-Footer: yes
    
       proxy: --MIMEmimeMIMEmime...
    
    Example2: (new proxy, old obj server)
    
       proxy: PUT /p/a/c/o
              X-Backend-Obj-Metadata-Footer: yes
    
         obj: 100 Continue
    
       proxy: <obj body>
    
    Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
    Co-Authored-By: Thiago da Silva <thiago@redhat.com>
    Co-Authored-By: John Dickinson <me@not.mn>
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
    Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
    Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
    Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
    Change-Id: Id38f7e93e3473f19ff88123ae0501000ed9b2e89

commit fa89064933fefa33702520b40734c11f08b2c569
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Tue Mar 17 08:32:57 2015 +0000

Per-policy DiskFile classes
    
    Adds specific disk file classes for EC policy types.
    
    The new ECDiskFile and ECDiskFileWriter classes are used by the
    ECDiskFileManager.
    
    ECDiskFileManager is registered with the DiskFileRouter for use with
    EC_POLICY type policies.
    
    Refactors diskfile tests into BaseDiskFileMixin and BaseDiskFileManagerMixin
    classes which are then extended in subclasses for the legacy
    replication-type DiskFile* and ECDiskFile* classes.
    
    Refactor to prefer use of a policy instance reference over a policy_index
    int to refer to a policy.
    
    Add additional verification to DiskFileManager.get_dev_path to validate the
    device root with common.constraints.check_dir, even when mount_check is
    disabled for use in on a virtual swift-all-in-one.
    
    Co-Authored-By: Thiago da Silva <thiago@redhat.com>
    Co-Authored-By: John Dickinson <me@not.mn>
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
    Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
    Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
    Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
    Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
    Change-Id: I22f915160dc67a9e18f4738c1ddf068344e8ad5d

commit dd9d97458ea007024220a78dba8dd663e8b425d7
Author: John Dickinson <me@not.mn>
Date:   Fri Mar 20 10:17:25 2015 +0000

Prevent unauthorized delete in versioned container
    
    An authenticated user can delete the most recent version of any
    versioned object who's name is known if the user has listing access
    to the x-versions-location container. Only Swift setups with
    allow_version setting are affected.
    
    This patch closes this bug, tracked as CVE-2015-1856
    
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Co-Authored-By: Christian Schwede <info@cschwede.de>
    Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
    
    Closes-Bug: 1430645
    Change-Id: Ibacc7413afe7cb6f77d92e5941dcfdf4768ffa18

commit a7078293349a01a7594f6457fccb1aca31d98fd4
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Mar 31 22:35:37 2015 -0700

Update test infrastructure
    
     * Get FakeConn ready for expect 100 continue
     * Use debug_logger more and with better interfaces
     * Fix patch_policies to be less annoying
    
    Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
    Co-Authored-By: Thiago da Silva <thiago@redhat.com>
    Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
    Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
    Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
    Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
    Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
    Change-Id: I28c0a3539d994cbb8e6b94d63a23ed4ea6cb956d

commit ed5406628884432e23bbabb02d2855d4b51a332d
Author: Tushar Gohad <tushar.gohad@intel.com>
Date:   Mon Jun 30 11:14:28 2014 -0700

Add support for policy types, 'erasure_coding' policy
    
    This patch extends the StoragePolicy class for non-replication storage
    policies, the first one being "erasure coding".
    
    Changes:
    
     - Add 'policy_type' support to BaseStoragePolicy class
     - Disallow direct instantiation of BaseStoragePolicy class
    
     - Subclass BaseStoragePolicy
    
       - "StoragePolicy":
         . Replication policy, default
         . policy_type = 'replication'
    
       - "ECStoragePolicy":
         . Erasure Coding policy
         . policy_type = 'erasure_coding'
         . Private member variables
           ec_type (EC backend),
           ec_num_data_fragments (number of fragments original
             data split into after erasure coding operation),
           ec_num_parity_fragments (number of parity fragments
           generated during erasure coding)
         . Private methods
           EC specific attributes and ring validator methods.
    
     - Swift will use PyECLib, a Python Erasure Coding library, for
       erasure coding operations. PyECLib is already an approved
       OpenStack core requirement.
       (https://bitbucket.org/kmgreen2/pyeclib/)
    
     - Add test cases for
       - 'policy_type' StoragePolicy member
       - policy_type == 'erasure_coding'
    
    DocImpact
    
    Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
    Co-Authored-By: Thiago da Silva <thiago@redhat.com>
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
    Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
    Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
    Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
    Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386
    Implements: blueprint ec-proxy-work

commit 16ee994c1ea4ddcfaeae26babada6382daa69bc9
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Wed Feb 18 10:17:23 2015 +0000

Set connection timeout in container sync
    
    Container sync might get stuck without a connection timeout if the remote proxy
    is not responding.
    
    This patch sets a default timeout of 5.0 seconds for the connection attempt. The
    value is much higher than other connection timeouts inside Swift (0.5); however
    there might be a much higher latency to the remote peer, thus playing it safe.
    There is also a retry if the attempt timed out.
    
    Note that this setting only applies to the connection request itself. Setting
    this timeout does not apply when the remote proxy goes away during a request.
    
    Also added a short test to ensure urlopen is called with the timeout value.
    
    Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
    
    Change-Id: Ic08a55157fa91fe1316653781adf4d66eead61bc
    Partial-Bug: 1419916

commit 376ecc74fccef390e4e4de66cae3ef20bcbd021f
Author: John Dickinson <me@not.mn>
Date:   Wed Apr 1 12:41:25 2015 -0700

make `git review` easier
    
    Change-Id: I6032a7915586db6da25aafccf3dced581b37df9c

commit 8d3b3b2ee0a86b37b6ad1f4d201bfaa0103af07d
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Sun Jan 11 13:41:35 2015 -0800

Add some debug output to the ring builder
    
    Sometimes, I get handed a builder file in a support ticket and a
    question of the form "why is the balance [not] doing $thing?". When
    that happens, I add a bunch of print statements to my local
    swift/common/ring/builder.py, figure things out, and then delete the
    print statements. This time, instead of deleting the print statements,
    I turned them into debug() calls and added a "--debug" flag to the
    rebalance command in hopes that someone else will find it useful.
    
    Change-Id: I697af90984fa5b314ddf570280b4585ba0ba363c

commit cf21db4bc68aff190c55d416b406a44bf204ba36
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu Mar 26 21:43:48 2015 -0700

Add Swift Design Principles to CONTRIBUTING.md
    
    This is based of work started a few design summits ago [1].
    
    With EC getting close I felt like I could the inspiration.
    
    1. https://etherpad.openstack.org/p/juno_swift_core_principles
    
    Change-Id: I90fa56d67003080fe3d7bc4e0fad053e3b0c8504
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit a8129410030eb03836cf9323a52dc8030b495e77
Author: Tushar Gohad <tushar.gohad@intel.com>
Date:   Fri Mar 13 01:36:53 2015 -0700

Bump eventlet version to 0.16.1
    
    This version adds support for multiple 100-continue responses
    via the API send_hundred_continue_response(), which we need for
    the two-phase scheme for PUTs with Erasure Coding
    
    Change-Id: I7154ef1dd96ab761894b09d5e6abf98a453f7b64

commit 75e84543aab5af8038e0e1a369fcf0ed10ea2509
Author: Hisashi Osanai <osanai.hisashi@jp.fujitsu.com>
Date:   Mon Mar 23 11:29:16 2015 +0900

Clarify the description of backward compatibility in Keystoneauth's docstring
    
    If no prefix implies for the operator_roles and/or service_roles
    parameters applies to all reseller_prefixes in the code.
    But from backward comatibility point of view, the service_roles
    parameter is newly introduced parameter so it is better to clarify
    the docstring.
    
    Change-Id: I8b0d5aeff785c21b8a8b0f5b20f774fad868016d

commit 76b106fc017922997cbe6bef3bcee309e39c7c82
Author: Martin Kletzander <mkletzan@redhat.com>
Date:   Thu Mar 19 17:25:57 2015 +0100

Fix common misspellings
    
    Wikipedia's list of common misspellings [1] has a machine-readable
    version.  This patch fixes those misspellings mentioned in the list
    which don't have multiple right variants (as e.g. "accension", which can
    be both "accession" and "ascension"), such misspellings are left
    untouched.  The list of changes was manually re-checked for false
    positives.
    
    [1] https://en.wikipedia.org/wiki/Wikipedia:Lists_of_common_misspellings/For_machines
    
    Change-Id: Ic9a5438629664f7cea216413a28acc0e8992da05
    Signed-off-by: Martin Kletzander <mkletzan@redhat.com>

commit a288d05ca5d4c66dab36df40b4216e2ed87e5a91
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Tue Mar 24 06:06:33 2015 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I6ae0dd0c34ac1d7173acacdd0dd4896619592e0c

commit 0a467936621009418a9c94f636b33bfb6d947a89
Author: Lorcan <lorcan.browne@hp.com>
Date:   Thu Mar 12 15:40:39 2015 +0000

Add swift-recon feature to track swift-drive-audit error count
    
    This is a follow-on from a previous commit which added recon info
    for swift-drive-audit (https://review.openstack.org/#/c/122468/).
    
    Here, the "--drievaudit" option is added to swift-recon tool. This
    feature gives the statistics for the system-wide drive errors flagged
    by swift-drive-audit. An example of the output is as follows:
    (verbose mode)
    
    swift-recon --driveaudit -v
    ===============================================================================
    --> Starting reconnaissance on 5 hosts
    ===============================================================================
    [2015-03-11 17:13:39] Checking drive-audit errors
    -> http://1.2.3.4:6000/recon/driveaudit: {'drive_audit_errors': 14}
    -> http://1.2.3.5:6000/recon/driveaudit: {'drive_audit_errors': 0}
    -> http://1.2.3.6:6000/recon/driveaudit: {'drive_audit_errors': 37}
    -> http://1.2.3.7:6000/recon/driveaudit: {'drive_audit_errors': 101}
    -> http://1.2.3.8:6000/recon/driveaudit: {'drive_audit_errors': 0}
    [drive_audit_errors] low: 0, high: 101, avg: 30.4, total: 152, Failed: 0.0%, no_result: 0, reported: 5
    ===============================================================================
    
    Change-Id: Ia16c52a9d613eeb3de1a5a428d88dd1233631912

commit 23d0842dec250905f68df601926ba8228392b322
Author: Thiago da Silva <thiago@redhat.com>
Date:   Tue Feb 17 16:55:34 2015 -0500

Refactoring the PUT method
    
    Extracting large chunks of the PUT method into smaller
    methods to improve maintainability and reuse of code.
    
    Based on the work that Clay Gerrard started:
    https://review.openstack.org/#/c/77812/
    
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    
    Change-Id: Id479fc5b159a2782361ac4a6e4a6d8bbaee4fe85
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit c3d2602c60d8a218409c9f95bef252e4811cbd29
Author: Mitsuhiro SHIGEMATSU <shigematsu.mitsuhiro@lab.ntt.co.jp>
Date:   Sat Mar 21 08:42:23 2015 +0900

Fix typo in swift/test/unit/account/test_backend.py
    
    Change-Id: I6cab110599019471b4005f1584fc26fa2a85d02e

commit 7e64c811fe9ecae738474239a9f73e3c6f95ddea
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Wed Feb 25 17:33:44 2015 +0000

Enable in-process functional test policy to be configured
    
    Currently the in-process tests build a 2 replica, 4 partition,
    2 device object ring. This patch allows an alternative policy
    and ring to be specified for testing via environment variables
    that may optionally be set.
    
    SWIFT_TEST_IN_PROCESS_CONF_DIR - This points the test setup to a
      directory which may have a swift.conf file and ring file. The
      test setup will then prefer these conf files over the samples
      in '/etc'.
    
    SWIFT_TEST_POLICY - This causes the in-process test to
      use the specified policy from the swift.conf file and its
      associated ring for testing (first copying the conf and ring file
      and modifying device parameters to suit in-process testing). If
      not set, the tests will use the default policy.
    
    The in-process tests now start sufficient object servers for the
    ring file being tested against.
    
    This should allow in-process functional testing of various policies
    and rings (e.g. EC policies) without needing to reconfigure an SAIO
    for each test scenario.
    
    The refactoring of the in_process test setup code should also
    allow easier addition of other 'hard-coded' test policies/rings
    in the future.
    
    Change-Id: I24f5a13de3d296b400da1691dcb53423a9f8a463

commit 9ae0a0702c4a32407b1d00e93bd1be01e2ec26bc
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Fri Mar 20 00:59:15 2015 -0700

Fix a lack of method arguments at tempauth
    
    This fixes a kind of trivial issue with TypeError caused by a lack
    of method argument at calling.
    
    When using 'Authorization' header with tempauth, tempauth calls
    _get_user_groups with 3 arguments (includes self) on current code
    but the method requires 4 arguments so it will go to fail with 500.
    
    This patch fixes to take correct arguments and adds an unit test
    for the case.
    
    Closes-Bug:1434465
    
    Change-Id: Ibf00c41fa8e5cfdaf49bc4d571e575d7f229ebfe

commit a624a593df7e92b06f5844e53a87a65c662aba4c
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Thu Mar 19 11:02:09 2015 +0000

Tighten up recon middleware unit tests
    
    Adds a test for get_expirer_info() and tightens up
    the test for get_async_info().
    
    While reviewing [1] I noticed that the existing tests on
    master will not catch a bug creeping into the arg strings
    that these methods pass to from_recon_cache().
    
    [1] https://review.openstack.org/#/c/163889
    
    Change-Id: Iee6f12aeff684d70435dbeea769ccc181021be79

commit 15b83f67d2fc6645a5e68ce923ca9a2d4d1b9805
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Wed Mar 18 12:43:03 2015 +0000

Test swift-object-info opens meta and ts files
    
    Adds a unit test to verify the change made in [1], i.e. that
    swift-object-info will read from .meta and .ts files as well
    as .data files.
    
    [1] change I43966d371218ad39414e9282cde579e48370a2a7
    
    Change-Id: I82dde36e3a96db1a21cfe9a4cca0d941e543dfd0
    Related-Bug: 1425679

commit 3f1feaee4215dde618431571983c64d0290bbea0
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Wed Mar 18 06:11:23 2015 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I2e0a6053fec208b3eea9a68416fdc616da0631a9

commit 3d3db0ab789f37a69fc178eeb74c8cc61e4c6c1b
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Fri Mar 13 13:19:44 2015 +0000

Bump PyECLib version to >= 1.0.3
    
    ECPyECLibException has been removed in 1.0.1, among other
    API/error handling enhancements in 1.0.3. Let's make sure
    the right version is used.
    
    Change-Id: Ie6e0c1b491017dd5096b14c23b610abc22b03d6a

commit fdb2ec0cc7428457a836a23f4b18348e037e6805
Author: Jiangmiao Gao <tolbkni@gmail.com>
Date:   Mon Mar 16 23:30:37 2015 +0800

Add the missing "file" keyword for print
    
    All non-keyword arguments are converted to strings and written to
    the stream, the argument "sys.stderr" should be given keyword
    "file" as keyword argument.
    
    Change-Id: I1f9c4e41ed7a1d3246eae9179fd9221c0a834292

commit 51a8f9c64fa16a2625066e406bb9489aae67cb7f
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Mon Mar 16 06:10:17 2015 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I0058932520e008aa4bde7ce34d6c9f25424684b4

commit 99a3d1a1c5d9591eed5378cf83af78578b01b386
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Fri Mar 13 04:39:16 2015 -0700

Fix the prefix of messages caputured from stderr
    
    Swift caputures both sys.stdout and sys.stderr messages and
    transfers them to syslog. However, whole of the messages
    uses the "STDOUT:" prefix, currently.
    
    It will be quite confusable for debugging purpose because we
    cannot make sure whether "Swift didn't caputure stderr" or
    "Swift captured stderr but didn't show it" when we want to log
    some stderr messages.
    
    This patch enables to allow a new prefix "STDERR:" and use it
    for sys.stderr.
    
    Change-Id: Idf4649e9860b77c3600a5cd0a3e0bd674b31fb4f

commit 9baafe7f305147c70d004402ebb05a22db85bc18
Author: Ricardo Ferreira <ricardo.sff@gmail.com>
Date:   Fri Mar 6 21:18:48 2015 +0000

Allow swift-object-info to inspect .meta and .ts files
    
    Removing the check for ".data" extension makes it check
    *any* file for metadata, so it works with .meta and .ts
    filetypes.
    
    Change-Id: I43966d371218ad39414e9282cde579e48370a2a7
    Closes-Bug:1425679

commit 0eb27b96f267a1a8419efeb96d68e3773d70e049
Author: Prashanth Pai <ppai@redhat.com>
Date:   Thu Mar 5 19:01:38 2015 +0530

Handle ENOSPC in mkstemp()
    
    mkstemp() can fail with ENOSPC when filesystem runs out of inodes.
    And fallocate() used to raise DiskFileNoSpace for all OSErrors.
    
    Change-Id: I8c95cb710107d8e481d068b00eda53dd805c00a5
    Signed-off-by: Prashanth Pai <ppai@redhat.com>

commit 16b435a4a8ae12f456a55872fd8ebe21b0841704
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Mon Feb 23 00:26:30 2015 -0800

Fix ContainerBroker to use policy-0 in default
    
    Fix ContainerBroker to initialize as policy-0 on policy_stat
    table in default when storage_policy_index argument is NOT given.
    
    Current ContainerBroker makes policy-1 stats in default because
    the "None" value will be passed through to the last function of
    db access query (i.e. a query like as "INSERT INTO policy_stat
    (storage_policy_index) VALUES (None)" will appear) which results
    in a row "(1, 0, 0)" (the first value is the policy index) by
    the PRIMARY KEY constraint on sqlite.
    
    In worst case, container db keeps two policies, and then, ContainerBroker.get_info
    might return invalid (non-touched) policy_stat information as container information.
    (See tests in detail)
    
    When using ContainerBroker with no storage_policy_index argument,
    it should always act policy-0 simply.
    
    Note that this patch doesn't affect immediately Swift behavior because
    current swift ensures to use policy-0 on "Container-Server" when invalid
    policy (includes None) is given. However, we should recheck also in
    ContainerBroker for safety to prevent the unfortunate behavior above.
    
    Change-Id: If64f0c94c069a2cc3140c99f21b8d371c183e28a

Christian Schwede (cschwede) on 2015-06-11

Changed in swift:
status:	In Progress → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.