Bug #1255049 “Invalid X-Copy-From value and Container Quotas res...” : Bugs : OpenStack Object Storage (swift)

Fabien Boucher (fabien-boucher) on 2013-11-26

Changed in swift:
assignee:	nobody → Fabien Boucher (fabien-boucher)

OpenStack Infra (hudson-openstack) on 2013-11-26

Changed in swift:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-02-14: Fix proposed to swift (feature/ec)

#1

Fix proposed to branch: feature/ec
Review: https://review.openstack.org/73471

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-02-15: Fix merged to swift (feature/ec)

#2

Download full text (18.8 KiB)

Reviewed: https://review.openstack.org/73471
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=aa0b0cf702cbaec1ee2711da15d89d3fedebbd76
Submitter: Jenkins
Branch: feature/ec

commit 39d22ccec726ec8a56ef2771c1315b4a7aee5922
Author: Christian Schwede <email address hidden>
Date: Wed Feb 12 20:26:29 2014 +0000

Don't create bin/* files magically

Just use import to make scripts available in bin/ instead of
creating these during setup.py install.

Change-Id: I7318bbb77f6564ed58736887e711e1c497873471

commit b1742169e163b46583dba38447b0c12a51ca324e
Author: anc <email address hidden>
Date: Wed Feb 12 11:18:19 2014 +0000

Remove duplicate doc entry for swob

Fixes swob module being referenced twice in misc.rst
resulting in duplicate sections in the doc.

Also fixes build_sphinx warning for section underline
too short in middleware.rst.

Change-Id: Ibe44895f933a6503ca04ccd3a084bc0cfd913213

commit 5fca0d07a873106b2175bcb3b5bfbee1cbd04a31
Author: Samuel Merritt <email address hidden>
Date: Wed Feb 5 11:57:20 2014 -0800

Ensure swift.source is set for DLO/SLO requests

    SLO was setting it sometimes (the PUT path), but not others (the GET
    path), and DLO just wasn't doing it at all. One could already
    distinguish DLO/SLO internal requests by the user agent, but we should
    set swift.source too.

Change-Id: I5d5b7dd49bd1522a9c830d0abd21fff92ae79a39

commit 8b4876f32a2e66bb2e1606a75dcc9a09ced43ab2
Author: Greg Lange <email address hidden>
Date: Mon Feb 10 14:31:14 2014 +0000

Fix recon docs

Change-Id: Icaa0f61e5796253dcc57b8c005577890de8aa537

commit 4a970ab5a6d2345670899ae082f26ff6a870e5de
Author: Greg Lange <email address hidden>
Date: Thu Feb 6 23:14:45 2014 +0000

Fix a comment in SLO middleware

Change-Id: I6ef435db55eb6cac5a92d9ef0f7ac3eaf271761d

commit 0f17c185c1bd9dd59796fc0ff5e66f821f0bd3b3
Author: Greg Lange <email address hidden>
Date: Thu Feb 6 17:10:19 2014 +0000

Fix a comment in bulk middleware.

Change-Id: I3a65d090c5c6ce42639e0af0fcac738a12e52252

commit aad81528d4a0b443eb7bfccbcd3c1e523ed8efa9
Author: Prashanth Pai <email address hidden>
Date: Tue Feb 4 16:31:47 2014 +0530

Make .expiring_objects account name configurable

    The account which tracks objects scheduled for deletion had its name
    hard-coded to 'expiring_objects'. This is made configurable via
    expiring_objects_account_name option. Backend file-systems
    integration efforts may want to treat these "special" accounts in a
    different way.

This would still go undocumented, hence 'pseudo-hidden'.

UpgradeImpact: None as the default value would continue to be the same
which is '.expiring_objects'.

Change-Id: I1a093b0d0e2bdd0c3d723090af03fc0adf2ad7e3
Signed-off-by: Prashanth Pai <email address hidden>

commit 1f3ae6d8dadef838f1ea7fa571339932074fbe38
Author: Christian Schwede <email address hidden>
Date: Thu Jan 9 10:15:50 2014 +0000

Remove swiftclient dependency

Removes the requirement for swiftclient in swift-dispersion-report
...

Reviewed:  https://review.openstack.org/73471
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=aa0b0cf702cbaec1ee2711da15d89d3fedebbd76
Submitter: Jenkins
Branch:    feature/ec

commit 39d22ccec726ec8a56ef2771c1315b4a7aee5922
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Wed Feb 12 20:26:29 2014 +0000

Don't create bin/* files magically
    
    Just use import to make scripts available in bin/ instead of
    creating these during setup.py install.
    
    Change-Id: I7318bbb77f6564ed58736887e711e1c497873471

commit b1742169e163b46583dba38447b0c12a51ca324e
Author: anc <alistair.coles@hp.com>
Date:   Wed Feb 12 11:18:19 2014 +0000

Remove duplicate doc entry for swob
    
    Fixes swob module being referenced twice in misc.rst
    resulting in duplicate sections in the doc.
    
    Also fixes build_sphinx warning for section underline
    too short in middleware.rst.
    
    Change-Id: Ibe44895f933a6503ca04ccd3a084bc0cfd913213

commit 5fca0d07a873106b2175bcb3b5bfbee1cbd04a31
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Wed Feb 5 11:57:20 2014 -0800

Ensure swift.source is set for DLO/SLO requests
    
    SLO was setting it sometimes (the PUT path), but not others (the GET
    path), and DLO just wasn't doing it at all. One could already
    distinguish DLO/SLO internal requests by the user agent, but we should
    set swift.source too.
    
    Change-Id: I5d5b7dd49bd1522a9c830d0abd21fff92ae79a39

commit 8b4876f32a2e66bb2e1606a75dcc9a09ced43ab2
Author: Greg Lange <greglange@gmail.com>
Date:   Mon Feb 10 14:31:14 2014 +0000

Fix recon docs
    
    Change-Id: Icaa0f61e5796253dcc57b8c005577890de8aa537

commit 4a970ab5a6d2345670899ae082f26ff6a870e5de
Author: Greg Lange <greglange@gmail.com>
Date:   Thu Feb 6 23:14:45 2014 +0000

Fix a comment in SLO middleware
    
    Change-Id: I6ef435db55eb6cac5a92d9ef0f7ac3eaf271761d

commit 0f17c185c1bd9dd59796fc0ff5e66f821f0bd3b3
Author: Greg Lange <greglange@gmail.com>
Date:   Thu Feb 6 17:10:19 2014 +0000

Fix a comment in bulk middleware.
    
    Change-Id: I3a65d090c5c6ce42639e0af0fcac738a12e52252

commit aad81528d4a0b443eb7bfccbcd3c1e523ed8efa9
Author: Prashanth Pai <ppai@redhat.com>
Date:   Tue Feb 4 16:31:47 2014 +0530

Make .expiring_objects account name configurable
    
    The account which tracks objects scheduled for deletion had its name
    hard-coded to 'expiring_objects'. This is made configurable via
    expiring_objects_account_name option. Backend file-systems
    integration efforts may want to treat these "special" accounts in a
    different way.
    
    This would still go undocumented, hence 'pseudo-hidden'.
    
    UpgradeImpact: None as the default value would continue to be the same
                   which is '.expiring_objects'.
    
    Change-Id: I1a093b0d0e2bdd0c3d723090af03fc0adf2ad7e3
    Signed-off-by: Prashanth Pai <ppai@redhat.com>

commit 1f3ae6d8dadef838f1ea7fa571339932074fbe38
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Thu Jan 9 10:15:50 2014 +0000

Remove swiftclient dependency
    
    Removes the requirement for swiftclient in swift-dispersion-report
    and swift-dispersion-populate. To prevent a dependency on
    keystoneclient and to avoid reinventing the wheel with an internal
    keystoneclient, authentication with keystone is only supported if
    swiftclient is available. If not, only auth v1 is supported.
    
    The dependency in swift/container/sync.py has also been removed.
    
    Implements: blueprint remove-swiftclient-dependency
    
    Change-Id: I6ec3b3c85a67b9ab6eb04b90ffc16daf1600e8a7

commit 6241ab2a9292ba7e97034eac5f1218e90aedddc5
Author: Greg Lange <greglange@gmail.com>
Date:   Wed Feb 5 22:50:31 2014 +0000

Fix a couple ifs in account quota middleware.
    
    Unless I'm misunderstanding something, this is better/more
    correct.
    
    Change-Id: I6c62fbfdddea908a2671021a34fa25600b0dff45

commit 24657b2b39ae56fcd2b8a8089658ead84f9897c8
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Wed Feb 5 06:51:46 2014 +0000

Add tests for swift-ring-builder
    
    Add some tests for essential methods in swift-ring-builder.
    Tests for removing or changing device settings are executed
    with different search values to cover many possible command
    line arguments.
    
    Currently tested methods:
    
    - create ring
    - add device
    - remove device
    - set weight
    - set info
    - set min_part_hours
    - set replicas
    
    Tests use swift.common.ring.RingBuilder to verify actions.
    
    Catching and testing output from print statements is not
    tested, because this requires redirecting sys.stdout during
    tests and that might have some sideeffects for testing tools.
    
    bin/swift-ring-builder has been moved to swift/cli/ringbuilder.py
    and slightly modified to work as before (mainly due to no more
    existing global variables since that part of the code has been
    moved inside a main() function).
    
    Change-Id: Ia63f59a8faca1fad990784f27532ca07a2125454

commit 5cb7ac15c28c6e1aadbb4e446700904b42e15222
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Tue Feb 4 11:20:56 2014 -0700

Fix numbering of steps in proxy section of install_multinode
    
    Steps were numbered 1, 2, (note reset), 1, 2, etc. Then a user says:
    "I'm on Step 2 in Proxy section, er..."
    
    See a bug, fix a bug.
    
    Change-Id: If6f32b3a33e1070e705812df7ab299e6736c9806

commit 6acea29fa65e87ca9639172295c611412d7d6148
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Nov 21 17:31:16 2013 -0800

Move all DLO functionality to middleware
    
    This is for the same reason that SLO got pulled into middleware, which
    includes stuff like automatic retry of GETs on broken connection and
    the multi-ring storage policy stuff.
    
    The proxy will automatically insert the dlo middleware at an
    appropriate place in the pipeline the same way it does with the
    gatekeeper middleware. Clusters will still support DLOs after upgrade
    even with an old config file that doesn't mention dlo at all.
    
    Includes support for reading config values from the proxy server's
    config section so that upgraded clusters continue to work as before.
    
    Bonus fix: resolve 'after' vs. 'after_fn' in proxy's required filters
    list. Having two was confusing, so I kept the more-general one.
    
    DocImpact
    
    blueprint multi-ring-large-objects
    
    Change-Id: Ib3b3830c246816dd549fc74be98b4bc651e7bace

commit b881a1896fb9d218325bc7671f02a037e529287d
Author: Peter Portante <peter.portante@redhat.com>
Date:   Mon Feb 3 17:00:53 2014 -0500

Fix up double mocking of swob.Request.__init__
    
    From Pete Zaitcev's work on PBes (https://review.openstack.org/47713):
    
      Fixes up a problem with double mocking of Request.__init__, which
      should be seen in stock code as well.
    
    Change-Id: Ibf8103425404b03aaab9a4e97a3a62d0fb8dbb14

commit 0d710140c55c6a52f3c3aad1be2c56beb69dacb1
Author: Peter Portante <peter.portante@redhat.com>
Date:   Mon Feb 3 16:01:14 2014 -0500

Reduce line count by two
    
    Change-Id: Ib0ee2bb5b202966d3d07c754656c92006149af9c

commit 204240222fdf63ba13d20946919549f0a3c374ad
Author: Peter Portante <peter.portante@redhat.com>
Date:   Mon Feb 3 00:42:39 2014 -0500

Attempt to make test_memcache_pool_timeout stable
    
    Too much time can pass after we create a set of co-routines and when
    the parent co-routine continues is setup and staging. Instead, we
    perform all the setup and staging, let the system run and then verify
    the final state after all the dust settles.
    
    Change-Id: I801148e380807119d3d6da5a24ba9cced39fb339
    Closes-Bug: 1272503

commit 15c04f6869c3ae4bb8906b3783551df0e718d462
Author: Chuck Thier <cthier@gmail.com>
Date:   Fri Jan 31 18:58:52 2014 +0000

Attempt to fix periodic memcache timeout test
    
    Not sure this will really fix the issue, but made a small change that
    seemed to make more sense for the flow of the test.
    
    Closes-Bug: #1272503
    Change-Id: Ifb25ed7c4a42b11f74d865787f63fea4e0beb745

commit cd4b4da8b6362e6621602a8b815bbeeea2000ff7
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Tue Dec 17 09:10:31 2013 +0000

Add some tests for bin/swift-recon
    
    Fix also minor bug in zone filtering when zone set to 0.
    
    Moved bin/swift-recon to swift/cli/recon.py, which makes
    it possible to import it without using some scary hacks.
    bin/swift-recon is now created by setup.py install.
    
    Closes-Bug: #1261692
    Change-Id: Id0729991c8ece73604467480dbf93fec7d8eb196

commit e078dc3da05ce9e7c2b36e05686d28101381eec8
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Mon Jan 27 01:08:37 2014 -0800

Add missing sample config of object-replicator
    
    Change-Id: I2bca67023aeb9a012927c69e23d582d4a0ff2098

commit b89ac55c05e723731d0df01c7a6e15c7d439f7f5
Author: David Goetz <dpgoetz@gmail.com>
Date:   Wed Jan 29 08:33:27 2014 -0800

Add a way to ratelimit all writes to an account
    
    This is in case a cluster gets a problem user who has distributed the
    writes to a bunch of containers but is just taking too much of the
    cluster's resources.
    
    Change-Id: Ibd2ffd0e911463a432117b478585b9f8bc4a2495

commit 282fa0c398e2f9e73d524a5ec15ab5822523a2ed
Author: Jon Snitow <otherjon@swiftstack.com>
Date:   Wed Nov 13 20:55:14 2013 +0000

Privileged acct ACL header, new ACL syntax, TempAuth impl.
    
     * Introduce a new privileged account header: X-Account-Access-Control
     * Introduce JSON-based version 2 ACL syntax -- see below for discussion
     * Implement account ACL authorization in TempAuth
    
    X-Account-Access-Control Header
    -------------------------------
    
    Accounts now have a new privileged header to represent ACLs or any other
    form of account-level access control.  The value of the header is an opaque
    string to be interpreted by the auth system, but it must be a JSON-encoded
    dictionary.  A reference implementation is given in TempAuth, with the
    knowledge that historically other auth systems often use TempAuth as a
    starting point.
    
    The reference implementation describes three levels of account access:
    "admin", "read-write", and "read-only".  Adding new access control
    features in a future patch (e.g. "write-only" account access) will
    automatically be forward- and backward-compatible, due to the JSON
    dictionary header format.
    
    The privileged X-Account-Access-Control header may only be read or written
    by a user with "swift_owner" status, traditionally the account owner but
    now also any user on the "admin" ACL.
    
    Access Levels:
    
    Read-only access is intended to indicate to the auth system that this
    list of identities can read everything (except privileged headers) in
    the account.  Specifically, a user with read-only account access can get
    a list of containers in the account, list the contents of any container,
    retrieve any object, and see the (non-privileged) headers of the
    account, any container, or any object.
    
    Read-write access is intended to indicate to the auth system that this
    list of identities can read or write (or create) any container.  A user
    with read-write account access can create new containers, set any
    unprivileged container headers, overwrite objects, delete containers,
    etc.  A read-write user can NOT set account headers (or perform any
    PUT/POST/DELETE requests on the account).
    
    Admin access is intended to indicate to the auth system that this list of
    identities has "swift_owner" privileges.  A user with admin account access
    can do anything the account owner can, including setting account headers
    and any privileged headers -- and thus changing the value of
    X-Account-Access-Control and thereby granting read-only, read-write, or
    admin access to other users.
    
    The auth system is responsible for making decisions based on this header,
    if it chooses to support its use.  Therefore the above access level
    descriptions are necessarily advisory only for other auth systems.
    
    When setting the value of the header, callers are urged to use the new
    format_acl() method, described below.
    
    New ACL Format
    --------------
    
    The account ACLs introduce a new format for ACLs, rather than reusing the
    existing format from X-Container-Read/X-Container-Write.  There are several
    reasons for this:
     * Container ACL format does not support Unicode
     * Container ACLs have a different structure than account ACLs
      + account ACLs have no concept of referrers or rlistings
      + accounts have additional "admin" access level
      + account access levels are structured as admin > rw > ro, which seems more
        appropriate for how people access accounts, rather than reusing
        container ACLs' orthogonal read and write access
    
    In addition, the container ACL syntax is a bit arbitrary and highly custom,
    so instead of parsing additional custom syntax, I'd rather propose a next
    version and introduce a means for migration.  The V2 ACL syntax has the
    following benefits:
     * JSON is a well-known standard syntax with parsers in all languages
     * no artificial value restrictions (you can grant access to a user named
        ".rlistings" if you want)
     * forward and backward compatibility: you may have extraneous keys, but
        your attempt to parse the header won't raise an exception
    
    I've introduced hooks in parse_acl and format_acl which currently default
    to the old V1 syntax but tolerate the V2 syntax and can easily be flipped
    to default to V2.  I'm not changing the default or adding code to rewrite
    V1 ACLs to V2, because this patch has suffered a lot of scope creep already,
    but this seems like a sensible milestone in the migration.
    
    TempAuth Account ACL Implementation
    -----------------------------------
    
    As stated above, core Swift is responsible for privileging the
    X-Account-Access-Control header (making it only accessible to swift_owners),
    for translating it to -sysmeta-* headers to trigger persistence by the
    account server, and for including the header in the responses to requests
    by privileged users.  Core Swift puts no expectation on the *content* of
    this header.  Auth systems (including TempAuth) are responsible for
    defining the content of the header and taking action based on it.
    
    In addition to the changes described above, this patch defines a format
    to be used by TempAuth for these headers in the common.middleware.acl
    module, in the methods format_v2_acl() and parse_v2_acl().  This patch
    also teaches TempAuth to take action based on the header contents.  TempAuth
    now sets swift_owner=True if the user is on the Admin ACL, authorizes
    GET/HEAD/OPTIONS requests if the user is on any ACL, authorizes
    PUT/POST/DELETE requests if the user is on the admin or read-write ACL, etc.
    
    Note that the action of setting swift_owner=True triggers core Swift to
    add or strip the privileged headers from the responses.  Core Swift (not
    the auth system) is responsible for that.
    
    DocImpact: Documentation for the new ACL usage and format appears in
    summary form in doc/source/overview_auth.rst, and in more detail in
    swift/common/middleware/tempauth.py in the TempAuth class docstring.
    I leave it to the Swift doc team to determine whether more is needed.
    
    Change-Id: I836a99eaaa6bb0e92dc03e1ca46a474522e6e826

commit 9f11819f1645dbcdf9272dde8532acd157085b5b
Author: Dirk Mueller <dirk@dmllr.de>
Date:   Thu Jan 16 13:39:08 2014 +0100

Remove dependencies on pep8, pyflakes and flake8
    
    They should be determined by the hacking dependency
    implicitely.
    
    Change-Id: I8805f7b0ef9f2b1f59be342e9c6d032330707688

commit 92fc917b4b753f95f6c950037b2cb5daabf11706
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Jan 16 12:50:48 2014 -0800

Let swift-object-info skip etag verification
    
    Now swift-object-info has a "-n" option that will cause the etag
    verification to be skipped; on large objects, the etag verification
    takes the vast majority of the runtime, and sometimes you just want to
    know which account owns the 5 GB monstrosity without waiting around while
    its checksum is verified.
    
    Change-Id: Id284570633eb7b98046cdb948d7c37a152de1195

commit 8e1e67c02d3fe0b2d18d4c7e60c50e1c5679788d
Author: Fabien Boucher <fabien.boucher@enovance.com>
Date:   Tue Nov 26 12:14:15 2013 +0100

Fix container quota MW for handling a bad source path
    
    The copy source must be container/object.
    This patch avoids the server to return
    an internal server error when user provides
    a path without a container.
    
    Fixes: bug #1255049
    Change-Id: I1a85c98d9b3a78bad40b8ceba9088cf323042412

commit 86903eab2c0742dc6555376c4231a8c2fbddb1f9
Author: John Dickinson <me@not.mn>
Date:   Sat Dec 28 08:15:15 2013 -0800

Documentation of the logs formats used in Swift
    
    Change-Id: I402014627bab7113105a6a63c0b0ce1267735ca0

commit cfd9d055a66b975e9e1bd2e08cd38cfc906d3b15
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Dec 19 19:03:06 2013 -0800

Sanify handoff search depth with non-integer replica counts
    
    On GET, the proxy will go search the primary nodes plus some number of
    handoffs for the account/container/object before giving up and
    returning a 404. That number is, by default, twice the ring's replica
    count. This was fine if your ring had an integral number of replicas,
    but could lead to some slightly-odd behavior if you have fractional
    replicas.
    
    For example, imagine that you have 3.49 replicas in your object ring;
    perhaps you're migrating a cluster from 3 replicas to 4, and you're
    being smart and doing it a bit at a time.
    
    On an object GET that all the primary nodes 404ed, the proxy would
    then compute 2 * 3.49 = 6.98, round it up to 7, and go look at 7
    handoff nodes. This is sort of weird; the intent was to look at 6
    handoffs for objects with 3 replicas, and 8 handoffs for objects with
    4, but the effect is 7 for everybody.
    
    You also get little latency cliffs as you scale up replica counts. If,
    instead of 3.49, you had 3.51 replicas, then the proxy would look at 8
    handoff nodes in every case [ceil(2 * 3.51) = 8], so there'd be a
    small-but-noticeable jump in the time it takes to produce a 404.
    
    The fix is to compute the number of handoffs based on the number of
    primary nodes for the partition, not the ring's replica count. This
    gets rid of the little latency cliffs and makes the behavior more like
    what you get with integral replica counts.
    
    If your ring has an integral number of replicas, there's no behavior
    change here.
    
    Change-Id: I50538941e571135299fd6b86ecd9dc780cf649f5

Revision history for this message

Daisuke Morita (morita-daisuke) wrote on 2014-08-04:

#3

Is this bus also fixed by the patch https://github.com/openstack/swift/commit/946984cb84414c84fe38d4b94b73174960e023b8 ?

Revision history for this message

Daisuke Morita (morita-daisuke) wrote on 2014-08-04:

#4

typo: this bus -> this bug

Revision history for this message

Tim Burke (1-tim-z) wrote on 2017-07-21:

#5

Addressed in https://github.com/openstack/swift/commit/8e1e67c (so, swift 1.13.0). Even better, with COPY pulled out to middleware in https://github.com/openstack/swift/commit/46d61a4 (swift 2.8.0), quotas no longer need to know about COPY at all!

Changed in swift:
status:	In Progress → Fix Released

OpenStack Object Storage (swift)

Invalid X-Copy-From value and Container Quotas results in 500

Bug Description

Other bug subscribers

Remote bug watches