Bug #1375332 “Recon does not sum up every policy's amount” : Bugs : OpenStack Object Storage (swift)

Daisuke Morita (morita-daisuke) on 2014-09-29

Changed in swift:
assignee:	nobody → Daisuke Morita (morita-daisuke)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-30: Related fix proposed to swift (master)

#1

Related fix proposed to branch: master
Review: https://review.openstack.org/125179

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-17: Related fix merged to swift (master)

#2

Reviewed: https://review.openstack.org/125179
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=2792fe81a93dbaa95e58f14099db5e11dd8cde68
Submitter: Jenkins
Branch: master

commit 2792fe81a93dbaa95e58f14099db5e11dd8cde68
Author: Daisuke Morita <email address hidden>
Date: Tue Sep 30 11:06:08 2014 -0400

Show the sum of every policy's amount in /recon/async

    After the release of Swift ver. 2.0.0, some recon responses do not
    show each policy's information yet. To make things worse, some recon
    results only count on policy-0's score, therefore the total is not
    shown in the recon results.

    With this patch, async_pending count of recon results becomes
    policy-aware. Suppose a number of async_pending files for policy-0 is 2
    and a number for policy-1 is 3, recon sums up every policy's amount
    as follows.

$ curl http://<host>:<port>/recon/async
{"async_pending": 5} # It showed 2 before this commit

Related-Bug: 1375332
Change-Id: Ifc88b8c9e06b9f022a926a87ed807e938e1e0412

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-28: Related fix proposed to swift (master)

#3

Related fix proposed to branch: master
Review: https://review.openstack.org/137769

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-12-01: Related fix proposed to swift (feature/ec)

#4

Related fix proposed to branch: feature/ec
Review: https://review.openstack.org/138165

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-12-01: Related fix merged to swift (feature/ec)

#5

Download full text (15.6 KiB)

Reviewed: https://review.openstack.org/138165
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0d3ebf09b94b41782b2c2a6bbcf255bf1203eca0
Submitter: Jenkins
Branch: feature/ec

commit 977d7c14daa38ab9c9d79bbf8b92371024b93fc8
Author: John Dickinson <email address hidden>
Date: Wed Nov 26 14:19:08 2014 -0800

Fix tempfile bugs from commit 6978275

    Commit 6978275 changed xprofile middleware's usage of mktemp
    and moved to using tempfile. But it was clearly never tested,
    because the os.close() calls never worked. This patch updates
    that previous patch to use a context to open and close the file.

Change-Id: I40ee42e8539551fd8e4dfb353f50146ab40a7847

commit dec97fc3ba2c71884f1c098e7d9cd1f709f74958
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Nov 26 06:13:29 2014 +0000

Imported Translations from Transifex

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: Ibf319f7cc1b5036ad8031776cf2c6018fb8a0159

commit 01f6e860066640a2ba1406a23c93a72b34ec495e
Author: Clay Gerrard <email address hidden>
Date: Fri Nov 21 17:28:13 2014 -0800

Add Expected Failure for ssync with sys-meta

    Sysmeta included with an object PUT persists with the PUT data - if an
    internal operation such as POST-as-copy during partial failure, or ssync
    with fast-POST (not supported), causes that data to be lost then the
    associated sysmeta will also be lost.

    Since object sys-meta persistence in the face of a POST when the
    original .data is unavailable requires fast-POST with .meta files the
    probetest that validates object sys-meta persistence of a POST when the
    most up-to-date copy of the object with sys-meta is unavailable
    configures an InternalClient with object_post_as_copy = false.

    This non-default configuration option is not supported by ssync and
    results in a loss of sys-meta very similar to the object sys-meta
    failure you would see with object_post_as_copy = true when the COPY part
    of the POST is unable to retrieve the most recently written object with
    sys-meta.

    Until we can fix the default POST behavior to make metadata updates
    without stomping on newer data file timestamps we should expect object
    sys-meta to be "very very best possible but not really guaranteed
    effort".

Until we can fix ssync to replicate metadata updates without stomping on
newer data file timestamps we should expect this test to fail.

    When ssync replication of fast-POST metadata update is fixed this test
    will fail signaling that the expected failure cruft should be removed,
    but other parts of ssync replication will still work and some other bugs
    can be fixed while we wait.

Change-Id: Ifc5d49514de79b78f7715408e0fe0908357771d3

commit a8751ae557616cab1cafd98a338cad352526a262
Author: Cedric Dos Santos <email address hidden>
Date: Tue Nov 25 12:37:05 2014 +0100

Correct misspelled words

In some files I found misspelling words.

bin/swift-reconciler-enqueue#l26
prima...

Reviewed:  https://review.openstack.org/138165
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0d3ebf09b94b41782b2c2a6bbcf255bf1203eca0
Submitter: Jenkins
Branch:    feature/ec

commit 977d7c14daa38ab9c9d79bbf8b92371024b93fc8
Author: John Dickinson <me@not.mn>
Date:   Wed Nov 26 14:19:08 2014 -0800

Fix tempfile bugs from commit 6978275
    
    Commit 6978275 changed xprofile middleware's usage of mktemp
    and moved to using tempfile. But it was clearly never tested,
    because the os.close() calls never worked. This patch updates
    that previous patch to use a context to open and close the file.
    
    Change-Id: I40ee42e8539551fd8e4dfb353f50146ab40a7847

commit dec97fc3ba2c71884f1c098e7d9cd1f709f74958
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Wed Nov 26 06:13:29 2014 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: Ibf319f7cc1b5036ad8031776cf2c6018fb8a0159

commit 01f6e860066640a2ba1406a23c93a72b34ec495e
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Nov 21 17:28:13 2014 -0800

Add Expected Failure for ssync with sys-meta
    
    Sysmeta included with an object PUT persists with the PUT data - if an
    internal operation such as POST-as-copy during partial failure, or ssync
    with fast-POST (not supported), causes that data to be lost then the
    associated sysmeta will also be lost.
    
    Since object sys-meta persistence in the face of a POST when the
    original .data is unavailable requires fast-POST with .meta files the
    probetest that validates object sys-meta persistence of a POST when the
    most up-to-date copy of the object with sys-meta is unavailable
    configures an InternalClient with object_post_as_copy = false.
    
    This non-default configuration option is not supported by ssync and
    results in a loss of sys-meta very similar to the object sys-meta
    failure you would see with object_post_as_copy = true when the COPY part
    of the POST is unable to retrieve the most recently written object with
    sys-meta.
    
    Until we can fix the default POST behavior to make metadata updates
    without stomping on newer data file timestamps we should expect object
    sys-meta to be "very very best possible but not really guaranteed
    effort".
    
    Until we can fix ssync to replicate metadata updates without stomping on
    newer data file timestamps we should expect this test to fail.
    
    When ssync replication of fast-POST metadata update is fixed this test
    will fail signaling that the expected failure cruft should be removed,
    but other parts of ssync replication will still work and some other bugs
    can be fixed while we wait.
    
    Change-Id: Ifc5d49514de79b78f7715408e0fe0908357771d3

commit a8751ae557616cab1cafd98a338cad352526a262
Author: Cedric Dos Santos <cedric.dos.sant@gmail.com>
Date:   Tue Nov 25 12:37:05 2014 +0100

Correct misspelled words
    
    In some files I found misspelling words.
    
    bin/swift-reconciler-enqueue#l26
       primarly => primarily
    swift/account/backend.py#l309
       ommited => omitted
    swift/container/replicator.py#l158
       successfull => successful
    test/unit/account/test_backend.py#1450
       non_existant_policy_index => non_existent_policy_index
    test/unit/account/test_backend.py#1451
       'test-non-existant-policy'=> 'test-non-existent-policy'
    test/unit/account/test_backend.py#1453
       non_existant_policy_index => non_existent_policy_index
    
    Change-Id: I976236e3200a6fbdc20be464acff182b6cface81

commit 98de48d898f1419b0a0cfc273ec778e60331e623
Author: Shilla Saebi <shilla.saebi@gmail.com>
Date:   Sat Nov 22 15:38:48 2014 -0500

Fix typo in apache_deployment doc
    
    Change-Id: I42d76f544290dbda62633de90608d41caadac084

commit a1872b0498e1aa4182a4373c89beeaaaa219ea17
Author: Shilla Saebi <shilla.saebi@gmail.com>
Date:   Sat Nov 22 15:35:10 2014 -0500

Fix 2 typos in admin_guide file
    
    Change-Id: Ibf1e5dbf6ff4747c7f23f6638321ab41bba3021b

commit 0dc4b0a7b75237c09caffdac8c0dfd92bf8e3286
Author: Shilla Saebi <shilla.saebi@gmail.com>
Date:   Sat Nov 22 16:11:37 2014 -0500

Fix typos in overview_large_objects and versioning doc
    
    
    Change-Id: I1a919ad1b0298d5817f9eb2caf5e3bd7b3243c2c

commit 7a0c4d248257259612d3471ab42669ca9d90c573
Author: Takashi Kajinami <kajinamit@nttdata.co.jp>
Date:   Mon Nov 24 22:05:07 2014 +0900

Remove invalid connection checking in db_replicator
    
    Account/Container-replicator checks connection generation and timeout
    in HTTP REPLICATE Request in _repl_to_node, but it doesn't really checks
    connection but only construction of ReplConnection class.
    This patch removes that invalid checking.
    
    Change-Id: Ie6b4062123d998e69c15638b741e7d1ba8a08b62
    Closes-Bug: #1359018

commit 1c9bc0b522bed333b04a46ed7bd2c66a4eb89860
Author: Jay S. Bryant <jsbryant@us.ibm.com>
Date:   Thu Oct 2 14:10:04 2014 -0500

Handle os.listdir failures in object-updater
    
    While investigating bug 1375348 I discovered the problem
    reported there was not limited to the object-auditor.  The
    object-updater has similar bugs.
    
    This patch catches the unhandled exception that can be thrown
    by os.listdir if the self.devices directory is inaccessible.
    
    Change-Id: I6293b840916bb63cf9eebbc05068d9a3c871bdc3
    Related-bug: 1375348

commit 8cc075a8fb7561c736cb38d629f5b3d8ddb67497
Author: Jay S. Bryant <jsbryant@us.ibm.com>
Date:   Thu Nov 20 15:56:58 2014 -0600

mock out os.listdir to return a list
    
    os.listdir returns a list of items.  The test case had been
    written to return a single item which, though not really changing
    the result of the test, was not the best approach.
    
    This patch updates the test case to return a list instead of a single
    item.
    
    Change-Id: I793e0636440c0de0ca339c6592adec3e8b4ee1b4

commit fb353e1756df02622ea257acc987df4ccd094872
Author: John Dickinson <me@not.mn>
Date:   Thu Nov 20 10:22:27 2014 -0800

update AUTHORS
    
    Change-Id: I416e81b20a129377782f5d9298f8b8f5be079c27

commit 6c02adc33e3238f3fe0b75f2857503d1036f4737
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu Nov 20 06:11:14 2014 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Co-Authored-By: Pearl Yajing Tan <pearl.y.tan@seagate.com>
    
    Change-Id: Ifa3e292b8d5afbef8a99121b233e5ea596e672b7

commit 87d8626505c31511911facd5e1a1c3b3a65e8663
Author: Eohyung Lee <liquidnuker@gmail.com>
Date:   Thu Nov 20 11:38:49 2014 +0900

fix example typo
    
    5 * 1024 * 1024 = 5242880
    
    Change-Id: I0eeb6e2d9fbd79103cd8c658627344f73fed9498

commit ddf8b0594bb7e5ea9022982a7c5e15d9b261c22e
Author: Andreas Jaeger <aj@suse.de>
Date:   Wed Nov 19 09:11:55 2014 -0500

Fix translation setup
    
    Fix the output directory, it should be swift/locale.
    This fixes the importing of translations.
    
    Change-Id: I48311773c9d200c3b1739dc796618849416096ed

commit e0307f950bccde1337898e16087af726429e13f4
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Mon Nov 17 12:30:15 2014 -0800

Always use FakeMemcache for in-process tests
    
    Better isolation and consistency for in-process functests to always use
    the FakeMemcache.  If you want to test the real memcache you have real
    functional tests.
    
    Change-Id: Ic483f794e122130bd7694c9a5f9a2b1cd0b9a653

commit 6f9ca6122efac6c1c252a948cd5cc18c58c625ff
Author: Anne Gentle <anne@openstack.org>
Date:   Mon Nov 17 16:11:05 2014 -0600

Adds v1 API documentation to doc/source/api
    
    After discussion https://review.openstack.org/#/c/129384/ moving
    to the doc directory in swift repo.
    
    This lets us eliminate the object-api repo along with all the <service>-
    api repos and move content to audience-centric locations.
    
    Change-Id: Ia0d9973847f7409a02dcc1a0e19400a3c3ecdf32

commit 11a72a4a5084dbcb5539596c50793e45c5dac525
Author: Thiago da Silva <thiago@redhat.com>
Date:   Mon Nov 17 11:33:41 2014 -0500

move slo, dlo after tempauth in pipeline
    
    Noticed that slo and dlo middleware were placed before
    tempauth, they should be placed after
    
    DocImpact
    
    Change-Id: Ia931e2280125d846f248b23e219aebad14c66210
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit 2792fe81a93dbaa95e58f14099db5e11dd8cde68
Author: Daisuke Morita <morita.daisuke@lab.ntt.co.jp>
Date:   Tue Sep 30 11:06:08 2014 -0400

Show the sum of every policy's amount in /recon/async
    
    After the release of Swift ver. 2.0.0, some recon responses do not
    show each policy's information yet. To make things worse, some recon
    results only count on policy-0's score, therefore the total is not
    shown in the recon results.
    
    With this patch, async_pending count of recon results becomes
    policy-aware. Suppose a number of async_pending files for policy-0 is 2
    and a number for policy-1 is 3, recon sums up every policy's amount
    as follows.
    
    $ curl http://<host>:<port>/recon/async
    {"async_pending": 5} # It showed 2 before this commit
    
    Related-Bug: 1375332
    Change-Id: Ifc88b8c9e06b9f022a926a87ed807e938e1e0412

commit c9f824637845f342b6996058e0fea8338bd1305d
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Mon Aug 11 17:09:48 2014 +0100

Make in process functional tests use sample proxy-server.conf
    
    This patch was first motivated by noticing that the proxy
    server pipeline used for in process functional tests was
    out of date with respect to the pipeline in
    /etc/proxy-server.conf.sample. Rather than cut and paste
    the current pipeline into the in process setup, it seems
    like a better idea would be to have the in process tests
    always use the sample config.
    
    A further benefit is that in process functional tests will
    pick up changes to the sample config introduced by patches -
    previously test/functional/__init__.py would need to be
    manually modified to run in process functional tests
    on new middleware for example.
    
    Note: because the pipeline is now loaded using entry points,
    'python setup.py [develop|install]' will now be needed
    before running the tests.
    
    Obvious next steps would be to do the same for the backend
    servers, and to allow alternative config files and dir's
    to be specified, but this patch is the first step.
    
    Also drive-by fixes some typos in proxy-server.conf.sample
    
    Change-Id: If442bd7c2b1721ec92839c4490924ba33e1545d8

commit e429cd81be711f42441a08e34c077dcd7a97bed0
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Nov 13 16:40:05 2014 -0800

Make error limits survive a ring reload
    
    The proxy was storing the error count and last-error time in the
    ring's internal data, specifically in the device dictionaries. This
    works okay, but it means that whenever a ring changes, all the error
    stats reset.
    
    Now the error stats live in the proxy server object, so they survive a
    ring reload.
    
    Better yet, the error stats are now keyed off of the node's
    IP/port/device triple, so if you have the same device in two rings
    (like with multiple storage policies), then the error stats are
    combined. If the proxy server sees a 507 for an objec request in
    policy X, then that will now result in that particular object disk
    being error-limited for requests in policies Y and Z as well.
    
    Change-Id: Icc72b68b99f37367bb16d43688e7e45327e3e022

commit b98fe3b77b6b422e5e5978f6cf82a11fb87aedfc
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Nov 11 17:03:29 2014 -0800

Prefer X-Backend-Timestamp for X-Newest
    
    When a X-Backend-Timestamp is available it would generally preferred
    over a less specific value and sorts correctly against any X-Timestamp
    values anyway.
    
    Change-Id: I08b7eb37ab8bd6eb3afbb7dee44ed07a8c69b57e

commit 466403723c4c1fd575b1340e0f9214ee28f0aeb7
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon Nov 3 14:20:08 2014 -0800

Make resetswift customizable via environment
    
    Instead of recommending to edit resetswift to replace "/dev/sdb1" with
    "/srv/swift-disk", use an environment variable instead. This way I can
    set SAIO_BLOCK_DEVICE=/srv/swift-disk in my .bashrc, and then when I'm
    testing out changes to resetswift, I don't need to remember to edit
    the modified script, nor do I end up submitting changes with the wrong
    default in there.
    
    The variable defaults to /dev/sdb1, so if you use the script unmodified
    and don't set SAIO_BLOCK_DEVICE, nothing changes for you.
    
    Change-Id: I741a8c91c2c54a4f32bc391cd794ef4206402753

commit 331b14238effc9d1928e478bba86122f7e2525c1
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Nov 7 13:53:46 2014 -0800

Reject object names with Unicode surrogates
    
    Technically, you can't encode surrogates into UTF-8 at all, but Python
    2 lets you get away with it. Python 3 does not.
    
    We already have a check for surrogate pairs (commit 0080337), but not
    one for lone surrogates. This commit forbids object names with lone
    surrogates in them.
    
    The problem with surrogates is trivially reproducible:
    
        swift@saio:~$ python2.7
        Python 2.7.3 (default, Feb 27 2014, 19:58:35)
        [GCC 4.6.3] on linux2
        Type "help", "copyright", "credits" or "license" for more information.
        >>> b'\xed\xa0\xbc'.decode('utf-8')
        u'\ud83c'
        >>>
    
        swift@saio:~$ python3.3
        Python 3.3.5 (default, Aug  4 2014, 15:27:24)
        [GCC 4.6.3] on linux
        Type "help", "copyright", "credits" or "license" for more information.
        >>> b'\xed\xa0\xbc'.decode('utf-8')
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
        UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: invalid continuation byte
        >>>
    
    See also http://bugs.python.org/issue9133
    
    Change-Id: I7c31022e8a028c3cdf2ed1586349509d96cfded9

commit 2659888c921d09bd1dd23cda6ee2f158187d80e6
Author: Matthew Oliver <matt@oliver.net.au>
Date:   Fri Jun 13 19:12:31 2014 +1000

When a filesystem does't support xattr return a 507
    
    Currently when the object server tries to write an object's metadata
    to a filesystem that doesn't support xattr, it errors with a stacktrace
    and returns a 500 error back to the user with no information.
    
    This patch catches the resulting IOError when attempting to read or write
    the xattr metadata, logs the error nicely and then returns a 507 error
    back to the user.
    
    Seeing as this change is sending back a 507, it also catches and logs
    the out of disk space errors (ENOSPC and EDQUOT).
    
    Change-Id: I31932b57582817a0b3b58dd315a996bd0bcbc99b
    Closes-Bug: #966671

commit 0a5268c34caa25487c48380a1821e4deac178538
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Tue Sep 16 14:46:08 2014 +0000

Fix bug in swift-ring-builder list_parts
    
    The number of shown replicas in the partition list might differ from the
    actual number of replicas (as shown in the bugreport).
    
    This codes simply iterates for the builder._replica2part2dev and
    remembers the number of replicas for each partition.
    
    The code to find the partitions was moved to swift/common/ring/utils.py
    to make it easier to test, and a test to ensure the correct number of
    replicas is returned was added.
    
    Closes-Bug: 1370070
    Change-Id: Id6a3ed437bb86df2f43f8b0b79aa8ccb50bbe13e

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-12-03: Related fix proposed to swift (master)

#6

Related fix proposed to branch: master
Review: https://review.openstack.org/138697

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-13: Related fix merged to swift (master)

#7

Reviewed: https://review.openstack.org/137769
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=f8fa1a9234cd0c1d7466dd51d8528e2abd859c2b
Submitter: Jenkins
Branch: master

commit f8fa1a9234cd0c1d7466dd51d8528e2abd859c2b
Author: Daisuke Morita <email address hidden>
Date: Fri Nov 28 20:19:02 2014 +0900

Show each policy's information on quarantined files in recon

    After the release of Swift ver. 2.0.0, some recon responses do not
    show each policy's information yet. To make things worse, some recon
    results only count on policy-0's score, therefore the total is not
    shown in the recon results.

    This patch makes the count of quarantined files policy-aware for recon
    requests. Suppose a number of quarantined objects for policy-0 is 2
    and a number for policy-1 is 3, recon sums up every policy's amount
    and shows information for each policy as follows.

    $ curl http://<host>:<port>/recon/quarantined
    {"accounts": 0, "containers": 0, "objects": 5, "policies": {"0":
    {"objects": 2}, "1": {"objects": 3}}}

Moreover, this patch adds stats for each policy in CLI output.

    Change-Id: I07217c635f6fc4ea809ddbc3d859c4e81c4fde37
    Related-Bug: 1375327
    Related-Bug: 1375332

Christian Schwede (cschwede) on 2015-02-16

Changed in swift:
status:	New → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-21: Related fix proposed to swift (feature/ec)

#8

Related fix proposed to branch: feature/ec
Review: https://review.openstack.org/158087

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-22: Related fix merged to swift (feature/ec)

#9

Download full text (15.3 KiB)

Reviewed: https://review.openstack.org/158087
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=c84d50f43fdfbdc72ea9c0867769cfa7fdbb09c0
Submitter: Jenkins
Branch: feature/ec

commit db29ffc98384a9f61c2d4cc48bb2faf9f5dd0478
Author: Samuel Merritt <email address hidden>
Date: Fri Feb 20 11:04:24 2015 -0800

Make proxy_logging close the WSGI iterator

    PEP 333 says that the WSGI framework will call .close() on the
    iterator returned by a WSGI application once it's done, provided such
    a method exists. So, if our code wraps an iterator, then we have to
    call .close() on it once we're done with it. proxy_logging wasn't.

    Since WSGIContext gets it right, I looked at making proxy_logging use
    WSGIContext. However, WSGIContext is all about forcing the first chunk
    out of the iterator so that it can capture the final HTTP status and
    headers; it doesn't help if you want to look at every chunk.
    proxy_logging wants every chunk so it can count the bytes sent.

    This didn't hurt anything in Swift, but pconstantine was complaining
    in IRC that our failure to call .close() was goofing up some other
    middleware he had.

Change-Id: Ic6ea0795ccef6cda2b5c6737697ef7d58eac9ab4

commit 4ca08cc395e686265574366497a6869e94eebcb2
Author: Alistair Coles <email address hidden>
Date: Tue Feb 17 10:27:44 2015 +0000

Update guest VM OS recommendation in SAIO doc

The target development platform has changed to Ubuntu 14.04 [1].
This patch makes the suggested SAIO platform the same.

    Also, remove pointer to wiki page for other platform install
    instructions that either redirects back to this SAIO doc or
    to another wike page and then a dead link.

[1] I0a96bcf692bb240f3ab5aab7fefd294a07735a83

DocImpact

Change-Id: I9f96104b5437c1f1f28f924c048ef83cf03338f4

commit 7bc09dfdea0893a49e50005b22b426ae21c11d22
Author: Arnaud JOST <email address hidden>
Date: Wed Feb 18 11:56:11 2015 +0100

Add support of x-remove- headers for container-sync

    If the used tool to send header doesn't support empty headers (older versions
    of curl), x-remove can be used to remove metadata.
    sync-key and sync-to metadata, used by container-sync, can now be removed using
    x-remove headers.

Change-Id: I0edb4d5425a99d20a973aa4fceaf9af6c2ddecc0

commit 949804eda4a85c3c960b0baa090e16f1aa48e95e
Author: John Dickinson <email address hidden>
Date: Mon Feb 16 14:00:24 2015 -0800

update the getting started doc

Change-Id: I0a96bcf692bb240f3ab5aab7fefd294a07735a83

commit dd1a05f52765a6273906b5d6ce55f81e269bad12
Author: OpenStack Proposal Bot <email address hidden>
Date: Mon Feb 16 06:30:54 2015 +0000

Imported Translations from Transifex

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I013976c6192a8bff891c9050f829ae7a1e2fec59

commit 7acc2911296f48f10165282eee6bfe8e5c817a69
Author: John Dickinson <email address hidden>
Date: Sun Feb 15 17:14:31 2015 -0800

added swift_source to ratelimit info calls

Change-Id: I2b4cc...

Reviewed:  https://review.openstack.org/158087
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=c84d50f43fdfbdc72ea9c0867769cfa7fdbb09c0
Submitter: Jenkins
Branch:    feature/ec

commit db29ffc98384a9f61c2d4cc48bb2faf9f5dd0478
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Feb 20 11:04:24 2015 -0800

Make proxy_logging close the WSGI iterator
    
    PEP 333 says that the WSGI framework will call .close() on the
    iterator returned by a WSGI application once it's done, provided such
    a method exists. So, if our code wraps an iterator, then we have to
    call .close() on it once we're done with it. proxy_logging wasn't.
    
    Since WSGIContext gets it right, I looked at making proxy_logging use
    WSGIContext. However, WSGIContext is all about forcing the first chunk
    out of the iterator so that it can capture the final HTTP status and
    headers; it doesn't help if you want to look at every chunk.
    proxy_logging wants every chunk so it can count the bytes sent.
    
    This didn't hurt anything in Swift, but pconstantine was complaining
    in IRC that our failure to call .close() was goofing up some other
    middleware he had.
    
    Change-Id: Ic6ea0795ccef6cda2b5c6737697ef7d58eac9ab4

commit 4ca08cc395e686265574366497a6869e94eebcb2
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Tue Feb 17 10:27:44 2015 +0000

Update guest VM OS recommendation in SAIO doc
    
    The target development platform has changed to Ubuntu 14.04 [1].
    This patch makes the suggested SAIO platform the same.
    
    Also, remove pointer to wiki page for other platform install
    instructions that either redirects back to this SAIO doc or
    to another wike page and then a dead link.
    
    [1] I0a96bcf692bb240f3ab5aab7fefd294a07735a83
    
    DocImpact
    
    Change-Id: I9f96104b5437c1f1f28f924c048ef83cf03338f4

commit 7bc09dfdea0893a49e50005b22b426ae21c11d22
Author: Arnaud JOST <arnaud.jost@ovh.net>
Date:   Wed Feb 18 11:56:11 2015 +0100

Add support of x-remove- headers for container-sync
    
    If the used tool to send header doesn't support empty headers (older versions
    of curl), x-remove can be used to remove metadata.
    sync-key and sync-to metadata, used by container-sync, can now be removed using
    x-remove headers.
    
    Change-Id: I0edb4d5425a99d20a973aa4fceaf9af6c2ddecc0

commit 949804eda4a85c3c960b0baa090e16f1aa48e95e
Author: John Dickinson <me@not.mn>
Date:   Mon Feb 16 14:00:24 2015 -0800

update the getting started doc
    
    Change-Id: I0a96bcf692bb240f3ab5aab7fefd294a07735a83

commit dd1a05f52765a6273906b5d6ce55f81e269bad12
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Mon Feb 16 06:30:54 2015 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I013976c6192a8bff891c9050f829ae7a1e2fec59

commit 7acc2911296f48f10165282eee6bfe8e5c817a69
Author: John Dickinson <me@not.mn>
Date:   Sun Feb 15 17:14:31 2015 -0800

added swift_source to ratelimit info calls
    
    Change-Id: I2b4ccb809c0f4505098e261f546f53c27440dd43

commit 023e29fa1e5ea9e22b12c897ea6f944c537c6bf3
Author: Richard Hawkins <richard.hawkins@rackspace.com>
Date:   Thu Feb 12 16:39:48 2015 -0600

Add functional tests for container TempURLs
    
    Change-Id: I7f54594df1522d72dd1d13556be0b9d33e811b30

commit ca0fce8542645b9f7b934817eb4cab7835ec0d3c
Author: Leah Klearman <lklrmn@gmail.com>
Date:   Fri Feb 13 16:55:45 2015 -0800

more probe test refactoring
    
    * move get_to_final_state into ProbeTest
    * get rid of kill_servers
    * add replicators manager and updaters manager to ProbeTest
    
    (this is all going someplace, i promise)
    
    Change-Id: I8393a2ebc0d04051cae48cc3c49580f70818dbf2

commit 2655465d82b8b0acb29babf5d7520808860702cc
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Feb 13 13:11:40 2015 -0800

Promote some of the best developers I know to CORE Emeritus
    
    Change-Id: I4095bdbb2c137615bf9492e373a837daadea8920

commit 596042b9c778114dcfab1670e1ca40aa3a8c71ee
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu Feb 12 16:15:42 2015 -0800

Minor cleanup post efficent multi-region replication
    
    One log line had a typo, and I refactored the per object cleanup code out of
    update_deleted into the per object hashdir cleanup method.
    
    Change-Id: I19d03d0706a75bd8ec2fe327a1eb1b5ec36de6d2

commit b8e85a42ec18d234bbbe45d0c94f386610e4cd8a
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Fri Feb 13 06:10:44 2015 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I7655ef8e64b6700be9b8aaa6387c36de32a2d7b8

commit a4ffd1d1c6f8027d408c73076113ddafdc79ebc0
Author: Leah Klearman <lklrmn@gmail.com>
Date:   Thu Feb 12 20:32:35 2015 -0800

move test comments around
    
    should make later refactoring easier
    
    Change-Id: I7af399a14c8bc78fcfc438e4440d2f023c8aa5db

commit 82e5090848ecf2e5225acf1741e40c670ad3f121
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Thu Feb 12 21:30:37 2015 +0000

Fix ssync send_delete
    
    The ssync_sender send_delete method treats its
    timestamp argument as a string when in fact it is
    passed a Timestamp object. As a result the method
    always raises an exception and deletes are never
    replicated.
    
    This patch fixes bug and adds unit and probe tests
    to verify expected behavior.
    
    Closes-Bug: 1421425
    
    Change-Id: I664fb8d5dfea7362313037a67927ea90021c3f62

commit a9b5982d525d6b1ed369e780835b5b26a250ad95
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Feb 12 09:59:48 2015 -0800

Fix account-reaper
    
    As part of commit efb39a5, the account reaper grew a bind_port
    attribute, but it wasn't being converted to int, so naturally "6002"
    != 6002, and it wouldn't reap anything.
    
    The bind_port was only used for determining the local devices. Rather
    than fix the code to call int(), this commit removes the need for
    bind_port entirely by skipping the port check. If your rings have IPs,
    this is the same behavior as pre-efb39a5, and if your rings have
    hostnames, this still works.
    
    Change-Id: I7bd18e9952f7b9e0d7ce2dce230ee54c5e23709a

commit 2c1b5af06218fa936c8c9fb0180f22db212bf18a
Author: Leah Klearman <lklrmn@gmail.com>
Date:   Thu Feb 12 11:30:21 2015 -0800

refactor probe tests
    
    * refactor probe tests to use probe.common.ProbeTest
    * move reset_environment functionality to ProbeTest.setUp()
    * choose rings and policies that meet the criteria - raise SkipTest if
    nothing matches
    * replace all AssertionErrors in setup with SkipTest
    
    Change-Id: Id56c497d58083f5fd55f5283cdd346840df039d3

commit f5b533fb2f217a987d9bd7898b6baed31e9a1a31
Author: Richard Hawkins <richard.hawkins@rackspace.com>
Date:   Wed Feb 11 12:44:08 2015 -0600

Update TempURL docs to include containers
    
    Change-Id: Ifb7c26b23ec81e3bde96b6e3bad0e950c0ca9408

commit f5ab30574ea92c1457a30dccaedc1f1883fab64e
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Wed Feb 11 17:49:41 2015 +0000

Fix logger test coupling
    
    Two tests fail if test/unit/common/test_daemon.py is excluded
    from the test set:
    
    test.unit.common.test_utils.TestUtils.test_swift_log_formatter
    test.unit.common.test_utils.TestStatsdLoggingDelegation.test_thread_locals
    
    They fail because they assert that logger.txn_id is None,
    when previous tests cause txn_id to be set. This coupling is masked
    when test_daemon.py is included because it reloads the common/utils
    module, and is executed just before test.unit.common.test_utils.
    
    The coupling can be observed by running, for example:
    
    nosetests ./test/unit/common/middleware/test_except.py ./test/unit/common/test_utils.py
    or
    nosetests ./test/unit/proxy ./test/unit/common/test_utils.py
    
    The failing tests should reset logger thread_locals before making
    assertions. test_utils.reset_loggers() attempts this but is broken
    because it sets thread_local on the wrong object.
    
    Changes in this patch:
     - fix reset_loggers() to reset the LogAdapter thread local state
     - add a reset_logger_state decorator to call reset_loggers
       before and after a decorated method
     - use the decorator for increased isolation of tests that previously
       called reset_loggers only on exit
    
    Change-Id: If9aa781a2dd1929a47ef69322ec8c53263d47660

commit ca90be414e0e50fd83c09550c5e999bc93c5c35c
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed Feb 11 16:36:51 2015 -0800

Add developer tools section to associated projects
    
    vagrant-swift-all-in-one is being used and maintained by a number of swift
    developers, it has an open source license.
    
    The ansible playbook project serves a similar goal but it's based on a Fedora
    distribution and includes Swift-on-File support.
    
    Drive-by fix for the Swift-on-File link which has migrated to stackforge.
    
    Change-Id: Id7478d58adcead57cf56ac4e1d05c6556c8c9b7b

commit f5d509471b36dfe1a008523a3a8e1ec4e5518f02
Author: Daisuke Morita <morita.daisuke@lab.ntt.co.jp>
Date:   Thu Feb 12 05:12:56 2015 +0900

Logging a policy index when container PUT request conflicts
    
    This is a continuing work of patch afdbf73. This patch enables
    proxy-server to log a policy index when container PUT request
    conflicts with existing container's policy index.
    
    Change-Id: I6d40044c510632a0f61b817a9af2f6c13a721d39
    Implements: blueprint logging-policy-number

commit 69797efcbba2076eae944754e0a8585122374544
Author: David Goetz <dpgoetz@gmail.com>
Date:   Wed Feb 11 11:37:31 2015 -0800

small bug with account to account copy
    
    Change-Id: I7fdf432666f7640082e9839b9dc3d7af94a0715f

commit 843ce7e301ce06c6dc14e576a46b942bcb640c04
Author: Donagh McCabe <donagh.mccabe@hp.com>
Date:   Wed Feb 11 10:24:23 2015 +0000

API document update for container-level tempurl/formpost
    
    Tempurl and Formpost now support secret keys at the container
    level [1]. Adds these headers:
    
        X-Container-Meta-Temp-Url-Key
        X-Container-Meta-Temp-Url-Key-2
    
    DocImpact -- because headers need to be added to [2]
    
    [1] https://review.openstack.org/#/c/154293
    [2] http://developer.openstack.org/api-ref-objectstorage-v1.html
    
    Change-Id: I157cd4a540dcc4a9b45bc3d806df152ab56e2354

commit 489dd5ff5db0590f5232e5c309bf50ba9481f804
Author: Richard Hawkins <richard.hawkins@rackspace.com>
Date:   Mon Feb 9 17:51:01 2015 -0600

Add support for container TempURL Keys
    
    Change-Id: Ic22b0b84b657e6cac7e0062fa410eefb09bc0f4d
    Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>

commit 20ca279d747523f5cb942567515f85d03779eb8a
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Thu May 29 00:54:07 2014 -0700

Efficient Replication for Distributed Regions
    
    This change provides a efficient way of replication
    between regions of a global distributed cluster.
    
    This approach makes object-replicator to push replicas
    to a primary node in a remote region, then, to skip
    pushing them to next primary node in the region with
    expecting asynchronous replication.
    
    This implementation includes a couple of changes on
    ssync_sender to allow object-replicator to delete local
    handoff objects correctly. One is to return a list of existing
    objects in remote region. The list includes local paths of the
    objects which exist both on the local device and the remote device.
    The other is supporting existence check for specified objects.
    It requires the object list build by the first change. When
    the object list is given, ssync_sender does only missing_check
    based on the list. These changes are needed because current
    swift can not handle the existence check in object-level.
    
    Note that this feature will work partially (i.e. only when
    primary-to-primary) with rsync.
    
    Implements: blueprint efficient-replication
    Change-Id: I5d990444d7977f4127bb37f9256212c893438df1

commit a82bfb25ba180dbde700d2f9108acf1c85f5d61f
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Thu Jan 29 14:26:01 2015 +0000

Make container GET call authorize when account not found
    
    When an account was not found, ContainerController would
    return 404 unconditionally for a container GET or HEAD request,
    without checking that the request was authorized.
    
    This patch modifies the GETorHEAD method to first call any
    callback method registered under 'swift.authorize' in the
    request environ and prefer any response from that over the 404.
    
    Closes-Bug: 1415957
    
    Change-Id: I4f41fd9e445238e14af74b6208885d83698cc08d

commit 8607833fddba074823de58f4f4bd2707bf90fc9c
Author: Daisuke Morita <morita.daisuke@lab.ntt.co.jp>
Date:   Fri Dec 12 10:37:33 2014 +0900

Output account-reaper's logs for PolicyError
    
    By some operational mistakes, policy settings can be inconsistent among
    servers of swift cluster. Before this patch, if a policy setting of the
    server where account-reaper daemon is running is different from other
    servers, reap_object process failed without outputing error logs and
    count up remaining objects.
    
    Change-Id: Ic84fef7b49c77f0197f0bec4d41401686114d50e
    Related-Bug: 1375384

commit afdbf73f1285319003fecb0415a691dd9c9e9c98
Author: Daisuke Morita <morita.daisuke@lab.ntt.co.jp>
Date:   Thu Nov 27 18:42:49 2014 +0900

Output logs of policy index
    
    To make it easier for Swift operators to specify problematic devices,
    a policy index will be recorded in log files of proxy and storage servers
    for each user request which is related to storage policy.
    
    This patch simply adds 'storage_policy_index' field in a log format.
    If there is no specified policy index, '-' is output in this field.
    
    Extra fix: Doc about the log line of storage nodes now properly reflects
               'server_pid' field.
    
    DocImpact
    
    Change-Id: I7286ae85bcbcec73b5377dc115cbdb0f57d1b025
    Implements: blueprint logging-policy-number

commit f8fa1a9234cd0c1d7466dd51d8528e2abd859c2b
Author: Daisuke Morita <morita.daisuke@lab.ntt.co.jp>
Date:   Fri Nov 28 20:19:02 2014 +0900

Show each policy's information on quarantined files in recon
    
    After the release of Swift ver. 2.0.0, some recon responses do not
    show each policy's information yet. To make things worse, some recon
    results only count on policy-0's score, therefore the total is not
    shown in the recon results.
    
    This patch makes the count of quarantined files policy-aware for recon
    requests. Suppose a number of quarantined objects for policy-0 is 2
    and a number for policy-1 is 3, recon sums up every policy's amount
    and shows information for each policy as follows.
    
    $ curl http://<host>:<port>/recon/quarantined
    {"accounts": 0, "containers": 0, "objects": 5, "policies": {"0":
    {"objects": 2}, "1": {"objects": 3}}}
    
    Moreover, this patch adds stats for each policy in CLI output.
    
    Change-Id: I07217c635f6fc4ea809ddbc3d859c4e81c4fde37
    Related-Bug: 1375327
    Related-Bug: 1375332

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-23: Related fix proposed to swift (feature/crypto)

#10

Related fix proposed to branch: feature/crypto
Review: https://review.openstack.org/158370

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-23: Related fix merged to swift (feature/crypto)

#11

Download full text (17.0 KiB)

Reviewed: https://review.openstack.org/158370
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=24d3f51386965197c36d506228e34ca1c4100165
Submitter: Jenkins
Branch: feature/crypto

commit db29ffc98384a9f61c2d4cc48bb2faf9f5dd0478
Author: Samuel Merritt <email address hidden>
Date: Fri Feb 20 11:04:24 2015 -0800

Make proxy_logging close the WSGI iterator

    PEP 333 says that the WSGI framework will call .close() on the
    iterator returned by a WSGI application once it's done, provided such
    a method exists. So, if our code wraps an iterator, then we have to
    call .close() on it once we're done with it. proxy_logging wasn't.

    Since WSGIContext gets it right, I looked at making proxy_logging use
    WSGIContext. However, WSGIContext is all about forcing the first chunk
    out of the iterator so that it can capture the final HTTP status and
    headers; it doesn't help if you want to look at every chunk.
    proxy_logging wants every chunk so it can count the bytes sent.

    This didn't hurt anything in Swift, but pconstantine was complaining
    in IRC that our failure to call .close() was goofing up some other
    middleware he had.

Change-Id: Ic6ea0795ccef6cda2b5c6737697ef7d58eac9ab4

commit 4ca08cc395e686265574366497a6869e94eebcb2
Author: Alistair Coles <email address hidden>
Date: Tue Feb 17 10:27:44 2015 +0000

Update guest VM OS recommendation in SAIO doc

The target development platform has changed to Ubuntu 14.04 [1].
This patch makes the suggested SAIO platform the same.

    Also, remove pointer to wiki page for other platform install
    instructions that either redirects back to this SAIO doc or
    to another wike page and then a dead link.

[1] I0a96bcf692bb240f3ab5aab7fefd294a07735a83

DocImpact

Change-Id: I9f96104b5437c1f1f28f924c048ef83cf03338f4

commit 7bc09dfdea0893a49e50005b22b426ae21c11d22
Author: Arnaud JOST <email address hidden>
Date: Wed Feb 18 11:56:11 2015 +0100

Add support of x-remove- headers for container-sync

    If the used tool to send header doesn't support empty headers (older versions
    of curl), x-remove can be used to remove metadata.
    sync-key and sync-to metadata, used by container-sync, can now be removed using
    x-remove headers.

Change-Id: I0edb4d5425a99d20a973aa4fceaf9af6c2ddecc0

commit 949804eda4a85c3c960b0baa090e16f1aa48e95e
Author: John Dickinson <email address hidden>
Date: Mon Feb 16 14:00:24 2015 -0800

update the getting started doc

Change-Id: I0a96bcf692bb240f3ab5aab7fefd294a07735a83

commit dd1a05f52765a6273906b5d6ce55f81e269bad12
Author: OpenStack Proposal Bot <email address hidden>
Date: Mon Feb 16 06:30:54 2015 +0000

Imported Translations from Transifex

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I013976c6192a8bff891c9050f829ae7a1e2fec59

commit 7acc2911296f48f10165282eee6bfe8e5c817a69
Author: John Dickinson <email address hidden>
Date: Sun Feb 15 17:14:31 2015 -0800

added swift_source to ratelimit info calls

Change-Id: I2...

Reviewed:  https://review.openstack.org/158370
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=24d3f51386965197c36d506228e34ca1c4100165
Submitter: Jenkins
Branch:    feature/crypto