OpenStack Object Storage (swift)

swift-object-auditor keeps forking processes if /srv is not accessible

Bug #1375348 reported by Jay Bryant on 2014-09-29

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Object Storage (swift)	Fix Released	Undecided	Unassigned	OpenStack Object Storage (swift) 2.2.0 "juno"

Bug Description

We have discovered that the swift-object- auditor will just keep forking off new processes in the case that the /srv directory has the wrong permissions set. The result is that it acts like a fork bomb which is not desirable.

Sep 29 11:22:30 oc0644314035 object-auditor: Begin object audit "forever" mode (ZBF)
Sep 29 11:22:30 oc0644314035 object-auditor: Begin object audit "forever" mode (ALL)
Sep 29 11:22:30 oc0644314035 object-auditor: ERROR auditing: [Errno 13] Permission denied: '/srv/node': #012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/swift/obj/auditor.py", line 335, in run_forever#012 self.audit_loop(parent, zbo_fps, **kwargs)#012 File "/usr/lib/python2.6/site-packages/swift/obj/auditor.py", line 274, in audit_loop#012 zbf_pid = self.fork_child(zero_byte_fps=True, **kwargs)#012 File "/usr/lib/python2.6/site-packages/swift/obj/auditor.py", line 260, in fork_child#012 self.run_audit(**kwargs)#012 File "/usr/lib/python2.6/site-packages/swift/obj/auditor.py", line 249, in run_audit#012 worker.audit_all_objects(mode=mode, device_dirs=device_dirs)#012 File "/usr/lib/python2.6/site-packages/swift/obj/auditor.py", line 91, in audit_all_objects#012 for location in all_locs:#012 File "/usr/lib/python2.6/site-packages/swift/obj/diskfile.py", line 412, in object_audit_location_generator#012 device_dirs = listdir(devices)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 2400, in listdir#012 return os.listdir(path)#012OSError: [Errno 13] Permission denied: '/srv/node'
Sep 29 11:22:30 oc0644314035 object-auditor: ERROR auditing: [Errno 13] Permission denied: '/srv/node': #012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/swift/obj/auditor.py", line 335, in run_forever#012 self.audit_loop(parent, zbo_fps, **kwargs)#012 File "/usr/lib/python2.6/site-packages/swift/obj/auditor.py", line 278, in audit_loop#012 pids.append(self.fork_child(**kwargs))#012 File "/usr/lib/python2.6/site-packages/swift/obj/auditor.py", line 260, in fork_child#012 self.run_audit(**kwargs)#012 File "/usr/lib/python2.6/site-packages/swift/obj/auditor.py", line 249, in run_audit#012 worker.audit_all_objects(mode=mode, device_dirs=device_dirs)#012 File "/usr/lib/python2.6/site-packages/swift/obj/auditor.py", line 91, in audit_all_objects#012 for location in all_locs:#012 File "/usr/lib/python2.6/site-packages/swift/obj/diskfile.py", line 412, in object_audit_location_generator#012 device_dirs = listdir(devices)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 2400, in listdir#012 return os.listdir(path)#012OSError: [Errno 13] Permission denied: '/srv/node'
---------------------------------------------------------------------------
-bash-4.1# ps aux | grep swift | grep auditor
swift 23462 0.7 0.7 228040 14216 ? Ss 17:06 0:00 /usr/bin/python /usr/bin/swift-account-auditor /etc/swift/account-server.conf
swift 23524 0.9 0.7 228024 14228 ? Ss 17:07 0:00 /usr/bin/python /usr/bin/swift-container-auditor /etc/swift/container-server.conf
swift 23586 1.1 0.8 237788 16260 ? Ss 17:07 0:00 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift 23598 0.0 0.7 238144 14096 ? S 17:07 0:00 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift 23599 0.0 0.7 238144 14096 ? S 17:07 0:00 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
-bash-4.1# ps aux | grep swift | grep auditor | wc -l
5
-bash-4.1# ps aux | grep swift | grep auditor | wc -l
9
-bash-4.1# ps aux | grep swift | grep auditor | wc -l
9
-bash-4.1# ps aux | grep swift | grep auditor
swift 23462 0.4 0.7 228040 14216 ? Ss 17:06 0:00 /usr/bin/python /usr/bin/swift-account-auditor /etc/swift/account-server.conf
swift 23524 0.5 0.7 228024 14228 ? Ss 17:07 0:00 /usr/bin/python /usr/bin/swift-container-auditor /etc/swift/container-server.conf
swift 23586 0.5 0.8 237788 16260 ? Ss 17:07 0:00 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift 23598 0.0 0.7 238144 14276 ? S 17:07 0:00 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift 23599 0.0 0.7 238144 14244 ? S 17:07 0:00 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift 23676 0.0 0.7 238144 14044 ? S 17:07 0:00 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift 23677 0.0 0.7 238144 14044 ? S 17:07 0:00 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift 23678 0.0 0.7 238144 14052 ? S 17:07 0:00 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift 23679 0.0 0.7 238144 14052 ? S 17:07 0:00 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
-bash-4.1# ps aux | grep swift | grep auditor | wc -l
9
-bash-4.1# ps aux | grep swift | grep auditor | wc -l
9
-bash-4.1# ps aux | grep swift | grep auditor | wc -l
17
-bash-4.1#

I am working on narrowing down the source of the problem.

Tags:

Revision history for this message

Jay Bryant (jsbryant) wrote on 2014-09-29:

def fork_child(self, zero_byte_fps=False, **kwargs):
    """Child execution"""
    pid = os.fork()
    if pid:
        return pid
    else:
        signal.signal(signal.SIGTERM, signal.SIG_DFL)
        if zero_byte_fps:
            kwargs['zero_byte_fps'] = self.conf_zero_byte_fps
        self.run_audit(**kwargs)
        sys.exit()

I think the problem is here. If self.run_audit(**kwargs) returns an exception we never call sys.exit() so, eventually we end up with a bunch of zombie processes. I am working on finding a way of handling this that allows me to log the issue. Having a problem there at the moment.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-02: Related fix proposed to swift (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/125746

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-03: Fix merged to swift (master)

Reviewed: https://review.openstack.org/125197
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=301a96f664d58b4ccad8e3cbf5d5a889cc76790f
Submitter: Jenkins
Branch: master

commit 301a96f664d58b4ccad8e3cbf5d5a889cc76790f
Author: Jay S. Bryant <email address hidden>
Date: Tue Sep 30 15:08:59 2014 -0500

Ensure sys.exit called in fork_child after exception

    Currently, the fork_child() function in auditor.py does not
    handle the case where run_audit() encounters an exception
    properly.

    A simple case is where the /srv directory is set
    with permissions such that the 'swift' user cannot access it.
    Such a situation causes a os.listdir() to return an OSError
    exception. When this happens the fork_child() process does not
    run to completion and sys.exit() is not executed. The process
    that was forked off continues to run as a result. Execution goes
    back up to the audit_loop function which restarts the whole process. The
    end result is an increasing number of processes on the system
    until the parent is terminated. This can quickly exhaust the
    process descriptors on a system.

    This change wraps run_audit() in a try block and adds an
    exception handler that prints what exception was encountered.
    The sys.exit() was moved to a finally: block so that it will
    always be run, avoiding the creation of zombies.

Change-Id: I89d7cd27112445893852e62df857c3d5262c27b3
Closes-bug: 1375348

Changed in swift:
status:	New → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-03: Related fix proposed to swift (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/125864

Thierry Carrez (ttx) on 2014-10-06

Changed in swift:
milestone:	none → 2.2.0-rc1
status:	Fix Committed → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-06: Fix proposed to swift (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/126371

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-07: Fix proposed to swift (feature/ec)

Fix proposed to branch: feature/ec
Review: https://review.openstack.org/126595

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-07: Fix merged to swift (feature/ec)

Download full text (11.3 KiB)

Reviewed: https://review.openstack.org/126595
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=06800cbe446ce4c937a57b69517b55c3bba9b6e1
Submitter: Jenkins
Branch: feature/ec

commit 7528f2b22169e90fe8ddd19b7ef7d46ecff5d231
Author: Christian Schwede <email address hidden>
Date: Mon Oct 6 10:01:03 2014 +0000

Fix minor typo

Fixes minor typo in one method and adds missing parameter in other
method. Only checked swift/container/reconciler.py for now.

Change-Id: I5c648010f09b6e4b1fb0380bc97b266e680602f8

commit 94fd95ba30c72fbcb03367aaa8da407a408948d5
Author: OpenStack Proposal Bot <email address hidden>
Date: Sat Oct 4 06:07:47 2014 +0000

Imported Translations from Transifex

Change-Id: I31b5e6b0f2922150902e1bfa52144302ee0c7a8e

commit d6a827792619f3343af07fc2519f4253fbdc67f7
Author: John Dickinson <email address hidden>
Date: Fri Oct 3 10:17:00 2014 -0400

updated AUTHORS and CHANGELOG for 2.2.0

Change-Id: I6c0bc1570f6a48439de5a029a86f1b582f30f8a6

commit 5b2c27a5874c2b5b0a333e4955b03544f6a8119f
Author: Richard (Rick) Hawkins <email address hidden>
Date: Wed Oct 1 09:37:47 2014 -0400

Fix metadata overall limits bug

    Currently metadata limits are checked on a per request basis. If
    multiple requests are sent within the per request limits, it is
    possible to exceed the overall limits. This patch adds an overall
    metadata check to ensure that multiple requests to add metadata to
    an account/container will check overall limits before adding
    the additional metadata.

Change-Id: Ib9401a4ee05a9cb737939541bd9b84e8dc239c70
Closes-Bug: 1365350

commit 301a96f664d58b4ccad8e3cbf5d5a889cc76790f
Author: Jay S. Bryant <email address hidden>
Date: Tue Sep 30 15:08:59 2014 -0500

Ensure sys.exit called in fork_child after exception

    Currently, the fork_child() function in auditor.py does not
    handle the case where run_audit() encounters an exception
    properly.

Change-Id: I89d7cd27112445893852e62df857c3d5262c27b3
Closes-bug: 1375348

commit 6d49cc3092168de6d22378557b2c37ea4063beeb
Author: Samuel Merritt <email address hidden>
Date: Thu Oct 2 17:14:58 2014 -0400

Fix ring-builder crash.

If you adjust ...

Reviewed:  https://review.openstack.org/126595
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=06800cbe446ce4c937a57b69517b55c3bba9b6e1
Submitter: Jenkins
Branch:    feature/ec

commit 7528f2b22169e90fe8ddd19b7ef7d46ecff5d231
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Mon Oct 6 10:01:03 2014 +0000

Fix minor typo
    
    Fixes minor typo in one method and adds missing parameter in other
    method. Only checked swift/container/reconciler.py for now.
    
    Change-Id: I5c648010f09b6e4b1fb0380bc97b266e680602f8

commit 94fd95ba30c72fbcb03367aaa8da407a408948d5
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Sat Oct 4 06:07:47 2014 +0000

Imported Translations from Transifex
    
    Change-Id: I31b5e6b0f2922150902e1bfa52144302ee0c7a8e

commit d6a827792619f3343af07fc2519f4253fbdc67f7
Author: John Dickinson <me@not.mn>
Date:   Fri Oct 3 10:17:00 2014 -0400

updated AUTHORS and CHANGELOG for 2.2.0
    
    Change-Id: I6c0bc1570f6a48439de5a029a86f1b582f30f8a6

commit 5b2c27a5874c2b5b0a333e4955b03544f6a8119f
Author: Richard (Rick) Hawkins <richard.hawkins@rackspace.com>
Date:   Wed Oct 1 09:37:47 2014 -0400

Fix metadata overall limits bug
    
    Currently metadata limits are checked on a per request basis. If
    multiple requests are sent within the per request limits, it is
    possible to exceed the overall limits.  This patch adds an overall
    metadata check to ensure that multiple requests to add metadata to
    an account/container will check overall limits before adding
    the additional metadata.
    
    Change-Id: Ib9401a4ee05a9cb737939541bd9b84e8dc239c70
    Closes-Bug: 1365350

commit 301a96f664d58b4ccad8e3cbf5d5a889cc76790f
Author: Jay S. Bryant <jsbryant@us.ibm.com>
Date:   Tue Sep 30 15:08:59 2014 -0500

Ensure sys.exit called in fork_child after exception
    
    Currently, the fork_child() function in auditor.py does not
    handle the case where run_audit() encounters an exception
    properly.
    
    A simple case is where the /srv directory is set
    with permissions such that the 'swift' user cannot access it.
    Such a situation causes a os.listdir() to return an OSError
    exception.  When this happens the fork_child() process does not
    run to completion and sys.exit() is not executed.  The process
    that was forked off continues to run as a result.  Execution goes
    back up to the audit_loop function which restarts the whole process.  The
    end result is an increasing number of processes on the system
    until the parent is terminated.  This can quickly exhaust the
    process descriptors on a system.
    
    This change wraps run_audit() in a try block and adds an
    exception handler that prints what exception was encountered.
    The sys.exit() was moved to a finally: block so that it will
    always be run, avoiding the creation of zombies.
    
    Change-Id: I89d7cd27112445893852e62df857c3d5262c27b3
    Closes-bug: 1375348

commit 6d49cc3092168de6d22378557b2c37ea4063beeb
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Oct 2 17:14:58 2014 -0400

Fix ring-builder crash.
    
    If you adjust the replica count after adding devices but before the
    first rebalance, the ring builder would crash.
    
    To reproduce:
    
        $ swift-ring-builder crashy.builder create 8 3 1
        $ swift-ring-builder crashy.builder add r1z1-10.1.1.1:6000/sda1 1
        $ swift-ring-builder crashy.builder set_replicas 5
        $ swift-ring-builder crashy.builder rebalance
    
    Change-Id: Id91cf8680961ccf6e3db153577e99cad545bee9d

commit 702d88bc89f56b96f9e47dc0923f66cbc7cbd5f6
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Wed Oct 1 14:03:51 2014 -0400

fix a comment
    
    Change-Id: I6310390f756bc0d326e56fceef32b630f0fe1ca1

commit 64aa8062bc96799993deafbdf184adb72c509f20
Author: YummyBian <yummy.bian@gmail.com>
Date:   Sat Sep 27 23:49:38 2014 +0800

Some statements are evaluated twice in the setUp of the
    TestObjectReplicator
    
    Remove the duplicated statements.
    
    Closes-Bug: #1374783
    
    Change-Id: If2b55e864fea497d7a7b218adf11eb7749c27765

commit 20e9ad538b6ba90674a75310d9cc184162ba9398
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Sun Sep 14 20:48:32 2014 +0000

Limit partition movement when adding a new tier
    
    When adding a new tier (region, zone, node, device) to an existing,
    already balanced ring all existing partitions in the existing tiers of
    the same level are gathered for reassigning, even when there is
    not enough space in the new tier. This will create a lot of unnecessary
    replication traffic in the backend network.
    
    For example, when only one region exists in the ring and a new region is
    added, all existing parts are selected to reassign, even when the new
    region has a total weight of 0. Same for zones, nodes and devices.
    
    This patch limits the number of partitions that are choosen to reassign
    by checking for devices on other tiers that are asking for more
    partitions.
    
    Failed devices are not considered when applying the limit.
    
    Co-Authored By: Florent Flament <florent.flament-ext@cloudwatt.com>
    Change-Id: I6178452e47492da4677a8ffe4fb24917b5968cd9
    Closes-Bug: 1367826

commit 2ce6341b527e81a43513adef6f5c54b4e09e27a4
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Mon Sep 29 14:49:42 2014 -0600

Fix up the return value of launch()
    
    This is "obviously" the right thing to do, except of course it's
    pure sugar: the return value is not used anywhere. Except if someone
    has a script that imports the whole thing somehow and then does
    isinstance(dict). Because that is so much easier than submitting
    a patch, I can imagine someone, somewhere doing that.
    
    The fix came in a patch by dfg, see review 121851.
    
    Change-Id: I39ddf70dc2d027b7db6e31097400248dc66eb137

commit e2285e46db11dd59edbe076540ac0db24ef6f3f2
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Sun Sep 28 06:08:55 2014 +0000

Imported Translations from Transifex
    
    Change-Id: I59b68ecb6f07e2b71e4655699d0165007d764718

commit 041cf67c36988e3aee0eab0e597d78cec8688df0
Author: YummyBian <yummy.bian@gmail.com>
Date:   Sat Sep 27 23:10:48 2014 +0800

Incorrect variable name in the diskfile.py
    
    You should use the suffix variable instead of the hash_ variable when
    updates the hashes dictionary in the get_hashes().
    
    Fixes Bug1374777
    
    Change-Id: If9a7ab5b6d271385e4fdc79ff9c216047a2f4d7e

commit 0f93fff46ad098a360f719a762dde4d17e852fbc
Author: Keshava Bharadwaj <kb.sankethi@gmail.com>
Date:   Thu Sep 25 00:48:49 2014 +0530

Fixes unit tests to clean up temporary directories
    
    This patch fixes the unit tests to remove the temporary directories
    created during run of unit tests. Some of unit tests did not tear down
    correctly, whatever it had set it up for running. This would over period
    of time bloat up the tmp directory. As on date, there were around 49 tmp
    directories left uncleared per round of unit tests. This patch fixes it.
    
    Change-Id: If591375ca9cc87d52c7c9c6dc16c9fb4b49e99fc

commit 38ba5790fb527967c2fcbaf094e76a73f4b94d38
Author: Keshava Bharadwaj <kb.sankethi@gmail.com>
Date:   Thu Sep 18 00:45:35 2014 +0530

Provides proper error handling on builder unpickle
    
    This patch provides the necessary error handling while unpickling
    a builder file. Earlier if a builder file is empty/invalid/corrupted,
    the stacktrace was shown to user with an exit code of 1. This fixes it
    to show a user-friendly message and also returns the exit code of 2,
    indicating there was a failure.
    
    Change-Id: I51eb24702c422299629f8053d4591dd10f5863f8
    Closes-Bug: #1370680

commit e567722c4ea125f4607093112396e16ae6c6dd42
Author: John Dickinson <me@not.mn>
Date:   Mon Sep 22 09:46:34 2014 -0700

updated hacking rules
    
    1) Added comment for H231, which we were already enforcing. H231
    is for Python 3.x compatible except statements.
    
    2) Added check for H201, which we were enforcing in reviews
    but waiting on hacking checks to be updated. H201 is for bare
    except statements, and the update in upstream hacking is to
    support the "  # noqa" flag on it.
    
    The H201 check catches some existing bare excepts that are fixed.
    
    Change-Id: I68638aa9ea925ef62f9035a426548c2c804911a8

commit 781a1620048edbbc057b15c94da390847b34d9df
Author: Michael Barton <mike@weirdlooking.com>
Date:   Thu Sep 25 16:29:57 2014 +0000

use get_container_info in ratelimit
    
    Everyone else cooperates to store the container_info cache in the env the
    first time it's requested.  This should save a duplicate memcache hit on
    requests that ratelimiting looks at.
    
    Change-Id: Ic6411d4619db6b53fafe9fdbf1d0a370d1258c38

commit 23c2d95e98ee5edf317da55e3ee4c303a18991b6
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Tue Sep 23 12:32:42 2014 -0700

Remove unused option from ObjectReplicator.__init__
    
    disk_chunk_size is unused by the ObjectReplicator class. Note that it
    *is* used by the object replicator process; it's just that
    DiskFileManager is the one that pulls it out of the conf and pays
    attention to it. Keeping it as an attribute on ObjectReplicator is
    unnecessary.
    
    Change-Id: I1eeef7b33873b4c8bb269ca02dcb067098b6fded

commit 6354e2da57a8d487caf3605d4005134f584cf935
Author: Gerry Drudy <gerry.drudy@hp.com>
Date:   Mon Sep 15 11:52:14 2014 +0100

direct_client not passing args between some functions
    
    The call to _get_direct_account_container in direct_get_account
    has several of its args =None instead of set to the value passed
    to direct_get_account.
    
    The same applies to _get_direct_account_container in
    direct_get_container.
    
    The direct_get_container is only called by the account-reaper
    and this bug will have limited impact on it. The marker,
    maintained in reap_container, is ignored by direct_get_container.
    This is not as bad as it sounds, if the account-reaper successfully
    deletes the first 10K objects, assuming the container has > 10K
    objects, the next call to direct_get_container will in fact return
    the next 10K objects even though it sets marker=None (assuming the
    first 10K objects were successfully deleted).
    
    This patch also updates test_direct_get_account and
    test_direct_get_container to ensure the appropriate
    args are included in the connection query_string.
    
    Closes-Bug: #1369558
    Change-Id: If1c8aa1240d38354ebc9b1ebca92dc1c8c36cb5f

commit c767ec9a37faa4414702aa9e39231afe0ba3b097
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed Jul 9 16:04:20 2014 -0700

Let eventlet.wsgi.server log tracebacks when eventlet_debug is enabled
    
    The "logging" available in eventlet.wsgi.server/BaseHTTPServer doesn't
    generally suite our needs, so it should be bypassed using a NullLogger in
    production.  But in development it can be useful if tracebacks generated from
    inside eventlet.wsgi (say a NameError in DiskFile.__iter__) end up in logs.
    Since we already have eventlet_debug parsed inside of run_server we can skip
    the NullLogger bypass and let stuff blast out to STDERR when configured for
    development/debug logging.
    
    Change-Id: I20a9e82c7fed8948bf649f1f8571b4145fca201d

Thierry Carrez (ttx) on 2014-10-16

Changed in swift:
milestone:	2.2.0-rc1 → 2.2.0

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-28: Fix merged to swift (stable/icehouse)

Reviewed: https://review.openstack.org/126371
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=fbe00380b0617476d84d2fde394c6ad347689aac
Submitter: Jenkins
Branch: stable/icehouse

commit fbe00380b0617476d84d2fde394c6ad347689aac
Author: Jay S. Bryant <email address hidden>
Date: Tue Sep 30 15:08:59 2014 -0500

Ensure sys.exit called in fork_child after exception

    Currently, the fork_child() function in auditor.py does not
    handle the case where run_audit() encounters an exception
    properly.

    Change-Id: I89d7cd27112445893852e62df857c3d5262c27b3
    Closes-bug: 1375348
    (cherry picked from commit 301a96f664d58b4ccad8e3cbf5d5a889cc76790f)

tags:

added: in-stable-icehouse

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-07: Related fix merged to swift (master)

Reviewed: https://review.openstack.org/125864
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=d895cea07f4f51b7d3e683344729a08e76f85f51
Submitter: Jenkins
Branch: master

commit d895cea07f4f51b7d3e683344729a08e76f85f51
Author: Jay S. Bryant <email address hidden>
Date: Thu Oct 2 21:13:30 2014 -0500

Handle os.listdir failures in container-updater

    While investigating bug 1375348 I discovered the problem
    reported there was not limited to the object-auditor. The
    container-updated has similar bugs.

This patch catches the unhandled exception that can be thrown by
os.listdir.

Change-Id: I7eed122bf6b663e6e7894ace136b6f4653db4985
Related-bug: 1375348

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-11: Related fix proposed to swift (feature/ec)

#10

Related fix proposed to branch: feature/ec
Review: https://review.openstack.org/133750

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-11: Related fix merged to swift (feature/ec)

#11

Download full text (9.6 KiB)

Reviewed: https://review.openstack.org/133750
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=01b77740103e0dbe69b1d25ac6bd51f12310cae6
Submitter: Jenkins
Branch: feature/ec

commit fdcd20f2b6496a9e857cb47ae8907938033be9df
Author: John Dickinson <email address hidden>
Date: Fri Nov 7 10:34:51 2014 +0100

added docs on specs workflow to CONTRIBUTING.md

Change-Id: Id83d1da2a7a594a07fc5332b918539b3728e101b

commit ecc946b4ffb09ca0a94998ef54a7af7d4c572aff
Author: Christian Schwede <email address hidden>
Date: Thu Nov 6 15:44:29 2014 +0100

Rename Swiftbrowser in associated projects

Let's use the full project name to avoid confusion with the recently added
Swiftbrowser based on AngularJS.

Change-Id: Ib07338268a1593bc2882908b49c1fb4a130ff43d

commit dff981a03eac34a4c776cfd9a1528f1c3824f29b
Author: Martin Geisler <email address hidden>
Date: Thu Nov 6 14:52:04 2014 +0100

Add Swift Browser as an associated project

This is a JavaScript based browser for Swift.

Change-Id: I2e304d4a0623c715f8712a358fef5067abc8935b

commit f9bed74d1bba6a512becd057c3139c54c176c226
Author: Clay Gerrard <email address hidden>
Date: Wed Oct 29 15:59:45 2014 -0700

Return 403 on unauthorized upload when over account quota

    If you try an unauthorized upload into a container that is over quota you get
    a 403 instead of a 413, but if you try to unauthorized upload when an
    *account* is over quota you can see the 413 even though the upload would have
    been rejected by the authorize callback. By wrapping the authorize callback
    associated with the incoming request we can make sure to only return our 413
    when the request would have been authorized otherwise.

Drive by doc fixes thanks to acoles:

     * State that container_quotas should be after auth middleware in
       the class doc string.
     * Add note to proxy-server.conf.sample that account_quotas should
       be after auth middleware.

The equivalent statements are already in place for each quota
middleware.

Doc-Impact

Closes-Bug: #1387415
Change-Id: I2a88b3ec79d35bfdd73ea6ad64e376b7c7af4ea6

commit cbc52a7a4ea32f7d84f54948aa9ebb5decd26813
Author: Christian Schwede <email address hidden>
Date: Thu Oct 23 16:23:20 2014 +0200

Return verbose message if account quota exceeded

This message is already used in the container quota middleware, so let's use it
in the account middleware too.

Change-Id: I136fe6102c28cc8ccc021555c42ec7b0be716444
Closes-Bug: 1381875

commit 83030b921dd83a84a2e966c88156e64d30fb9c24
Author: Christian Schwede <email address hidden>
Date: Wed Oct 29 10:34:53 2014 +0000

Update admin guide on handling drive failures

    Simply replacing a failed disk requires a very long time if the ring is not
    changed, because all data will be replicated to a single new disk. This extends
    the time to recover from missing replicas, and becomes even more important with
    bigger disks.

This patch updates the doc to include a faster alternative by setting the weight
of a fail...

Reviewed:  https://review.openstack.org/133750
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=01b77740103e0dbe69b1d25ac6bd51f12310cae6
Submitter: Jenkins
Branch:    feature/ec

commit fdcd20f2b6496a9e857cb47ae8907938033be9df
Author: John Dickinson <me@not.mn>
Date:   Fri Nov 7 10:34:51 2014 +0100

added docs on specs workflow to CONTRIBUTING.md
    
    Change-Id: Id83d1da2a7a594a07fc5332b918539b3728e101b

commit ecc946b4ffb09ca0a94998ef54a7af7d4c572aff
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Thu Nov 6 15:44:29 2014 +0100

Rename Swiftbrowser in associated projects
    
    Let's use the full project name to avoid confusion with the recently added
    Swiftbrowser based on AngularJS.
    
    Change-Id: Ib07338268a1593bc2882908b49c1fb4a130ff43d

commit dff981a03eac34a4c776cfd9a1528f1c3824f29b
Author: Martin Geisler <martin@geisler.net>
Date:   Thu Nov 6 14:52:04 2014 +0100

Add Swift Browser as an associated project
    
    This is a JavaScript based browser for Swift.
    
    Change-Id: I2e304d4a0623c715f8712a358fef5067abc8935b

commit f9bed74d1bba6a512becd057c3139c54c176c226
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed Oct 29 15:59:45 2014 -0700

Return 403 on unauthorized upload when over account quota
    
    If you try an unauthorized upload into a container that is over quota you get
    a 403 instead of a 413, but if you try to unauthorized upload when an
    *account* is over quota you can see the 413 even though the upload would have
    been rejected by the authorize callback.  By wrapping the authorize callback
    associated with the incoming request we can make sure to only return our 413
    when the request would have been authorized otherwise.
    
    Drive by doc fixes thanks to acoles:
    
     * State that container_quotas should be after auth middleware in
       the class doc string.
     * Add note to proxy-server.conf.sample that account_quotas should
       be after auth middleware.
    
    The equivalent statements are already in place for each quota
    middleware.
    
    Doc-Impact
    
    Closes-Bug: #1387415
    Change-Id: I2a88b3ec79d35bfdd73ea6ad64e376b7c7af4ea6

commit cbc52a7a4ea32f7d84f54948aa9ebb5decd26813
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Thu Oct 23 16:23:20 2014 +0200

Return verbose message if account quota exceeded
    
    This message is already used in the container quota middleware, so let's use it
    in the account middleware too.
    
    Change-Id: I136fe6102c28cc8ccc021555c42ec7b0be716444
    Closes-Bug: 1381875

commit 83030b921dd83a84a2e966c88156e64d30fb9c24
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Wed Oct 29 10:34:53 2014 +0000

Update admin guide on handling drive failures
    
    Simply replacing a failed disk requires a very long time if the ring is not
    changed, because all data will be replicated to a single new disk. This extends
    the time to recover from missing replicas, and becomes even more important with
    bigger disks.
    
    This patch updates the doc to include a faster alternative by setting the weight
    of a failed disk to 0.  In this case the partitions from the failed disk are
    distributed and replicated to the remaining disks in the cluster, and because
    each disk gets only a fraction of the partitions it's also much faster.
    
    Change-Id: I16617756359771ad89ca5d4690b58a014f481d9b

commit d895cea07f4f51b7d3e683344729a08e76f85f51
Author: Jay S. Bryant <jsbryant@us.ibm.com>
Date:   Thu Oct 2 21:13:30 2014 -0500

Handle os.listdir failures in container-updater
    
    While investigating bug 1375348 I discovered the problem
    reported there was not limited to the object-auditor.  The
    container-updated has similar bugs.
    
    This patch catches the unhandled exception that can be thrown by
    os.listdir.
    
    Change-Id: I7eed122bf6b663e6e7894ace136b6f4653db4985
    Related-bug: 1375348

commit 0080337897c5b8ebaba392da36303e7df69262a8
Author: Michael Barton <mike@weirdlooking.com>
Date:   Mon Oct 27 16:29:07 2014 +0000

reject problematic object names
    
    We had this problem:
    
        >> : x = '\xed\xa0\xbc\xed\xbc\xb8'
        >> : x == x.decode('utf-8').encode('utf-8')
        << : False
    
    That str contains two utf-8 codepoints, which I guess python is normalizing
    into one unicode character, which it then encodes as one utf-8 codepoint.
    Like this:
    
        >> : u'\ud83c\udf38'
        << : u'\U0001f338'
    
    I don't entirely understand that, but having a different byte representation
    after round-tripping through unicode causes problems with replication and
    listings.
    
    This patch just rejects anything that doesn't re-encode to the same thing.
    If someone smarter wants to do something different, please speak up.
    
    Change-Id: I9ac48ac2693e4121be6585c6e4f5d0079e9bb3e4

commit dd3564a58773052c472639d49ec39d1ab04e33aa
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Oct 24 15:34:45 2014 -0700

Mention Python 2.7 in Getting Started guide
    
    Change-Id: I2992d1227f294d4cb1c6a8676137df44e231bd80

commit 047215afeb4a493d8ac29b67e453996c09a1c832
Author: Gregory Haynes <greg@greghaynes.net>
Date:   Fri Oct 24 15:22:29 2014 -0700

Update Getting Started requirements versions
    
    These docs currently say we target ubuntu 10.04 and an eventlet version
    that our requirements file does not allow. Update these versions.
    
    Change-Id: I052b6561f88ec90f865454e426032f1baf4586c0

commit f21e0255fc286a146e7a9da53d8be8e0867cc365
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Wed Oct 22 09:28:43 2014 +0100

Change default keystone auth version to 3 in test/sample.conf
    
    The functional tests devstack gate job currently skips tests
    that require keystone v3 API. In conjunction with changes to
    devstack [0] this patch will cause the gate job to use keystone
    v3 API and stop skipping those tests.
    
    Devstack sets auth version in its /etc/swift/test.conf by
    uncommenting the value found in test/sample.conf, so the change
    in this patch is sufficient to cause the v3 API to be used.
    
    The devstack patch [0] sets the correct auth url for the v3 API
    and creates a fourth test account that is required for the
    currently skipped tests to run.
    
    [0] change id I557de01bf196075f2f3adcdf4dd1b43756d8a0ae
    
    Change-Id: I44e159b939f5e3fc3780612823d12262013ea793

commit 556568b1c38dd0daa9eb6a9de34f54272e4c6465
Author: Michael Barton <mike@weirdlooking.com>
Date:   Thu Oct 16 01:56:48 2014 +0000

use replication_ip in ssync
    
    Update ssync_sender to use replication_ip and replication_port from the ring.
    
    Those attributes are supposed to allow for a separate replication network, and
    are used by rsync replication.
    
    Change-Id: Ib4cc3cbc1503b85dfdfa0edab58a49c95eac5993

commit 61de3abae68449663359c92009ff1fe9be16505e
Author: YummyBian <yummy.bian@gmail.com>
Date:   Thu Oct 9 13:36:22 2014 +0800

Some assignment statements are redundant
    
    Don't need to initialize with false, only determine true conditional
    directly.
    
    Closes-Bug: 1379149
    
    Change-Id: I8685896b8dd9f7008b8b2311a2ad2a56c95a8bc6

commit d9bfa06be026e8acdb75fed0ae130b5c209251ce
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Oct 10 17:08:06 2014 -0700

Fix rebalance crash post-2.1.0 upgrade.
    
    If you had a builder file from an earlier version of Swift (before
    6d77c37) and you attempted to rebalance it with a new version of Swift
    (after 6d77c37, before this commit), you could get a crash.
    
    Commit 6d77c37 changed the calculation for parts_wanted slightly
    (round up instead of round down), so if a rebalance is attempted
    without recomputing parts_wanted, then swift-ring-builder crashes.
    
    The crash only occurs if the first command run is "rebalance". Adding
    a device, removing a device, changing a device's weight, or changing
    replica count (basically, anything that gives you a reason to
    rebalance) will all cause parts_wanted to be recalculated, so
    subsequent rebalances will succeed.
    
    Change-Id: I6f7a69701f74546a00d3183aa7763e9e708408c9

commit a79682f2389e5ae5e6f40d37f23383ac977ca2a8
Author: David Goetz <dpgoetz@gmail.com>
Date:   Fri Oct 10 02:04:44 2014 -0700

Fix bug with expirer and unicode
    
    Looks like I wasn't careful enough last time i fixed this bug- this
    is basically the same deal, just further down the line. What happens
    without this bug fix is you upload a object with a unicode name and
    a x-delete-after header and when it does expire the customer object
    gets removed but the expirer marker object doesn't.
    
    Change-Id: I82359c87a0919d71693e49ecf08e0f1eedc9d18e

commit 7c1c8c0b4d36ec1795c11950652fa42564352264
Author: David Goetz <dpgoetz@gmail.com>
Date:   Tue Oct 7 15:25:59 2014 -0700

Ssync does not replicate custom object headers
    
    Closes-Bug: #1329093
    
    Change-Id: Ie9d80089a38d7a9b3464c66237d4d2d23331ebd5

commit dec9448c871ca998bb63b6ffe0541378cbc7a5f7
Author: Filippo Giunchedi <fgiunchedi@wikimedia.org>
Date:   Fri Oct 10 12:51:50 2014 +0100

swift-drive-audit: pass logger to get_errors
    
    Closes-Bug: #1379767
    Change-Id: Ie8ec5e9ffb977ab39aedeb878e6aae9f0b62530b

commit 0c0ac09a467d2a37c1144e76d1289ec1ee62dd85
Author: Madhuri Kumari <madhuri.rai07@gmail.com>
Date:   Wed Oct 1 16:26:15 2014 +0530

Convert maximum length to integer in name_check
    
    This patch converts maximum_length to integer in
    name_check middleware if read from proxy-server.conf
    file.
    
    Change-Id: I28d66fd1b7ce56ca5540cb7189d084c36e063d43
    Closes-Bug: 1372397

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-12: Related fix proposed to swift (stable/juno)

#12

Related fix proposed to branch: stable/juno
Review: https://review.openstack.org/134082

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-26: Related fix merged to swift (master)

#13

Reviewed: https://review.openstack.org/125746
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=1c9bc0b522bed333b04a46ed7bd2c66a4eb89860
Submitter: Jenkins
Branch: master

commit 1c9bc0b522bed333b04a46ed7bd2c66a4eb89860
Author: Jay S. Bryant <email address hidden>
Date: Thu Oct 2 14:10:04 2014 -0500

Handle os.listdir failures in object-updater

    While investigating bug 1375348 I discovered the problem
    reported there was not limited to the object-auditor. The
    object-updater has similar bugs.

This patch catches the unhandled exception that can be thrown
by os.listdir if the self.devices directory is inaccessible.

Change-Id: I6293b840916bb63cf9eebbc05068d9a3c871bdc3
Related-bug: 1375348

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-12-01: Related fix proposed to swift (feature/ec)

#14

Related fix proposed to branch: feature/ec
Review: https://review.openstack.org/138165

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-12-01: Related fix merged to swift (feature/ec)

#15

Download full text (15.6 KiB)

Reviewed: https://review.openstack.org/138165
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0d3ebf09b94b41782b2c2a6bbcf255bf1203eca0
Submitter: Jenkins
Branch: feature/ec

commit 977d7c14daa38ab9c9d79bbf8b92371024b93fc8
Author: John Dickinson <email address hidden>
Date: Wed Nov 26 14:19:08 2014 -0800

Fix tempfile bugs from commit 6978275

    Commit 6978275 changed xprofile middleware's usage of mktemp
    and moved to using tempfile. But it was clearly never tested,
    because the os.close() calls never worked. This patch updates
    that previous patch to use a context to open and close the file.

Change-Id: I40ee42e8539551fd8e4dfb353f50146ab40a7847

commit dec97fc3ba2c71884f1c098e7d9cd1f709f74958
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Nov 26 06:13:29 2014 +0000

Imported Translations from Transifex

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: Ibf319f7cc1b5036ad8031776cf2c6018fb8a0159

commit 01f6e860066640a2ba1406a23c93a72b34ec495e
Author: Clay Gerrard <email address hidden>
Date: Fri Nov 21 17:28:13 2014 -0800

Add Expected Failure for ssync with sys-meta

    Sysmeta included with an object PUT persists with the PUT data - if an
    internal operation such as POST-as-copy during partial failure, or ssync
    with fast-POST (not supported), causes that data to be lost then the
    associated sysmeta will also be lost.

    Since object sys-meta persistence in the face of a POST when the
    original .data is unavailable requires fast-POST with .meta files the
    probetest that validates object sys-meta persistence of a POST when the
    most up-to-date copy of the object with sys-meta is unavailable
    configures an InternalClient with object_post_as_copy = false.

    This non-default configuration option is not supported by ssync and
    results in a loss of sys-meta very similar to the object sys-meta
    failure you would see with object_post_as_copy = true when the COPY part
    of the POST is unable to retrieve the most recently written object with
    sys-meta.

    Until we can fix the default POST behavior to make metadata updates
    without stomping on newer data file timestamps we should expect object
    sys-meta to be "very very best possible but not really guaranteed
    effort".

Until we can fix ssync to replicate metadata updates without stomping on
newer data file timestamps we should expect this test to fail.

    When ssync replication of fast-POST metadata update is fixed this test
    will fail signaling that the expected failure cruft should be removed,
    but other parts of ssync replication will still work and some other bugs
    can be fixed while we wait.

Change-Id: Ifc5d49514de79b78f7715408e0fe0908357771d3

commit a8751ae557616cab1cafd98a338cad352526a262
Author: Cedric Dos Santos <email address hidden>
Date: Tue Nov 25 12:37:05 2014 +0100

Correct misspelled words

In some files I found misspelling words.

bin/swift-reconciler-enqueue#l26
prima...

Reviewed:  https://review.openstack.org/138165
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0d3ebf09b94b41782b2c2a6bbcf255bf1203eca0
Submitter: Jenkins
Branch:    feature/ec

commit 977d7c14daa38ab9c9d79bbf8b92371024b93fc8
Author: John Dickinson <me@not.mn>
Date:   Wed Nov 26 14:19:08 2014 -0800

Fix tempfile bugs from commit 6978275
    
    Commit 6978275 changed xprofile middleware's usage of mktemp
    and moved to using tempfile. But it was clearly never tested,
    because the os.close() calls never worked. This patch updates
    that previous patch to use a context to open and close the file.
    
    Change-Id: I40ee42e8539551fd8e4dfb353f50146ab40a7847

commit dec97fc3ba2c71884f1c098e7d9cd1f709f74958
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Wed Nov 26 06:13:29 2014 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: Ibf319f7cc1b5036ad8031776cf2c6018fb8a0159

commit 01f6e860066640a2ba1406a23c93a72b34ec495e
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Nov 21 17:28:13 2014 -0800

Add Expected Failure for ssync with sys-meta
    
    Sysmeta included with an object PUT persists with the PUT data - if an
    internal operation such as POST-as-copy during partial failure, or ssync
    with fast-POST (not supported), causes that data to be lost then the
    associated sysmeta will also be lost.
    
    Since object sys-meta persistence in the face of a POST when the
    original .data is unavailable requires fast-POST with .meta files the
    probetest that validates object sys-meta persistence of a POST when the
    most up-to-date copy of the object with sys-meta is unavailable
    configures an InternalClient with object_post_as_copy = false.
    
    This non-default configuration option is not supported by ssync and
    results in a loss of sys-meta very similar to the object sys-meta
    failure you would see with object_post_as_copy = true when the COPY part
    of the POST is unable to retrieve the most recently written object with
    sys-meta.
    
    Until we can fix the default POST behavior to make metadata updates
    without stomping on newer data file timestamps we should expect object
    sys-meta to be "very very best possible but not really guaranteed
    effort".
    
    Until we can fix ssync to replicate metadata updates without stomping on
    newer data file timestamps we should expect this test to fail.
    
    When ssync replication of fast-POST metadata update is fixed this test
    will fail signaling that the expected failure cruft should be removed,
    but other parts of ssync replication will still work and some other bugs
    can be fixed while we wait.
    
    Change-Id: Ifc5d49514de79b78f7715408e0fe0908357771d3

commit a8751ae557616cab1cafd98a338cad352526a262
Author: Cedric Dos Santos <cedric.dos.sant@gmail.com>
Date:   Tue Nov 25 12:37:05 2014 +0100

Correct misspelled words
    
    In some files I found misspelling words.
    
    bin/swift-reconciler-enqueue#l26
       primarly => primarily
    swift/account/backend.py#l309
       ommited => omitted
    swift/container/replicator.py#l158
       successfull => successful
    test/unit/account/test_backend.py#1450
       non_existant_policy_index => non_existent_policy_index
    test/unit/account/test_backend.py#1451
       'test-non-existant-policy'=> 'test-non-existent-policy'
    test/unit/account/test_backend.py#1453
       non_existant_policy_index => non_existent_policy_index
    
    Change-Id: I976236e3200a6fbdc20be464acff182b6cface81

commit 98de48d898f1419b0a0cfc273ec778e60331e623
Author: Shilla Saebi <shilla.saebi@gmail.com>
Date:   Sat Nov 22 15:38:48 2014 -0500

Fix typo in apache_deployment doc
    
    Change-Id: I42d76f544290dbda62633de90608d41caadac084

commit a1872b0498e1aa4182a4373c89beeaaaa219ea17
Author: Shilla Saebi <shilla.saebi@gmail.com>
Date:   Sat Nov 22 15:35:10 2014 -0500

Fix 2 typos in admin_guide file
    
    Change-Id: Ibf1e5dbf6ff4747c7f23f6638321ab41bba3021b

commit 0dc4b0a7b75237c09caffdac8c0dfd92bf8e3286
Author: Shilla Saebi <shilla.saebi@gmail.com>
Date:   Sat Nov 22 16:11:37 2014 -0500

Fix typos in overview_large_objects and versioning doc
    
    
    Change-Id: I1a919ad1b0298d5817f9eb2caf5e3bd7b3243c2c

commit 7a0c4d248257259612d3471ab42669ca9d90c573
Author: Takashi Kajinami <kajinamit@nttdata.co.jp>
Date:   Mon Nov 24 22:05:07 2014 +0900

Remove invalid connection checking in db_replicator
    
    Account/Container-replicator checks connection generation and timeout
    in HTTP REPLICATE Request in _repl_to_node, but it doesn't really checks
    connection but only construction of ReplConnection class.
    This patch removes that invalid checking.
    
    Change-Id: Ie6b4062123d998e69c15638b741e7d1ba8a08b62
    Closes-Bug: #1359018

commit 1c9bc0b522bed333b04a46ed7bd2c66a4eb89860
Author: Jay S. Bryant <jsbryant@us.ibm.com>
Date:   Thu Oct 2 14:10:04 2014 -0500

Handle os.listdir failures in object-updater
    
    While investigating bug 1375348 I discovered the problem
    reported there was not limited to the object-auditor.  The
    object-updater has similar bugs.
    
    This patch catches the unhandled exception that can be thrown
    by os.listdir if the self.devices directory is inaccessible.
    
    Change-Id: I6293b840916bb63cf9eebbc05068d9a3c871bdc3
    Related-bug: 1375348

commit 8cc075a8fb7561c736cb38d629f5b3d8ddb67497
Author: Jay S. Bryant <jsbryant@us.ibm.com>
Date:   Thu Nov 20 15:56:58 2014 -0600

mock out os.listdir to return a list
    
    os.listdir returns a list of items.  The test case had been
    written to return a single item which, though not really changing
    the result of the test, was not the best approach.
    
    This patch updates the test case to return a list instead of a single
    item.
    
    Change-Id: I793e0636440c0de0ca339c6592adec3e8b4ee1b4

commit fb353e1756df02622ea257acc987df4ccd094872
Author: John Dickinson <me@not.mn>
Date:   Thu Nov 20 10:22:27 2014 -0800

update AUTHORS
    
    Change-Id: I416e81b20a129377782f5d9298f8b8f5be079c27

commit 6c02adc33e3238f3fe0b75f2857503d1036f4737
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu Nov 20 06:11:14 2014 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Co-Authored-By: Pearl Yajing Tan <pearl.y.tan@seagate.com>
    
    Change-Id: Ifa3e292b8d5afbef8a99121b233e5ea596e672b7

commit 87d8626505c31511911facd5e1a1c3b3a65e8663
Author: Eohyung Lee <liquidnuker@gmail.com>
Date:   Thu Nov 20 11:38:49 2014 +0900

fix example typo
    
    5 * 1024 * 1024 = 5242880
    
    Change-Id: I0eeb6e2d9fbd79103cd8c658627344f73fed9498

commit ddf8b0594bb7e5ea9022982a7c5e15d9b261c22e
Author: Andreas Jaeger <aj@suse.de>
Date:   Wed Nov 19 09:11:55 2014 -0500

Fix translation setup
    
    Fix the output directory, it should be swift/locale.
    This fixes the importing of translations.
    
    Change-Id: I48311773c9d200c3b1739dc796618849416096ed

commit e0307f950bccde1337898e16087af726429e13f4
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Mon Nov 17 12:30:15 2014 -0800

Always use FakeMemcache for in-process tests
    
    Better isolation and consistency for in-process functests to always use
    the FakeMemcache.  If you want to test the real memcache you have real
    functional tests.
    
    Change-Id: Ic483f794e122130bd7694c9a5f9a2b1cd0b9a653

commit 6f9ca6122efac6c1c252a948cd5cc18c58c625ff
Author: Anne Gentle <anne@openstack.org>
Date:   Mon Nov 17 16:11:05 2014 -0600

Adds v1 API documentation to doc/source/api
    
    After discussion https://review.openstack.org/#/c/129384/ moving
    to the doc directory in swift repo.
    
    This lets us eliminate the object-api repo along with all the <service>-
    api repos and move content to audience-centric locations.
    
    Change-Id: Ia0d9973847f7409a02dcc1a0e19400a3c3ecdf32

commit 11a72a4a5084dbcb5539596c50793e45c5dac525
Author: Thiago da Silva <thiago@redhat.com>
Date:   Mon Nov 17 11:33:41 2014 -0500

move slo, dlo after tempauth in pipeline
    
    Noticed that slo and dlo middleware were placed before
    tempauth, they should be placed after
    
    DocImpact
    
    Change-Id: Ia931e2280125d846f248b23e219aebad14c66210
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit 2792fe81a93dbaa95e58f14099db5e11dd8cde68
Author: Daisuke Morita <morita.daisuke@lab.ntt.co.jp>
Date:   Tue Sep 30 11:06:08 2014 -0400

Show the sum of every policy's amount in /recon/async
    
    After the release of Swift ver. 2.0.0, some recon responses do not
    show each policy's information yet. To make things worse, some recon
    results only count on policy-0's score, therefore the total is not
    shown in the recon results.
    
    With this patch, async_pending count of recon results becomes
    policy-aware. Suppose a number of async_pending files for policy-0 is 2
    and a number for policy-1 is 3, recon sums up every policy's amount
    as follows.
    
    $ curl http://<host>:<port>/recon/async
    {"async_pending": 5} # It showed 2 before this commit
    
    Related-Bug: 1375332
    Change-Id: Ifc88b8c9e06b9f022a926a87ed807e938e1e0412

commit c9f824637845f342b6996058e0fea8338bd1305d
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Mon Aug 11 17:09:48 2014 +0100

Make in process functional tests use sample proxy-server.conf
    
    This patch was first motivated by noticing that the proxy
    server pipeline used for in process functional tests was
    out of date with respect to the pipeline in
    /etc/proxy-server.conf.sample. Rather than cut and paste
    the current pipeline into the in process setup, it seems
    like a better idea would be to have the in process tests
    always use the sample config.
    
    A further benefit is that in process functional tests will
    pick up changes to the sample config introduced by patches -
    previously test/functional/__init__.py would need to be
    manually modified to run in process functional tests
    on new middleware for example.
    
    Note: because the pipeline is now loaded using entry points,
    'python setup.py [develop|install]' will now be needed
    before running the tests.
    
    Obvious next steps would be to do the same for the backend
    servers, and to allow alternative config files and dir's
    to be specified, but this patch is the first step.
    
    Also drive-by fixes some typos in proxy-server.conf.sample
    
    Change-Id: If442bd7c2b1721ec92839c4490924ba33e1545d8

commit e429cd81be711f42441a08e34c077dcd7a97bed0
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Nov 13 16:40:05 2014 -0800

Make error limits survive a ring reload
    
    The proxy was storing the error count and last-error time in the
    ring's internal data, specifically in the device dictionaries. This
    works okay, but it means that whenever a ring changes, all the error
    stats reset.
    
    Now the error stats live in the proxy server object, so they survive a
    ring reload.
    
    Better yet, the error stats are now keyed off of the node's
    IP/port/device triple, so if you have the same device in two rings
    (like with multiple storage policies), then the error stats are
    combined. If the proxy server sees a 507 for an objec request in
    policy X, then that will now result in that particular object disk
    being error-limited for requests in policies Y and Z as well.
    
    Change-Id: Icc72b68b99f37367bb16d43688e7e45327e3e022

commit b98fe3b77b6b422e5e5978f6cf82a11fb87aedfc
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Nov 11 17:03:29 2014 -0800

Prefer X-Backend-Timestamp for X-Newest
    
    When a X-Backend-Timestamp is available it would generally preferred
    over a less specific value and sorts correctly against any X-Timestamp
    values anyway.
    
    Change-Id: I08b7eb37ab8bd6eb3afbb7dee44ed07a8c69b57e

commit 466403723c4c1fd575b1340e0f9214ee28f0aeb7
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon Nov 3 14:20:08 2014 -0800

Make resetswift customizable via environment
    
    Instead of recommending to edit resetswift to replace "/dev/sdb1" with
    "/srv/swift-disk", use an environment variable instead. This way I can
    set SAIO_BLOCK_DEVICE=/srv/swift-disk in my .bashrc, and then when I'm
    testing out changes to resetswift, I don't need to remember to edit
    the modified script, nor do I end up submitting changes with the wrong
    default in there.
    
    The variable defaults to /dev/sdb1, so if you use the script unmodified
    and don't set SAIO_BLOCK_DEVICE, nothing changes for you.
    
    Change-Id: I741a8c91c2c54a4f32bc391cd794ef4206402753

commit 331b14238effc9d1928e478bba86122f7e2525c1
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Nov 7 13:53:46 2014 -0800

Reject object names with Unicode surrogates
    
    Technically, you can't encode surrogates into UTF-8 at all, but Python
    2 lets you get away with it. Python 3 does not.
    
    We already have a check for surrogate pairs (commit 0080337), but not
    one for lone surrogates. This commit forbids object names with lone
    surrogates in them.
    
    The problem with surrogates is trivially reproducible:
    
        swift@saio:~$ python2.7
        Python 2.7.3 (default, Feb 27 2014, 19:58:35)
        [GCC 4.6.3] on linux2
        Type "help", "copyright", "credits" or "license" for more information.
        >>> b'\xed\xa0\xbc'.decode('utf-8')
        u'\ud83c'
        >>>
    
        swift@saio:~$ python3.3
        Python 3.3.5 (default, Aug  4 2014, 15:27:24)
        [GCC 4.6.3] on linux
        Type "help", "copyright", "credits" or "license" for more information.
        >>> b'\xed\xa0\xbc'.decode('utf-8')
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
        UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: invalid continuation byte
        >>>
    
    See also http://bugs.python.org/issue9133
    
    Change-Id: I7c31022e8a028c3cdf2ed1586349509d96cfded9

commit 2659888c921d09bd1dd23cda6ee2f158187d80e6
Author: Matthew Oliver <matt@oliver.net.au>
Date:   Fri Jun 13 19:12:31 2014 +1000

When a filesystem does't support xattr return a 507
    
    Currently when the object server tries to write an object's metadata
    to a filesystem that doesn't support xattr, it errors with a stacktrace
    and returns a 500 error back to the user with no information.
    
    This patch catches the resulting IOError when attempting to read or write
    the xattr metadata, logs the error nicely and then returns a 507 error
    back to the user.
    
    Seeing as this change is sending back a 507, it also catches and logs
    the out of disk space errors (ENOSPC and EDQUOT).
    
    Change-Id: I31932b57582817a0b3b58dd315a996bd0bcbc99b
    Closes-Bug: #966671

commit 0a5268c34caa25487c48380a1821e4deac178538
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Tue Sep 16 14:46:08 2014 +0000

Fix bug in swift-ring-builder list_parts
    
    The number of shown replicas in the partition list might differ from the
    actual number of replicas (as shown in the bugreport).
    
    This codes simply iterates for the builder._replica2part2dev and
    remembers the number of replicas for each partition.
    
    The code to find the partitions was moved to swift/common/ring/utils.py
    to make it easier to test, and a test to ensure the correct number of
    replicas is returned was added.
    
    Closes-Bug: 1370070
    Change-Id: Id6a3ed437bb86df2f43f8b0b79aa8ccb50bbe13e

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-01-09: Related fix proposed to swift (stable/juno)

#16

Related fix proposed to branch: stable/juno
Review: https://review.openstack.org/146211

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-12-07: Change abandoned on swift (stable/juno)

#17

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/134082
Reason: I'm abandoning this change in preparation for deleting the stable/juno branch, which is now at end of life.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-12-07:

#18

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/146211
Reason: I'm abandoning this change in preparation for deleting the stable/juno branch, which is now at end of life.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

python-roundup #9133
[2:10] Edit

Bug watches keep track of this bug in other bug trackers.