Excessive resource consumption looking for containers to sync
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Fix Released
|
Undecided
|
Eran Rom |
Bug Description
With every run, the container sync daemon iterates over all containers in the node looking for containers to sync.
This is a waste of resources, esp. when there is a small fraction of the containers that are configured for sync.
the issue was raised as part of a discussion on the need to improve the container sync performance.
Changed in swift: | |
assignee: | nobody → Eran Rom (eranr) |
status: | New → In Progress |
clayg (clay-gerrard) wrote : | #1 |
Eran Rom (eranr) wrote : | #2 |
Yeah, I spent some time considering both options, and decided to go for the first one.
I would say that at the high level the tradeoff between the two is simplicity (1) Vs. fairness (2)
I decided for now to go with the simpler approach (at least as a starting point) as there is going to be the addition of parallelism to the container sync daemon, and I prefer to take this in small steps.
Thanks very much for the feedback!
clayg (clay-gerrard) wrote : | #3 |
Swift Design Principles #2 - Simple Scales
it's not just a describing the idea of building up large scale architectures with smaller simpler components (I think the kids now-a-days call this a "micro-service" architecture?), it's also a riff on Kernighans's old adage about being smart enough to debug your cleverness ;)
I think #1 is a good call. Although I have some vested interest in #2 as well [1].
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (master) | #4 |
Fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (master) | #5 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (master) | #6 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 69e7424d3cb6c59
Author: Eran Rom <email address hidden>
Date: Tue Jul 28 09:41:12 2015 +0300
Add container sync probe test to SAIO default set
SAIO Configuration and documentation changes enabling to run the
container sync probe test by default
Change-Id: Iccf59533d0d4fe
Related-Bug: #1476623
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (feature/crypto) | #7 |
Related fix proposed to branch: feature/crypto
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (feature/crypto) | #8 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: feature/crypto
commit e02609c66a80484
Author: Samuel Merritt <email address hidden>
Date: Tue Sep 1 15:19:50 2015 -0700
Preserve traceback in swift-dispersio
Commit c690bcb fixed a bug in the dispersion report, but changed this
from a bare "raise" to "raise err", which loses the traceback. Not a
big deal, but worth putting back IMO.
Change-Id: Id5b72153a4b8df
commit d06d4ad0fd2dfe6
Author: Minwoo Bae <email address hidden>
Date: Tue Sep 1 15:08:44 2015 -0500
Included reference in swift.obj.diskfile to enumerate the string
used for data file paths.
Change-Id: Ie22caa678bc00d
commit 524c89b7eeff037
Author: John Dickinson <email address hidden>
Date: Fri Aug 21 13:39:41 2015 -0700
Updated CHANGELOG, AUTHORS, and .mailmap for 2.4.0 release.
Change-Id: Ic6301146b839c9
commit 05de1305a903ee4
Author: Clay Gerrard <email address hidden>
Date: Thu Aug 27 18:35:09 2015 -0700
Make ssync_sender send valid chunked requests
The connect method of ssync_sender tells the remote connection that it's
going to send a valid HTTP chunked request, but if the remote end needs
to respond with an error of any kind sender throws HTTP right out the
window, picks up his ball, and closes the socket down hard - much to the
surprise of the eventlet.wsgi server who up to this point had been
playing along quite nicely with this 'SSYNC' nonsense assuming that
everyone here is consenting mature adults.
If you're going to make a "Transfer-Encoding: chunked" request have the
good decency to finish the job with a proper '0\r\n\r\n'. [1]
N.B. It might be possible to handle an error status during the
initialize_
honestly it's not entirely clear to me when the server isn't going to
close the connection if the client is still expected to send the body
[2] - further if the error comes later during missing_check or updates
we'll for sure want to send the chunk transfer termination line before
we close down the socket and this way we cover both.
1. Really, eventlet.wsgi shouldn't be so blasted brittle about this [3]
2. https:/
3. https:/
Closes-Bug #1489587
Change-Id: Ic17c6c3075553f
commit 993ee4e37af1961
Author: nakagawamsa <email address hidden>
Date: Fri Aug 28 11:49:43 2015 +0900
Remove duplicate X-Backend-
There is duplicate 'X-Backend-
One key has fixed policy index value, and another ha...
tags: | added: in-feature-crypto |
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (feature/hummingbird) | #9 |
Related fix proposed to branch: feature/hummingbird
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (feature/hummingbird) | #10 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: feature/hummingbird
commit cb683d391cb66d0
Author: OpenStack Proposal Bot <email address hidden>
Date: Sat Sep 5 06:17:51 2015 +0000
Imported Translations from Transifex
For more information about this automatic import see:
https:/
Change-Id: I2d92b8e34a665f
commit e4542455c8a07b7
Author: Emett Speer <email address hidden>
Date: Wed Sep 2 17:18:03 2015 -0700
[Labs] Update links to Cloud Admin Guide
Update links to the Cloud Admin Guide after the
RST conversion of that book altered URLs.
Change-Id: I899f8938498b74
commit 58fcc0752397830
Author: Christian Schwede <email address hidden>
Date: Wed Sep 2 10:52:34 2015 +0000
Test if container_sweep is executed on unmounted devices
This change ensures that container_sweep is not run if a device is not mounted
and mount_check is set to True.
Change-Id: I823083c8431d9e
commit e02609c66a80484
Author: Samuel Merritt <email address hidden>
Date: Tue Sep 1 15:19:50 2015 -0700
Preserve traceback in swift-dispersio
Commit c690bcb fixed a bug in the dispersion report, but changed this
from a bare "raise" to "raise err", which loses the traceback. Not a
big deal, but worth putting back IMO.
Change-Id: Id5b72153a4b8df
commit d06d4ad0fd2dfe6
Author: Minwoo Bae <email address hidden>
Date: Tue Sep 1 15:08:44 2015 -0500
Included reference in swift.obj.diskfile to enumerate the string
used for data file paths.
Change-Id: Ie22caa678bc00d
commit 615c7a204b9386e
Author: Brian Cline <email address hidden>
Date: Tue Sep 1 10:51:20 2015 -0500
Adds useful dispersion info from changelog
Change-Id: I1a45088fc32620
commit 3b8755098a1786c
Author: janonymous <email address hidden>
Date: Sun Aug 2 21:29:13 2015 +0530
Replace a / b with a // b to use integer division where needed
Change-Id: I72c81faa62786e
commit 524c89b7eeff037
Author: John Dickinson <email address hidden>
Date: Fri Aug 21 13:39:41 2015 -0700
Updated CHANGELOG, AUTHORS, and .mailmap for 2.4.0 release.
Change-Id: Ic6301146b839c9
commit 05de1305a903ee4
Author: Clay Gerrard <email address hidden>
Date: Thu Aug 27 18:35:09 2015 -0700
Make ssync_sender send valid chunked requests
The connect method of ssync_sender tells the remote connection that it's
going to send a valid HTTP chunked request, but if the remote end needs
to respond with an error of any kind sender th...
tags: | added: in-feature-hummingbird |
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master) | #11 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 85a0a6a28e166bc
Author: Eran Rom <email address hidden>
Date: Sun Jul 26 13:31:17 2015 +0300
Container-Sync to iterate only over synced containers
This change introduces a sync_store which holds only containers that
are enabled for sync. The store is implemented using a directory
structure that resembles that of the containers directory, but has
entries only for containers enabled for sync.
The store is maintained in two ways:
1. Preemptively by the container server when processing
PUT/POST/DELETE operations targeted at containers with
x-container
2. In the background using the containers replicator
whenever it processes a container set up for sync
The change updates [1]
[1] http://
Change-Id: I9ae4d4c7ff6336
Closes-Bug: #1476623
Changed in swift: | |
status: | In Progress → Fix Released |
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/swift 2.6.0 | #12 |
This issue was fixed in the openstack/swift 2.6.0 release.
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (feature/crypto) | #13 |
Fix proposed to branch: feature/crypto
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (feature/crypto) | #14 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: feature/crypto
commit e13a03c379273ee
Author: Tim Burke <email address hidden>
Date: Wed Jan 20 16:06:26 2016 -0800
Stop overriding builtin range
Change-Id: I315f8b554bb9e9
commit 0a404def7d54d1e
Author: John Dickinson <email address hidden>
Date: Wed Jan 20 15:19:35 2016 -0800
remove unneeded duplicate dict keys
Change-Id: I926d7aaa9df093
commit 221f94fdd39fd2d
Author: John Dickinson <email address hidden>
Date: Tue Jan 19 14:50:24 2016 -0800
authors and changelog updates for 2.6.0
Change-Id: Idd0ff9e70abc07
commit 58359269b0e971e
Author: Samuel Merritt <email address hidden>
Date: Tue Dec 8 16:36:05 2015 -0800
Fix memory/socket leak in proxy on truncated SLO/DLO GET
When a client disconnected while consuming an SLO or DLO GET response,
the proxy would leak a socket. This could be observed via strace as a
socket that had shutdown() called on it, but was never closed. It
could also be observed by counting entries in /proc/<pid>/fd, where
<pid> is the pid of a proxy server worker process.
This is due to a memory leak in SegmentedIterable. A SegmentedIterable
has an 'app_iter' attribute, which is a generator. That generator
references 'self' (the SegmentedIterable object). This creates a
cyclic reference: the generator refers to the SegmentedIterable, and
the SegmentedIterable refers to the generator.
Python can normally handle cyclic garbage; reference counting won't
reclaim it, but the garbage collector will. However, objects with
finalizers will stop the garbage collector from collecting them* and
the cycle of which they are part.
For most objects, "has finalizer" is synonymous with "has a __del__
method". However, a generator has a finalizer once it's started
running and before it finishes: basically, while it has stack frames
associated with it**.
When a client disconnects mid-stream, we get a memory leak. We have
our SegmentedIterable object (call it "si"), and its associated
generator. si.app_iter is the generator, and the generator closes over
si, so we have a cycle; and the generator has started but not yet
finished, so the generator needs finalization; hence, the garbage
collector won't ever clean it up.
The socket leak comes in because the generator *also* refers to the
request's WSGI environment, which contains wsgi.input, which
ultimately refers to a _socket object from the standard
library. Python's _socket objects only close their underlying file
descriptor when their reference counts fall to 0***.
This commit makes SegmentedIterab
self.
making it eligible for garbage collection.
* in Python < 3...
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (feature/crypto) | #15 |
Fix proposed to branch: feature/crypto
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Change abandoned on swift (feature/crypto) | #16 |
Change abandoned by Alistair Coles (<email address hidden>) on branch: feature/crypto
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (feature/hummingbird) | #17 |
Fix proposed to branch: feature/hummingbird
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (feature/hummingbird) | #18 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: feature/hummingbird
commit d6b4587a554b51b
Author: Olga Saprycheva <email address hidden>
Date: Tue Mar 8 10:57:56 2016 -0600
Removed redundant file for flake8 check
Change-Id: I4322978aa20ee7
commit 643dbce13414053
Author: OpenStack Proposal Bot <email address hidden>
Date: Tue Mar 8 06:35:49 2016 +0000
Imported Translations from Zanata
For more information about this automatic import see:
https:/
Change-Id: I96b8ff1287bf21
commit 83713d37f0331c5
Author: Daisuke Morita <email address hidden>
Date: Mon Mar 7 18:30:47 2016 -0800
Missing comments for storage policy parameter
There are missing comments about storege_
comments are added.
Change-Id: I3de3f0e6864e65
commit 2cff2dec3d1c458
Author: Olga Saprycheva <email address hidden>
Date: Fri Mar 4 15:19:39 2016 -0600
Fixed pep8 and flake8 errors in doc/source/conf.py and updated flake8 commands in tox.ini to test it.
Change-Id: I2add370e4cfb55
commit 043fbca6d08648b
Author: Christian Schwede <email address hidden>
Date: Fri Mar 4 09:33:17 2016 +0000
Remove Erasure Coding beta status from docs
This removes notes stating support for Erasure coding as beta. Questions
regarding the stability of EC are coming up regularly, and are often referring
to the docs that state EC as still in beta.
Besides this, a note marking statsd support as beta has been removed as well.
Change-Id: If4fb6a5c4cb741
commit 09c73b86e9255f2
Author: Pete Zaitcev <email address hidden>
Date: Thu Mar 3 10:24:28 2016 -0700
Fix a crash in exception printout
Says the number of arguments does not match the number of '%'.
Change-Id: I8b5e395a07328f
commit fad5fabe0a22e8a
Author: Tim Burke <email address hidden>
Date: Thu Mar 3 15:07:08 2016 +0000
During functional tests, 404 response to a DELETE is successful
Previously, we would only consider 204 responses successful, which would
cause some spurious gate failures, such as
Change-Id: Ic8c30064792435
commit e91de49d6864b37
Author: Alistair Coles <email address hidden>
Date: Mon Aug 10 10:30:10 2015 -0500
Update container on fast-POST
This patch makes a number of changes to enable content-type
metadata to be updated when using the fast-POST mode of
operation, as proposed in the associated spec ...
To my knowledge two ideas has been discussed as a possible resolution to this issue:
1) when a daemon that already walks the tree finds a database with container_sync keys in set in the stats table it drops a pointer to the container in a smaller tree that can be walked more efficiently by the container-sync daemon
2) when the replicator encounters a container with contaienr_sync keys it replicates the rows into sharded container sync accounts exactly like the reconciler, and the container_sync daemon is redesigned to horizontally scale using coordinated processes like the expirier.
Thank you for documenting this issue. Please feel free to jot down/discuss any other ideas you are pursuing.