Reviewed: https://review.opendev.org/713632 Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=79bd2e59e5c15ee84814ec1c4f0893176ba79412 Submitter: Zuul Branch: feature/losf commit f2ffd900593db2829a39a073f0c033d82985c40f Author: Clay Gerrard Date: Fri Feb 28 11:09:51 2020 -0600 Apply limit to list versioned containers Change-Id: I28e062273d673c4f07cd3c5da088aa790b77a599 Closes-Bug: #1863841 commit dc40779307095b8d0b2761b77b9cb2904ec721ae Author: Clay Gerrard Date: Fri Feb 28 10:00:25 2020 -0600 Use float consistently for proxy timeout settings Change-Id: I433c97df99193ec31c863038b9b6fd20bb3705b8 commit 55049beda5b9d7038a3604a87f28312d7702ccb2 Author: Tim Burke Date: Fri Feb 28 18:59:32 2020 -0800 tests: Use timedelta to adjust dates, not string manipulations Change-Id: I8f65ccd7f2a79d5b877bfbef0274fb7857e21391 commit 3b65a5998cc921d2763cf1a9ec1e40b88491262d Author: Tim Burke Date: Wed Jan 10 06:16:41 2018 +0000 Fix up some Content-Type handling in account/container listings Update content type on 204 (not just 200) to properly handle HEAD requests from xml/txt listings. Add "Vary: Accept" header to listings, since otherwise, browsers may serve the wrong content type from cache (even though we *would have* sent the *right* type if it actually sent the request). Change-Id: Iaa333aaca36a8dc2df65d38ef2173e3a6e2000ee commit ecca23eb806e11cf6517f0456483da7a065350a8 Author: Clay Gerrard Date: Fri Feb 21 15:33:21 2020 -0600 Extend eventlet_debug logging to GreenAsyncPile Change-Id: Ibd9fe5c9a1e75b86eb7d540594d5cf516758e17e commit 0fb3371484f1d0f629d0b0e33f6aafbff0e43ee9 Author: Sam Morrison Date: Tue Feb 18 10:17:50 2020 +1100 Delay importing swiftclient until after monkey-patching Commit message below partly copied from nova: Eventlet monkey patching should be as early as possible We were seeing infinite recursion opening an ssl socket when running various combinations of python3, eventlet, and urllib3. It is not clear exactly what combination of versions are affected, but for background there is an example of this issue documented here: https://github.com/eventlet/eventlet/issues/371 The immediate cause in swift's case was that we were calling eventlet.monkey_patch() after importing swiftclient (which imported requests, which finally imported urllib3). We only use the imported function in one place, however; hold off on importing until we actually need it to ensure that monkey patching happens first. Note that we *don't* want to monkey-patch at import time, as we've previously had bugs related to import-time side-effects. Change-Id: I24f4bcc3d62dc37fd9559032bfd25f5b15f98745 Closes-bug: #1863680 Related-bug: #1380815 commit a5afe767581d2cb97cf3690067e6d626c7682c2c Author: Tim Burke Date: Wed Feb 19 10:09:49 2020 -0800 Revert "Make rolling-upgrade job non-voting until virtualenv problem is solved" This reverts commit cd4ba90cc1a13c94e2a02f1af015e20d6882e05f. The virtualenv problem seems to have been resolved. Change-Id: Icae429b4eba796328e767317e204ca30cc105da8 commit cd4ba90cc1a13c94e2a02f1af015e20d6882e05f Author: Tim Burke Date: Thu Feb 13 22:02:26 2020 -0800 Make rolling-upgrade job non-voting until virtualenv problem is solved Change-Id: I4c8a3756afccb2b42acbd9aeac5908f3d4febcae commit 2a8d47f00eb3553755aaf2329d6619bee57d9cd9 Author: Tim Burke Date: Thu Jan 2 15:19:27 2020 -0800 middlewares: Clean up app iters better Previously, logs would often show 499s in places where some other status would be more appropriate. Change-Id: I68dbb8593101cd3b5b64a1a947c68e340e36ce02 commit 61bf5ee1c428410b70bfab0a6eae361df948d03e Author: Charles Hsu Date: Thu Feb 6 15:35:11 2020 +0800 Ensure domain stored in memcached gets utf8 decoded on py2 Change-Id: I73b5af9645f3f7349144384609bf18a79620e92f Closes-Bug: #1862115 commit baaa5c57e18da5e7d4baa79e1b5420621c7adf0b Author: Sean McGinnis Date: Wed Feb 5 16:20:53 2020 -0600 Remove reno from test-requirements Reno is only needed for building release notes. Reno has now dropped py2 support, so attempts to install all requirements for py27 jobs fail with the latest reno release. Reno is already correctly listed in doc/requirements.txt, so this just removes the unnecessary entry in test-requirements.txt. Need-by: https://review.opendev.org/705994 Change-Id: Id1c54915ebe43f2b46f611a95109d2ab0d02d34d Signed-off-by: Sean McGinnis commit 09b7ed600bfda6f708db85cd850c0539c3448a0a Author: Tim Burke Date: Wed Feb 5 08:36:24 2020 -0800 sharding: filter shards based on prefix param when listing Otherwise, we make a bunch of backend requests where we have no real expectation of finding data. Change-Id: I7eaa012ba938eaa7fc22837c32007d1b7ae99709 commit ff885d30e46f65042843480f58a6f07ce66211cc Author: Tim Burke Date: Mon Feb 3 13:01:51 2020 -0800 py3: Fix up probe tests Change-Id: Ic0f54f393002e2170e7f1459625ee5a2b37df900 commit d358b9130d80f5fff492de03abf74676a06e61c3 Author: John Dickinson Date: Thu Dec 13 10:45:51 2018 -0800 added value and notes to a sample config file for s3token Change-Id: I18accffb2cf6ba6a3fff6fd5d95f06a424d1d919 commit 0606f404fe92a3c56014feba55069a206ddb1cc8 Author: John Dickinson Date: Thu May 2 10:53:44 2019 -0600 enforce account quotas on empty accounts Change-Id: Icade637b0d384dcc87aaa31da3861cd2646d4ccd Closes-Bug: #1827394 commit b8c16de0238054b7757c6c0f541af853232b54a9 Author: Thiago da Silva Date: Thu Jan 23 19:15:05 2020 +1100 Return correct etag for raw manifest When client sends a '?multipart-manifest=get&format=raw' request middleware will change the manifest returned from object server. This patch makes sure the response etag is updated to reflect changes to manifest content Change-Id: I0ac6dd0808fb041ba7663f4a472a06ee3f1d9a71 commit 5cd854b783ce80356ec62e709f0817cbe699c8cd Author: Tim Burke Date: Mon Jan 27 16:11:55 2020 -0800 Authors/changelog for swift 2.24.0 Change-Id: I60d4ad62c7d14ed15e114ed95102a2368c48f9f4 commit c7cb34ad61feb5b830ccb79437b32965916220ef Author: Thiago da Silva Date: Thu Jan 30 15:33:41 2020 +1100 Dockerhub description of saio image Change-Id: Ifcb5013a5728d93cf5491fbff81b2677450698e6 commit 4a776c4fbd69271b6197b2c20beaaae8a2b2e980 Author: Thiago da Silva Date: Thu Jan 30 15:56:36 2020 +1100 Add new versioning flag to docker image Change-Id: If6a25d6573502eebff3e28aa503721eda73af556 commit e3ee6ce31aed8d24fefdd20900d20688a12bb55d Author: Tim Burke Date: Mon Oct 14 18:35:00 2019 -0700 py38: cgi lost some names Drive-by: always use quote=True; it basically never hurts. Change-Id: Id91fdd19f226e9ec0d9c702d40d041c385c52b88 commit 6097660f0c350e12c026d62fd2fbd437140b917b Author: karen chan Date: Fri Jan 27 07:43:37 2017 -0800 s3api: Implement object versioning API Translate AWS S3 Object Versioning API requests to native Swift Object Versioning API, speficially: * bucket versioning status * bucket versioned objects listing params * object GETorHEAD & DELETE versionId * multi_delete versionId Change-Id: I8296681b61996e073b3ba12ad46f99042dc15c37 Co-Authored-By: Tim Burke Co-Authored-By: Clay Gerrard commit ebaf154fc4d7f91ff704ca2367556c137ed52137 Author: Romain LE DISEZ Date: Tue Jan 28 13:51:05 2020 -0500 Add a note to the part power increase documentation Specify that the relink command must be run as the same user than the daemon processes to avoid creating files that are not manipulable by the server/replicator/... Change-Id: If23592855db5f5bb0ec1e7c679de15769fd86871 commit bd063b678992a710f423a86d392a34237a8379ba Author: Tim Burke Date: Mon Jan 27 12:09:28 2020 -0800 s3api: use native strings in s3api.auth_details access_key Change-Id: I331b6871e5b62f61809338a1abddafe1263e7f02 commit a84e358dcfc4422cb67b8a19fba4203c6424bc43 Author: Romain LE DISEZ Date: Thu Mar 21 18:04:58 2019 +0100 SSYNC: log body of errors in SSYNC subrequests It helps an operator to understand why an SSYNC replication is not progressing on a specific partition. Change-Id: I75ccc833d579a98eff22377a83b703fd7e9c4720 commit d8821c75bd79c2b95ab7635b58754d059be2079b Author: Romain LE DISEZ Date: Mon Jan 20 10:03:27 2020 -0500 proxy-logging: add fields ttfb and pid Change-Id: I1611e34846e586703e9d3709fa64e8df41f2d685 commit 27fd97cef927f39a057f46cf09a6c38ed52c6aa9 Author: Romain LE DISEZ Date: Thu Dec 19 14:24:51 2019 -0500 Middleware that allows a user to have quoted Etags Users have complained for a while that Swift's ETags don't match the expected RFC formats. We've resisted fixing this for just as long, worrying that the fix would break innumerable clients that expect the value to be a hex-encoded MD5 digest and *nothing else*. But, users keep asking for it, and some consumers (including some CDNs) break if we *don't* have quoted etags -- so, let's make it an option. With this middleware, Swift users can set metadata per-account or even per-container to explicitly request RFC compliant etags or not. Swift operators also get an option to change the default behavior cluster-wide; it defaults to the old, non-compliant format. See also: - https://tools.ietf.org/html/rfc2616#section-3.11 - https://tools.ietf.org/html/rfc7232#section-2.3 Closes-Bug: 1099087 Closes-Bug: 1424614 Co-Authored-By: Tim Burke Change-Id: I380c6e34949d857158e11eb428b3eda9975d855d commit 51c9b13082b70bde91e553c5803bb2f403657b34 Author: Tim Burke Date: Mon Jan 27 08:47:23 2020 -0800 Add unittest2 back to lower-constraints.txt testtools (required by os-testr) still requires unittest2; until * testtools no longer requires unittest2 and * we bump up our minimum-supported version of testtools to a version that doesn't reference unittest2, we should continue listing unittest2 in lower-constraints.txt (just in case they ever have a new release). Admittedly, that seems unlikely at this point -- unittest2's latest release was back in 2015. Change-Id: I09301bf81d68357ec006a21dd37e72c56cefffe3 commit 2759d5d51c5e3c684081e066f9efeb495c141524 Author: Clay Gerrard Date: Fri Sep 13 12:25:24 2019 -0500 New Object Versioning mode This patch adds a new object versioning mode. This new mode provides a new set of APIs for users to interact with older versions of an object. It also changes the naming scheme of older versions and adds a version-id to each object. This new mode is not backwards compatible or interchangeable with the other two modes (i.e., stack and history), especially due to the changes in the namimg scheme of older versions. This new mode will also serve as a foundation for adding S3 versioning compatibility in the s3api middleware. Note that this does not (yet) support using a versioned container as a source in container-sync. Container sync should be enhanced to sync previous versions of objects. Change-Id: Ic7d39ba425ca324eeb4543a2ce8d03428e2225a1 Co-Authored-By: Clay Gerrard Co-Authored-By: Tim Burke Co-Authored-By: Thiago da Silva commit 26ff2eb1cbec8052797dc5e28af3a8049d5e5dfb Author: Thiago da Silva Date: Tue Dec 10 16:32:47 2019 +1100 container-sync: Sync static links similar to how we sync SLOs This allows static symlinks to be synced before their target. Dynamic symlinks could already be synced even if target object has not been synced, but static links previously required that target object existed before it can be PUT. Now, have container_sync middleware plumb in an override like it does for SLO. Change-Id: I3bfc62b77b247003adcee6bd4d374168bfd6707d commit 2be4716bc9ebbdb1da69231d4618489dff8af85f Author: Clay Gerrard Date: Mon Sep 16 13:06:15 2019 -0500 More multi-char-delimiter func test coverage Change-Id: I4cad872ef97313ec19a8e4ef042696565a56d4c5 commit 55cdb55075657c283ce786ca634f8a42ec786490 Author: Tim Burke Date: Wed Jan 22 11:00:18 2020 -0800 Lower the log level for falling back to mkstemp We recently started trying to use O_TMPFILE without the kernel version check, we gracefully degrade when it isn't available, and we don't give operators any way to avoid the O_TMPFILE attempt -- logging at warning seems way too noisy. Change-Id: I8f42127fd6eb930282161930c65e8614f3e3175f Related-Change: I3599e2ab257bcd99467aee83b747939afac639d8 commit cd78ee983d0ae0cebdb27327b6ea7950308d0dbf Author: Tim Burke Date: Wed Jan 22 09:40:54 2020 -0800 Bump up ceph tests timeout Looking at the last few hundred runs, we've had ~8% TIMEOUTs, and the average time is already 21 minutes. Let's give it an extra 10 minutes, see if the TIMEOUT rate comes down. Change-Id: Ifd9fbac032825bae1c8f5edd88c5d692a0b2cef1 commit f68e22d43a1bfbbdf182b1cc6819bdc83d166d5d Author: gillesbiannic Date: Fri Oct 25 20:11:30 2019 +0000 Stop retrying every deletes in container-sync Internal clients used to retry operations even on client errors, where we had no expectation that we would get a different response. Among other things, this would cause container-sync to needlessly retry deletes that would 404, even though we had code specifically in place to say that 404s are OK. While we're at it, also flag 409s as being OK (since the remote has newer data than us). Change-Id: I1833a8ff2ac2f4126b6274d6bba47e2b58a41745 Closes-Bug: #1849841 commit 73a0e8e9cf7d26e3a06d3b48164d345478de4161 Author: Tim Burke Date: Fri Oct 25 21:56:01 2019 -0700 probe: Add test for syncing a delete when the remote 404s This works fine; we continue processing the other rows in the DB. But it *does* take longer than it really ought to require. See the related bug; we ought to be able to shave some 17s off the test time by not retrying on the 404. Change-Id: I9ca2511651e9b2bc0045894baa4062d20bc15369 Related-Bug: #1849841 commit dad8fb7c3c0b4c1dbaecde341a5c07d02a0a578a Author: Tim Burke Date: Mon Jan 20 21:15:17 2020 -0800 packaging: Build universal wheels Apparently this had been done for us previously? See also: http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012097.html Change-Id: I27288f9e503f26827a8274d46b07c1330a21de10 commit 7bba3ad04dec35a6d12f950c862846a7377df190 Author: Tim Burke Date: Fri Jan 17 08:32:34 2020 -0800 Store normalized x-delete-at on PUT/POST Change-Id: I81e5d49623e884def8b5f6cbbf82f9b0d667d64a Closes-Bug: 1860149 commit ff0753fe19be3d7442aef3972e08adc48d25fe93 Author: Romain LE DISEZ Date: Thu Jan 9 09:08:25 2020 -0500 Allow bulk to fwd some headers at tar extraction Whitelisted headers include X-Delete-At/X-Delete-After and all Object metadata headers (X-Object-Meta-*) Closes-Bug: 1857546 Change-Id: If5fb164693e395f89d57899fb8ab355f1e3f817c commit 57ca3570e911bd290ce0e44bdebaa4b5b4ceca13 Author: Tim Burke Date: Thu Jan 16 10:07:33 2020 -0800 Allow Timestamp comparisons against out-of-range values Prior to the related change, clients may have written down X-Delete-At headers that are outside of the Timestamp range, for example. Change-Id: Ib8ae7ebcbdb32e0aa58446bd1ef949e5e2f63e74 Related-Change: I23666ec8a067d829eaf9bfe54bd086c320b3429e Related-Bug: 1821204 Partial-Bug: 1860149 commit 0901464513bc371e5fff2c52004de364d56c271f Author: Tim Burke Date: Wed Jan 15 18:32:19 2020 -0800 Clean up container-sync docs Change-Id: I98ebe15353d675ca00fee387003bf6572ac385e6 commit 9483630ae130c5b2616f8617da25bcb51b647208 Author: Tim Burke Date: Tue Jan 7 21:16:37 2020 -0800 py3: Fix formpost unicode filename issues Previously, we took the native string filename attribute and put it directly in the (WSGI string) PATH_INFO field. Now, we convert it to a WSGI string first. Change-Id: I30e3beb8707b88c36bd3cdc7a0887d069e943ba6 Closes-Bug: #1858259 commit 5b26b749b58f93ee37bbca26ed3f6a13d9a31bc1 Author: Sean McGinnis Date: Sun Jan 12 03:10:25 2020 -0600 Drop use of unittest2 unittest2 was needed for Python version <= 2.6, so it hasn't been needed for quite some time. See unittest2 note one: https://docs.python.org/2.7/library/unittest.html This drops unittest2 in favor of the standard unittest module. Change-Id: I2e787cfbf1709b7f9c889230a10c03689e032957 Signed-off-by: Sean McGinnis commit 61401907911d09fc1d2232118717b048e77a6649 Author: Clay Gerrard Date: Mon Jan 6 16:27:06 2020 -0600 Make FakeSwift query param order agnostic Apparently FakeSwift has always been persnickety about the order of query params of registered responses and the app making the calls. Since query params can often be converted to dictionaries the order of the encoded params should be able to change without effecting the apps dispatching of registered responses. Change-Id: Ied68c9334201a7663e9c85f3bdaa5b0643d4b6db commit 47eb78897d60ee60a4e42953f5efc52a46d6cd18 Author: zhufl Date: Mon Jan 6 10:34:42 2020 +0800 Fix the duplicated words issue like "the the " This is to fix the duplicated words issue, like: * "both of which which must be stripped off" * "In addition the the values set" and so on Change-Id: Id3d84281f15815b4185c76874575e91a3589981b commit b1178b4a967ab621e66356a9e7bca08de5be5e2a Author: Clay Gerrard Date: Fri Dec 20 11:47:13 2019 -0600 Allow reconciler to handle reserved names Change-Id: Ib918f10e95970b9f562b88e923c25608b826b83f commit 4601548dabdec0a4dc89cefba11e963217255be3 Author: Clay Gerrard Date: Tue Dec 31 13:48:13 2019 -0600 Deprecate per-service auto_create_account_prefix If we move it to constraints it's more globally accessible in our code, but more importantly it's more obvious to ops that everything breaks if you try to mis-configure different values per-service. Change-Id: Ib8f7d08bc48da12be5671abe91a17ae2b49ecfee commit 0f13db30b6425b9c973b4add984e22ae56ce84c2 Author: Pete Zaitcev Date: Fri Jan 3 18:34:11 2020 -0600 Import directly from the inside of the ring module An odd thing happened: when my code did only "from swift.common.ring import Ring", and nothing else, the interpreter looped on the CPU. The import from the top of the ring module is our standard protocol for Ring. It causes no problem in places like container updater or list_endpoints.py. It is a big mystery why it causes Python 3.7.3 to loop, and only in specific circumstances. But we noticed that the recursive import is unnecesasry in this case, so an obviously harmless fix exists. Change-Id: I7373bbb0a50d090d6becf141e5832f8ae98381a4 commit 3f8890701259e24ef81c93947faf2d4ccf223c5f Author: Tim Burke Date: Wed Dec 18 15:14:00 2019 -0800 sharding: Better-handle newlines in container names Previously, if you were on Python 2.7.10+ [0], such a newline would cause the sharder to fail, complaining about invalid header values when trying to create the shard containers. On older versions of Python, it would most likely cause a parsing error in the container-server that was trying to handle the PUT. Now, quote all places that we pass around container paths. This includes: * The X-Container-Sysmeta-Shard-(Quoted-)Root sent when creating the (empty) remote shards * The X-Container-Sysmeta-Shard-(Quoted-)Root included when initializing the local handoff for cleaving * The X-Backend-(Quoted-)Container-Path the proxy sends to the object-server for container updates * The Location header the container-server sends to the object-updater Note that a new header was required in requests so that servers would know whether the value should be unquoted or not. We can get away with reusing Location in responses by having clients opt-in to quoting with a new X-Backend-Accept-Quoted-Location header. During a rolling upgrade, * old object-servers servicing requests from new proxy-servers will not know about the container path override and so will try to update the root container, * in general, object updates are more likely to land in the root container; the sharder will deal with them as misplaced objects, and * shard containers created by new code on servers running old code will think they are root containers until the server is running new code, too; during this time they'll fail the sharder audit and report stats to their account, but both of these should get cleared up upon upgrade. Drive-by: fix a "conainer_name" typo that prevented us from testing that we can shard a container with unicode in its name. Also, add more UTF8 probe tests. [0] See https://bugs.python.org/issue22928 Change-Id: Ie08f36e31a448a547468dd85911c3a3bc30e89f1 Closes-Bug: 1856894 commit 286082222dc70da18bbca2e3d18a4e0aeb22cb2c Author: Clay Gerrard Date: Fri Dec 20 14:30:20 2019 -0600 Use less responses from handoffs Since we don't use 404s from handoffs anymore, we need to not let errors on handoffs overwhelm primary responses either Change-Id: I2624e113c9d945542f787e5f18f487bd7be3d32e Closes-Bug: #1857909 commit 55ab08a5fadda87ffc904d9fe0ba1d1e09612ae3 Author: Tim Burke Date: Thu Jan 2 16:21:19 2020 -0800 account-server: Add test for leading delimiter Related-Change: I27701a31bfa22842c272b7781738e8c546b82cbc Related-Change: If912f71d8b0d03369680374e8233da85d8d38f85 Change-Id: I0c850b8ae40d1ab477d0f5d18f92579c457da54e Related-Bug: 1475018 commit e8b654f318cfab485a772e19db33cb2fe4c3d858 Author: Tim Burke Date: Fri Dec 6 23:26:37 2019 +0000 Have slo tell the object-server that it wants whole manifests Otherwise, we waste a request on some 416/206 response that won't be helpful. To do this, add a new X-Backend-Ignore-Range-If-Metadata-Present header whose value is a comma-separated list of header names. Middlewares may include this header to tell object-servers to send the whole object (rather than a 206 or 416) if *any* of the metadata are present. Have dlo and symlink use it, too; it won't save us any round-trips, but it should clean up some object-server logging. Change-Id: I4ff2a178d0456e7e37d561109ef57dd0d92cbd4e commit 40edf07ab347a9a73e944f4a18dd96b504f7e194 Author: Tim Burke Date: Fri Dec 20 21:36:07 2019 +0000 sharding: Let swift-manage-shard-ranges accept a relative path Previously, passing a relative path would confuse the ContainerBroker about which DB files are available, leading to an IndexError when none were found. Just call realpath() on whatever the user provided so we don't have to muck with any of the broker code. Change-Id: Icdf100dfcd006d975b49d151b99aa9272452d013 commit a06c0a47574639a27ea3e4742904c522dbee7d19 Author: Tim Burke Date: Fri Dec 20 06:54:13 2019 +0000 sharder: quote() more Swift paths when logging The lack of quoting gets extra troublesome with reserved names, where messages get truncated. Change-Id: I415901d3a8cd24cb3cedc72235292bb9d1705bbc commit d246bf20ed30047f9b285dcc3378f9c324ed938e Author: Tim Burke Date: Thu Dec 19 22:27:27 2019 -0800 sharding: Tolerate blank limits when listing Otherwise, we can 500 with ValueError: invalid literal for int() with base 10: '' Change-Id: I35614aa4b42e61d97929579dcb16f7dfc9fef96f commit 54312b0b79ef2517f87a84a101507038d833104f Author: Tim Burke Date: Thu Dec 19 22:09:57 2019 -0800 py3: Clean up some needlessly-noisy tracebacks Change-Id: I5b391664b9260c762bd122a92387084c0582a362 Related-Change: I5f3bc6416207cab2c7e3a77ee6689360b55990e7 commit c94f625fe463d808d4bae1137739a4dfcf6f4cd8 Author: Tim Burke Date: Tue Dec 17 15:11:36 2019 -0800 Forward-port stable-release changelog entries Change-Id: Ib2df3ec683c1670cadba72d1fc0d666975f64aab commit 226e2c12012d492969ebce3e12343b007b054c77 Author: Tim Burke Date: Tue Dec 17 14:50:09 2019 -0800 Fix type on py3 This shouldn't have any practical effect, since *any* False-y value for chunk will give us the correct behavior, but let's be consistent in the type used with that name. Change-Id: I1928883792e7a9945f3731ddeea992d14882d679 commit 1f7b97ec0f49b26b8a33d67cf924f5ab8345a614 Author: Tim Burke Date: Tue Nov 19 21:25:45 2019 -0800 Add normalize_etag() helper function ... and drive-by a import rename Co-Authored-By: Clay Gerrard Change-Id: I1eaf075ff9855cfa03e7991bdf33375b0e4397e6 commit b65d8b10c5ea1c897bcb02c97943af82fd27d578 Author: Tim Burke Date: Sat Dec 14 01:15:06 2019 +0000 Early-return on non-Swift get_info requests Change-Id: Iadc61a1c3bcbfbc47f65ec65df36d8da3694ee74 commit 6b33cf99f43892a22f1ab4124b10c49462b9cbc2 Author: Clay Gerrard Date: Wed Dec 11 12:14:50 2019 -0600 WSGI server workers must drop_privledges ... just like they always have and server per port strategy still does. Related-Change-Id: I3e5229d2fb04be67e53533ff65b0870038accbb7 Change-Id: I14e3ed201ceaceef0f8dbc44685395f350a0e7fc commit 05e4bb30b64d223c024127335870fa7c267f08ae Author: Thiago da Silva Date: Wed Dec 11 20:30:38 2019 +1100 Update list of experimental upgrade tests Removed test of upgrade from queens and added train Change-Id: Iadab0f87d6fd66fe49c79ee200d175f9bed92baa commit 8c0fd3f1386cc71c618a6711f03a9f220692bd07 Author: Tim Burke Date: Wed Dec 11 01:07:19 2019 +0000 py3: Make seamless reloads work Starting with Python 3.4, newly-created file descriptors are non-inheritable [0], which causes trouble when we try to use a pipe for IPC. Fortunately, the same PEP that implemented this change *also* provided a new API to mark file descriptors as being inheritable -- so just do that. While we're at it, * Fix up the probe tests to work on py3 * Fix up the probe tests to work when policy-0 is erasure-coded * Decode the bytes read so py3 doesn't log a b'pid' * Log a warning if the read() is empty; something surely went wrong in the re-exec [0] https://www.python.org/dev/peps/pep-0446/ Change-Id: I2a8a9f3dc78abb99bf9cbcf6b44c32ca644bb07b Related-Change: I3e5229d2fb04be67e53533ff65b0870038accbb7 commit da4e07164e09d099b8148df7669cb82862ffdcc1 Author: Tim Burke Date: Thu Dec 5 14:23:42 2019 -0800 Set swift_source more in s3api middleware Change-Id: I89f3a4b5a3a8c160afb298aad726acce09c65265 commit 51b7b110a104806f10c32ddb7c4bf1859b66f1ab Author: Tim Burke Date: Thu Dec 5 14:33:02 2019 -0800 Set swift_source more in versioned_writes Change-Id: Id0b10cb5f1f3e2c39c8d134f55774a36dc1e75df commit c35ab2e1d8da21f5942791c6fd9aac1e66978b74 Author: Tim Burke Date: Thu Dec 5 14:18:18 2019 -0800 Set swift_source in account_quotas middleware Change-Id: Ib54a65a920fc335c1e0a496cb91c0de3003e2cc5 commit f33c061ae92f33ea467c58749d3f15d2b1cc942c Author: Tim Burke Date: Thu Dec 5 09:42:33 2019 -0800 s3api: Better handle 498/429 responses Currently, they just 500 as an unexpected response status. Much better would be S3's '503 Slow Down' response. Of course, that's all dependent on where you place ratelimit in your pipeline -- and we haven't really given clear guidance on that. I'm not actually sure you *want* ratelimit to be after s3api and auth... but if you *do*, let's at least handle it gracefully. Change-Id: I36f0954fd9949d7d1404a0c381b917d1cfb17ec5 Related-Bug: 1669888 commit 698717d886b2b55ea9b490719851c85c20b57240 Author: Clay Gerrard Date: Fri Sep 13 12:25:24 2019 -0500 Allow internal clients to use reserved namespace Reserve the namespace starting with the NULL byte for internal use-cases. Backend services will allow path names to include the NULL byte in urls and validate names in the reserved namespace. Database services will filter all names starting with the NULL byte from responses unless the request includes the header: X-Backend-Allow-Reserved-Names: true The proxy server will not allow path names to include the NULL byte in urls unless a middlware has set the X-Backend-Allow-Reserved-Names header. Middlewares can use the reserved namespace to create objects and containers that can not be directly manipulated by clients. Any objects and bytes created in the reserved namespace will be aggregated to the user's account totals. When deploying internal proxys developers and operators may configure the gatekeeper middleware to translate the X-Allow-Reserved-Names header to the Backend header so they can manipulate the reserved namespace directly through the normal API. UpgradeImpact: it's not safe to rollback from this change Change-Id: If912f71d8b0d03369680374e8233da85d8d38f85 commit a2aaf598520331874c8d49a3230f6d4892fb3207 Author: Clay Gerrard Date: Tue Nov 26 09:51:09 2019 -0600 Fix BadResponseLength error when creating symlink Closes-Bug: #1854074 Change-Id: I587275e68083aaed3c04486049b67791b52562b7 commit e11c58ef892084f3dd1b08e0d0bfd97668705988 Author: Tim Burke Date: Fri Nov 22 17:38:54 2019 +0000 Turn off dns_compliant_bucket_names for ceph tests We get a handful more passing tests that way, following their move to boto3. Change-Id: I73e9c38bde00a7117cec97e98f013e86350aa5be commit b5c9dc1c9f68afe8083b74eaf8080e17a9ef8f8b Author: Tim Burke Date: Tue Nov 19 09:02:27 2019 -0800 py3: Fix s3api header casing Closes-Bug: #1853367 Change-Id: Ifb15ce50fc3bcfda9532a2c3dec542c272ea4933 commit 249a302d0c43bf7d847b3691b795a7ba3c5b6c10 Author: Tim Burke Date: Tue Nov 19 14:57:02 2019 -0800 Remove a bunch of known-failures that moved from boto to boto3 Change-Id: I775a03e0ba1e10982eb6f7ef52be773c8831b1ec commit e5dad445b258b6fd823bddb7b1e3ca10064ba3e8 Author: Tim Burke Date: Tue Nov 19 10:16:28 2019 -0800 Switch py2 DSVM jobs to only run swift under py2 Other services are fine to move on to py3. We still want to test py2, though. Change-Id: I488f88a54948663c682b35dc976b62db6e72eef7 commit 29019985ed63790851df7a2a0ed75593b11906b4 Author: Thierry Carrez Date: Tue Nov 19 17:32:02 2019 +0100 Start README.rst with a better title Now that we are using gitea the contents of our README.rst are more prominently displayed. Starting it with a "Team and repository tags" title is a bit confusing. This change makes it start with the name of the project instead. Change-Id: Iaf6c162b26a32d16a083aa721aa084807af8cd60 commit cfb3ae6019d9a4437675ff9d6161d67071fe528d Author: Tim Burke Date: Fri Nov 15 16:09:21 2019 -0800 Update known-failures and config for up-rev'ed ceph/s3tests Change-Id: I3833843cd8d23c498a2afe6c68a3f0afe26343c0 commit d6db7360e751e12895a5ba7b9d82db2df01fa057 Author: OpenStack Proposal Bot Date: Fri Nov 15 10:25:47 2019 +0000 Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I395f78516b33b675eab4aa7397742bff90e0167a commit da5b94914302128c20b8ae9e3a2752a02ff29e73 Author: Tim Burke Date: Fri Oct 18 15:15:08 2019 -0700 Add py38 unit test job Depends-On: https://review.opendev.org/#/c/688594/ Change-Id: Idf6c492d554a3ac3d38bd9dfc9c89494cae82bb1 commit 3de330ead3d5f68bcd04844ed9616456ee3ebe73 Author: Adrien Pensart Date: Fri Nov 8 12:18:06 2019 +0800 print deleted flag in account and container info cli Change-Id: I37c76e0c58a3d12b2062d5a6dbda2bf074f8bc9a commit 1107f24179c0c6fdcb58771f3a6e6c025352b5d3 Author: Darrell Bishop Date: Fri Oct 25 12:48:36 2019 -0700 Seamlessly reload servers with SIGUSR1 Swift servers can now be seamlessly reloaded by sending them a SIGUSR1 (instead of a SIGHUP). The server forks off a synchronized child to wait to close the old listen socket(s) until the new server has started up and bound its listen socket(s). The new server is exec'ed from the old one so its PID doesn't change. This makes Systemd happier, so a ReloadExec= stanza can now be used. The seamless part means that incoming connections will alwyas get accepted either by the old server or the new one. This eliminates client-perceived "downtime" during server reloads, while allowing the server to fully reload, re-reading configuration, becoming a fresh Python interpreter instance, etc. The SO_REUSEPORT socket option has already been getting used, so nothing had to change there. This patch also includes a non-invasive fix for a current eventlet bug; see https://github.com/eventlet/eventlet/pull/590 That bug prevents a SIGHUP "reload" from properly servicing existing requests before old worker processes close sockets and exit. The existing probtests missed this, but the new ones, in this patch, caught it. New probe tests cover both old SIGHUP "reload" behavior as well as the new SIGUSR1 seamless reload behavior. Change-Id: I3e5229d2fb04be67e53533ff65b0870038accbb7 commit 2f1111a4364858fe87696f748405e6a3ff8f5655 Author: Romain LE DISEZ Date: Fri Nov 1 17:23:31 2019 -0400 proxy: stop sending chunks to objects with a Queue During a PUT of an object, the proxy instanciates one Putter per object-server that will store data (either the full object or a fragment, depending on the storage policy). Each Putter is owning a Queue that will be used to bufferize data chunks before they are written to the socket connected to the object-server. The chunks are moved from the queue to the socket by a greenthread. There is one greenthread per Putter. If the client is uploading faster than the object-servers can manage, the Queue could grow and consume a lot of memory. To avoid that, the queue is bounded (default: 10). Having a bounded queue also allows to ensure that all object-servers will get the data at the same rate because if one queue is full, the greenthread reading from the client socket will block when trying to write to the queue. So the global rate is the one of the slowest object-server. The thing is, every operating system manages socket buffers for incoming and outgoing data. Concerning the send buffer, the behavior is such that if the buffer is full, a call to write() will block, otherwise the call will return immediately. It behaves a lot like the Putter's Queue, except that the size of the buffer is dynamic so it adapts itself to the speed of the receiver. Thus, managing a queue in addition to the socket send buffer is a duplicate queueing/buffering that provides no interest but is, as shown by profiling and benchmarks, very CPU costly. This patch removes the queuing mecanism. Instead, the greenthread reading data from the client will directly write to the socket. If an object-server is getting slow, the buffer will fulfill, blocking the reader greenthread. Benchmark shows a CPU consumption reduction of more than 30% will the observed rate for an upload is increasing by about 45%. Change-Id: Icf8f800cb25096f93d3faa1e6ec091eb29500758 commit 7f6b09dc930706d63a943dc239dff84ddae52014 Author: SeongSoo Cho Date: Thu Nov 7 15:32:21 2019 +0800 Add delete_account in InternalClient Currently, The internal client module is missing a delete_account method. I added a delete_account in InternalClient Change-Id: I97cb090820b4099944ba49eaeb5b02f9bbcf1d58 Closes-Bug: #1680377 commit 30d99f514071a9272caa018d140c9f7a367a7b6f Author: Tim Burke Date: Thu Nov 7 00:15:25 2019 +0000 Get rid of _delete_db It was parctically the same between both, only the table name was different -- and we already have a way to accomodate that. Change-Id: I112bef1cf44c7d6eeaacd30cea27cb71fb2e2406 commit e7cd8df5e93a89fba6a62d2f2f4270a1ce789e93 Author: Clay Gerrard