Reviewed: https://review.openstack.org/323599 Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0330478b70d0a699a0f9c21ef87c7e639d92564b Submitter: Jenkins Branch: feature/hummingbird commit 5fe392b562de3baed080704df433fb392cb4fb31 Author: Ondřej Nový Date: Tue May 31 16:25:50 2016 +0200 Fixed typo Change-Id: I7a35c0076360c7a23cf405189828d3c252ec6708 commit b52eccb3b1ea0591f0040587228d3705b5d3f68d Author: Clay Gerrard Date: Wed May 25 11:21:25 2016 -0700 Clarify overload best practices in admin guide Change-Id: Ib7c08bdeab6374771bb8e2b05053e7e16973524d commit f1fd50723bb84c4941e949895576733f6eb67793 Author: Christian Schwede Date: Wed May 25 09:53:31 2016 +0200 Add dispersion --verbose example to admin guide Change-Id: I5f9cacedde2a329332ccf744800b6f2453e8b28e commit b3ab715c055283ccfea9a504d6da20741d82e7ad Author: Matthew Oliver Date: Wed May 25 14:35:54 2016 +1000 Add ring-builder dispersion command to admin guide This change updates the admin guide to point out the dispersion command in swift-ring-builder and mentions the dispersion verbose table to make it more obvious to operators. Change-Id: I72b4c8b2d718e6063de0fdabbaf4f2b73694e0a4 commit fb7a8e9ab7596a36a6992a3a8f8c6d005a2c2829 Author: Tim Burke Date: Tue May 24 13:37:58 2016 -0700 Add links to mitaka install guides Change-Id: I62331923751c521daded4468b5cc5f03655226bc commit e09c4ee7800e82aa09ca2f6ae375420b766182a4 Author: Tim Burke Date: Fri Apr 29 12:12:00 2016 -0500 Allow concurrent bulk deletes Before, server-side deletes of static large objects could take a long time to complete since the proxy would wait for a response to each segment DELETE before starting the next DELETE request. Now, operators can configure a concurrency factor for the slo and bulk middlewares to allow up to N concurrent DELETE requests. By default, two DELETE requests will be allowed at a time. Note that objects and containers are now deleted in separate passes, to reduce the likelihood of 409 Conflict responses when deleting containers. Upgrade Consideration ===================== If operators have enabled the bulk or slo middlewares and would like to preserve the prior (single-threaded) DELETE behavior, they must add the following line to their [filter:slo] and [filter:bulk] proxy config sections: delete_concurrency = 1 This may be done prior to upgrading Swift. UpgradeImpact Closes-Bug: 1524454 Change-Id: I128374d74a4cef7a479b221fd15eec785cc4694a commit 226557afc42c245e050d84162497f46341407ef7 Author: Tim Burke Date: Thu May 19 18:55:40 2016 -0700 Turn on H703, so our translators don't punch us Change-Id: I4ce3068f79563e4d4296c6e1078bc12f0cf84c96 Related-Bug: 1559431 commit 7b706926a8ed5bbcec3a678e868e301c9a6ed8f1 Author: Alistair Coles Date: Mon May 23 15:20:06 2016 +0100 Fix setup of manifest responses in SLO tests The swift_bytes param is removed from the content-type in the proxy object controller, so the SLO unit tests should not be registering GET responses with FakeSwift that have swift_bytes appended to the content-type. Nor should submanifest segment dicts have swift_bytes appended to their content-type values. Also adds a test for the object controller and container server handling of SLO swift_bytes. Change-Id: Icf9bd87eee25002c8d9728b16e60c8347060f320 commit d0ec1adb78b26f5c24312090796c69912e9e3da9 Author: Alistair Coles Date: Thu May 19 19:58:56 2016 +0100 Make SLO manifest copies retain correct content-type When copying an SLO manifest with multipart-manifest=get the actual manifest content-type should get copied to the destination, rather than the application/json value that is synthesised by SLO in a GET response. That way the result of a HEAD on the copied manifest is the same as a HEAD to the source, and the container listings for the two are consistent. This patch also un-skips a functional test and adds functional tests that verify this patch and also verify that etags and size also get correctly copied and updated in destination container (bug #1260446). Closes-Bug: #1260446 Closes-Bug: #1583756 Change-Id: Ie7fa82f70b3ec3ef568f5355c69f6bce460ba25d commit 1b7001adeb935893017ff2428291d7edf9470b6b Author: Brian Cline Date: Sat May 21 03:54:33 2016 -0500 Improve linting for third-party middleware Currently any third-party CI doing linting of their own middlewares against the Swift codebase don't do so hot -- if they use swob's Response.status_int in any way, the linter will fail because it only gets set when status does. This patch defaults it to None in the class definition, seeing as a linter can't truly know that status_int will only be created lazily at runtime, and further, that its existence is tied to setting Response.status (via __init__ or setting the property directly). I couldn't actually find a code path within Swift where we could end up with a None when we access it; at init time, we default the status code to 200, and there don't seem to be any code paths that init with a None status code. (You might try, but you can't do that even today.) As a result, this seems to have no ill effect on anything, but is a huge help for linters. This doesn't seem to break any unit tests, func tests via v1, v2, and v3 auth, nor tempest tests via v2 or v3. Change-Id: Id28fb56bc7133b9eb59df7989a71b892fd919643 commit 14eb1803e931c5c501d99e612a6d59a57eccf464 Author: Kota Tsuyuzaki Date: Thu May 12 23:57:49 2016 -0700 Fix probe failure and small things This is follow up for https://review.openstack.org/#/c/283351/. Probe fix: - The probe in the patch now fails (sometimes success luckily) because inbound X-Timestamp is deprecated at the change, f581fccf71034818d19062593eeb52a4347bb174, so we can not use X-Timestamp to make an object with arbitrary timestamp anymore from outside of Swift. This patch makes the probe to use internal client to put the objects to make the inconsistent situation. Small things: - Enable expirer split brain test even if we have just one policy. - FAIL rather than ERROR if the object was expired incorrectly - ObjectBrainSplitter now uses the policy set at instance variable in default instead of random choice of ENABLED_POLICIES. Co-Authored-By: Alistair Coles Change-Id: I757dbb0f1906932ef5d508b48b4120f2794b3d07 commit 9ae3ee5a8b6aa169e1f4179cee748c721f480229 Author: Alfredo Moralejo Date: Thu May 19 15:56:15 2016 +0200 Fix locale directory in MANIFEST.in Actual location is swift/locale, not locale. This makes build to not include .mo files if they have been created previously with compile_catalog. Change-Id: I907e9bd6dde57cb9ed718656e56a313e82bad128 Closes-Bug: 1583618 commit 876df35f847e607901e2ed8a344c71c607495705 Author: Matthew Oliver Date: Tue May 17 10:48:25 2016 +1000 disable_fallocate also disables fallocate_reserve Currently when disable_fallocate is true it disables calling the fallocate syscall, but it doesn't disable fallocate_reserve. This patch fixes this. This problem has caused functional tests to fail in our SAIOs, since SAIOs have disable_fallocate set but the fallocate_reserve space free checking was still being run creating 507 responses. This is thanks to the change in fallocate_reserve default changing from 0 to 1%. Because fallocate_reserve and disable_fallocate causes SAIO functional tests to fail a section called 'Known Issues' has been added to the SAIO developer documentation which includes a warning about using fallocate_reserve on SAIOs. Change-Id: I727bfb0861ea26fe2f16ad55f4d36ae088864d8f commit 2744492f30c758a4f076e2b30eaaf1e2e7fa586c Author: Tim Burke Date: Wed Apr 27 13:31:11 2016 -0500 Use the same key for memcache and env['swift.infocache'] When we were caching directly to the WSGI environment, it made sense to have different keys for the different caches. Now that we have a separate data structure for the per-request cache, however, we ought to be consistent. Change-Id: I199cba6e5fc9ab4205bba369e6a2f34fc5ce22d4 commit 4248123169f88bf62dbc5413ae2c15d53010482e Author: venkatamahesh Date: Sun May 15 21:27:46 2016 +0530 Remove the invalid project link swiftsync project is no longer maintained and so is removed from the list Change-Id: Ieae67c728da16e5516babd33d2e5937b14738d6a Closes-Bug: #1581975 commit 1adc6047f0dc56caffb3ab552de948c720b50682 Author: Brian Ober Date: Fri May 13 16:43:50 2016 -0500 Bypass Account Mgmt Functional Cases When Disabled The testPUT case is failing when keystone was enabled and allow_account_management is set to True. There were a few issues needing addressed. First the case was renamed to call out what it was actually doing which is verifying an error scenario for which a PUT on a storage account was not allowed. Second the case was running even when allow_account_management is enabled, which is incorrect. It "accidently" works with TempAuth because it requires a reseller permission, so the Keystone failure here has more to do with not requiring a reseller permission to do a PUT on a storage account for which a user has an operator role on. The common sense fix here is to not execute this test case when allow_account_management is enabled. Change-Id: Id29f5ca48f92cd139535be7064107b8a61b02856 commit 1c88d2cb818cec07d12ac17be5820cdd769aea5d Author: Samuel Merritt Date: Thu Feb 11 15:51:45 2016 -0800 Fix up get_account_info and get_container_info get_account_info used to work like this: * make an account HEAD request * ignore the response * get the account info by digging around in the request environment, where it had been deposited by elves or something Not actually elves, but the proxy's GETorHEAD_base method would take the HEAD response and cache it in the response environment, which was the same object as the request environment, thus enabling get_account_info to find it. This was extraordinarily brittle. If a WSGI middleware were to shallow-copy the request environment, then any middlewares to its left could not use get_account_info, as the left middleware's request environment would no longer be identical to the response environment down in GETorHEAD_base. Now, get_account_info works like this: * make an account HEAD request. * if the account info is in the request environment, return it. This is an optimization to avoid a double-set in memcached. * else, compute the account info from the response headers, store it in caches, and return it. This is much easier to think about; get_account_info can get and cache account info all on its own; the cache check and cache set are right next to each other. All the above is true for get_container_info as well. get_info() is still around, but it's just a shim. It was trying to unify get_account_info and get_container_info to exploit the commonalities, but the number of times that "if container:" showed up in get_info and its helpers really indicated that something was wrong. I'd rather have two functions with some duplication than one function with no duplication but a bunch of "if container:" branches. Other things of note: * a HEAD request to a deleted account returns 410, but get_account_info would return 404 since the 410 came from the account controller *after* GETorHEAD_base ran. Now get_account_info returns 410 as well. * cache validity period (recheck_account_existence and recheck_container_existence) is now communicated to get_account_info via an X-Backend header. This way, get_account_info doesn't need a reference to the swift.proxy.server.Application object. * both logged swift_source values are now correct for get_container_info calls; before, on a cold cache, get_container_info would call get_account_info but not pass along swift_source, resulting in get_account_info logging "GET_INFO" as the source. Amusingly, there was a unit test asserting this bogus behavior. * callers that modify the return value of get_account_info or of get_container_info don't modify what's stored in swift.infocache. * get_account_info on an account that *can* be autocreated but has not been will return a 200, same as a HEAD request. The old behavior was a 404 from get_account_info but a 200 from HEAD. Callers can tell the difference by looking at info['account_really_exists'] if they need to know the difference (there is one call site that needs to know, in container PUT). Note: this is for all accounts when the proxy's "account_autocreate" setting is on. Change-Id: I5167714025ec7237f7e6dd4759c2c6eb959b3fca commit ce90a1e79e16472ff75d205c8d907a5991888b3a Author: Samuel Merritt Date: Thu Jan 21 13:19:30 2016 -0800 Make info caching work across subrequests Previously, if you called get_account_info, get_container_info, or get_object_info, then the results of that call would be cached in the WSGI environment as top-level keys. This is okay, except that if you, in middleware, copy the WSGI environment and then make a subrequest using the copy, information retrieved in the subrequest is cached only in the copy and not in the original. This can mean lots of extra trips to memcache for, say, SLO validation where the segments are in another container; the object HEAD ends up getting container info for the segment container, but then the next object HEAD gets it again. This commit moves the cache for get_*_info into a dictionary at environ['swift.infocache']; this way, you can shallow-copy the request environment and still get the benefits from the cache. Change-Id: I3481b38b41c33cd1e39e19baab56193c5f9bf6ac commit 30e39cc9fa6353be37cd3b5a515f09b42ceb3b20 Author: Samuel Merritt Date: Wed May 4 12:02:07 2016 -0700 Skip SLO-reconciling probe test when SLO is off The probe test in question tries to make a manifest referencing unavailable objects and expects that to fail with a 400. If the SLO middleware is enabled, then it checks the segments, can't find some, and returns the 400. If it's disabled, the PUT succeeds and makes an object whose contents are some JSON blob. In the latter case, the probe test would fail because it expected a real SLO manifest but didn't find one. Now we skip the remainder of the test when we detect that SLO is not enabled. Change-Id: I3e7e8e98107608e675efc24156e703bc167458bb commit 721c788b9a0ceb22ffac0b2c797890898096a9f4 Author: Kota Tsuyuzaki Date: Thu May 12 18:57:31 2016 -0700 Import BrainSpliter directly in expirer probe Change-Id: Ib3cfe38d1b17c09c7248b5cbcfe4e2eadff7acd9 commit eda1b330f3cf7bf9dd643854d4a221b6f024884d Author: Paul Dardeau Date: Thu May 12 18:58:15 2016 +0000 fix docs word usage for large obj copy Change-Id: I2b6ca3f9277b344ff5b05a85304084a7edcbcb66 commit 6834547f668a4649f5a42b8a4edc43b844fd6bbe Author: Samuel Merritt Date: Fri Apr 22 14:55:35 2016 -0700 Clean up fallocate tests a little Change-Id: I01f1ad8ef0f8910718fd2fb30c9e8285358baf84 commit b430c384db38ea1153f52d0accf0561dac1378a6 Author: Alistair Coles Date: Thu May 12 14:57:56 2016 +0100 Put back missing sysmeta unit tests We lost some unit tests from test_sysmeta.py in [1]. This patch restores them. Since the proxy no longer implements post-as-copy or COPY then we need to plumb in a copy middleware to the test app in test_sysmeta. It is a little odd perhaps to have these tests under test/unit/proxy but the alternative is to duplicate the test setup and base test code that already exists in test_sysmeta.py into test_copy.py. [1] commit 46d61a4dcd9a5d9157625c06d6fe7d916e80c3d2 (server side copy middleware) Change-Id: Iec02387ccbddbe3841a417880389c707cd5c0346 commit ce022f4417e973215b563412b504a8a1b9ce240c Author: Saverio Proto Date: Thu May 12 15:30:17 2016 +0200 README: fix broken link There is a broken link to CONTRIBUTING.rst introduced in commit a829bd59770681f9d6c1ef02a6e1d5e441587a23 Change-Id: Iaedf5ff3995229cf292202793809080f9f2c7fed commit 9729bc83eb6845e65f8244b4104b2c1678f8fe38 Author: Christian Schwede Date: Mon Apr 11 11:56:07 2016 +0200 Don't delete misplaced dbs if not replicated If one uses only a single replica and a database file is placed on a wrong partition, it will be removed instead of replicated to the correct partition. There are two reasons for this: 1. The list of nodes is empty when there is only a single replica 2. all(responses) is True even if there is no response at all, and the latter is always True if there is no node to replicate to. This patch fixes this by adding a special case if used with only one replica to the node selection loop and ensures that the list of responses is not empty. Also adds a test that fails on current master and passes with this change. Closes-Bug: 1568591 Change-Id: I028ea8c1928e8c9a401db31fb266ff82606f8371 commit 92274d479010f691004b1885e354224dc9c3b61c Author: Andreas Jaeger Date: Mon Mar 28 18:21:22 2016 +0200 List system dependencies for running common tests Add an other-requirements.txt file containing a cross-platform list of dependencies needed for running included tox-based tests. Also include a tox environment for convenience calling the bindep[*] utility to list any missing system requirements. Document bindep and other-requirements usage. This change is self-testing. For other-requirements.txt see also http://docs.openstack.org/infra/manual/drivers.html#package-requirements [*] http://docs.openstack.org/infra/bindep/ Change-Id: Iea6f5fecba3b7cb9f6dac7029c0f17fc31fc0e3c commit ad16e2c77bb61bdf51a7d3b2c258daf69bfc74da Author: Tim Burke Date: Wed May 11 19:54:47 2016 -0700 Stop complaining about auditor_status files Following fd86d5a, the object-auditor would leave status files so it could resume where it left off if restarted. However, this would also cause the object-reconstructor to print warnings like: Unexpected entity in data dir: u'/srv/node4/sdb8/objects/auditor_status_ZBF.json' ...which isn't actually terribly useful or actionable. The auditor will clean it up (eventually); the operator doesn't have to do anything. Now, the reconstructor will specifically ignore those status files. Change-Id: I2f3d0bd2f1e242db6eb263c7755f1363d1430048 commit 46d61a4dcd9a5d9157625c06d6fe7d916e80c3d2 Author: Prashanth Pai Date: Wed Feb 18 11:59:31 2015 +0530 Refactor server side copy as middleware Rewrite server side copy and 'object post as copy' feature as middleware to simplify the PUT method in the object controller code. COPY is no longer a verb implemented as public method in Proxy application. The server side copy middleware is inserted to the left of dlo, slo and versioned_writes middlewares in the proxy server pipeline. As a result, dlo and slo copy_hooks are no longer required. SLO manifests are now validated when copied so when copying a manifest to another account the referenced segments must be readable in that account for the manifest copy to succeed (previously this validation was not made, meaning the manifest was copied but could be unusable if the segments were not readable). With this change, there should be no change in functionality or existing behavior. This is asserted with (almost) no changes required to existing functional tests. Some notes (for operators): * Middleware required to be auto-inserted before slo and dlo and versioned_writes * Turning off server side copy is not configurable. * object_post_as_copy is no longer a configurable option of proxy server but of this middleware. However, for smooth upgrade, config option set in proxy server app is also read. DocImpact: Introducing server side copy as middleware Co-Authored-By: Alistair Coles Co-Authored-By: Thiago da Silva Change-Id: Ic96a92e938589a2f6add35a40741fd062f1c29eb Signed-off-by: Prashanth Pai Signed-off-by: Thiago da Silva commit cb8cda6588735411c542a1e4d78315ee047a357b Author: Paul Dardeau Date: Mon May 9 20:28:42 2016 +0000 resurrect gholt blog posts on building consistent hashing ring These are blog posts authored by Greg Holt (gholt) and used with permission to add here. Content was only reformatted as rst and wrap prose lines at 70 characters. Change-Id: I7aa47c24b5019aa598ee005e01612a49514da25f commit ba1a568f815a8a40bf75c0beb96b789bd07adcb6 Author: Alistair Coles Date: Wed Mar 9 10:48:54 2016 +0000 Rename hash_cleanup_listdir tests hash_cleanup_listdir was removed in [1], this patch renames all references to it in test_diskfile to refer to the cleanup_ondisk_files method that is now tested directly. Also remove the final references to the now non-existent function in a few comments. [1] I0b96dfde32b4c666eebda6e88228516dd693ef92 Change-Id: I1e151799fc2774de9a1af092afff875af24a630c Related-Bug: #1550569 commit 6827affe62771949a041d958db17ba6cd29232ab Author: John Dickinson Date: Tue Apr 5 11:35:22 2016 -0700 Rework the contributor docs This started as a new "new_contributor" doc. But we've already got at least 3 different docs like that. Change-Id: Ia2303ab55eeea01cc71acbccaeab55dad0ef5ff9 commit deaef2f9d62bb8ae7f4e8538e9fd9ddffbb8ebb2 Author: Samuel Merritt Date: Wed May 4 16:48:28 2016 -0700 Remove unneeded setting of SO_REUSEADDR. This has been in eventlet.listen() since version 0.15. Change-Id: Ib77b28231a2180f1ea082f356c4687c39681a6f7 commit b3dd6a5df1ac4a5265b330787abcd52c7e116208 Author: Christopher Bartz Date: Tue May 3 14:33:05 2016 +0200 Improves log message in swift.obj.replicator Before this commit, when a local device has not found been found in a object-replication run, the policy was not mentioned in the error log. But it is of interest to know the policy, for example for error searching, when no local device has been found. Change-Id: Icb9f9f1d4aec5c4a70dd8abdf5483d4816720418 commit 99305b93000f28530731c05b34c69e0c7ce0463f Author: Samuel Merritt Date: Mon May 2 17:29:32 2016 -0700 Fix probe tests from commit cf48e75 Commit cf48e75 changed the default account/container/object ports in a lot of places, including the probetests. However, it didn't change them in doc/saio/bin/remakerings, and since the probe tests must match the rings, they started failing. This commit just backs out the changes to the test/probe directory so that remakerings and the probe tests match again. Change-Id: I316a09e6ee1a911f37ce9df3d641644739f88eeb commit cf48e75c259350daa667155e3ffd38b17946bf44 Author: Shashirekha Gundur Date: Mon Feb 1 18:06:54 2016 +0000 change default ports for servers Changing the recommended ports for Swift services from ports 6000-6002 to unused ports 6200-6202; so they do not conflict with X-Windows or other services. Updated SAIO docs. DocImpact Closes-Bug: #1521339 Change-Id: Ie1c778b159792c8e259e2a54cb86051686ac9d18 commit 4c11833a9cbff499725365e535e217f3eae3c442 Author: Christian Schwede Date: Thu Apr 28 11:53:51 2016 -0500 Remove ThreadPool class With the removement of threads_per_disk there is no longer a need to use run_in_thread() at all; it was just calling the function itself when running with 0 threads. Similar to force_run_in_thread() - with 0 threads it was basically doing the same like in tpool_reraise(), therefore replacing the call and finally removing the complete ThreadPool class. Note that this might break external consumers that are inheriting BaseDiskFileManager; in this case you need to adopt this change in your codebase then. Change-Id: I39489dd660935bdbfbc26b92af86814369369fb5 commit 33fdd0a356d83f12f63b9a94ff423ca815c96066 Author: Bryan Keller Date: Fri Apr 22 10:03:23 2016 -0700 SwiftLogFormatter will log transaction IDs on INFO level Previously SwiftLogFormatter would make two checks. One to see if the transaction id was already in the message field and another check to make sure the log level wasn't set to info. If either of these was true, then it would not log the transaction ID in the transaction ID field. This commit removes the check for the info log. Now transaction IDs will be recorded in all cases that have them. Change-Id: Ic06538ab55a75d298169ae1745671573ee9c09e8 Closes-Bug: #1504344 commit c6ac69c552f87d07cd98b6e3f54258913bdc90db Author: ZhiQiang Fan Date: Fri Apr 29 20:39:02 2016 +0800 [Trivial] Remove unnecessary executable privilege swift/common/storage_policy.py is not required to be executable. Change-Id: I733d70a88be25b32d8caf590dee4c53d14757fef commit 96c1838a15645639154f1048285d53a891809937 Author: Brian Cline Date: Fri Apr 29 05:00:36 2016 -0500 Update mailmap, resolve identity crisis Change-Id: Ia8e30a8c0f81068c259d5d08adad4f1a98460561 commit a67d91987dff4859924aeaa3ca8fe30190b4c210 Author: Brian Cline Date: Fri Apr 29 04:01:39 2016 -0500 Fix minor typos in review guidelines Change-Id: I3acb3daeacb784c4038bf54d4103fab66c52f41a commit 9d6a055b31f267e2380d0470021cac44cdf36203 Author: Christian Schwede Date: Thu Apr 28 12:06:24 2016 -0500 Remove threads_per_disk setting This patch removes the threads_per_disk setting. It was already a deprecated setting and by default set to 0, which effectively meant to not use a per-disk thread pool at all. Users are encouraged to use servers_per_port instead. DocImpact Change-Id: Ie76be5c8a74d60a1330627caace19e06d1b9383c commit a1ed76f1cbb3dcd5b4253920278b8bc4f32978a2 Author: ZhiQiang Fan Date: Thu Apr 28 06:29:41 2016 +0800 Fix doc build if git is absent When building packages if git is absent, then we should not set html_last_updated_fmt. It can still be set via the -D switch when building with sphinx-build. Change-Id: I5d0b6cc87f27a052d6d0265546c0d347f00c4bb8 Closes-Bug: #1552251 commit 00dd89fe6936e4c2da667b496a9996638c26196b Author: Clay Gerrard Date: Thu Mar 3 06:08:12 2016 +0000 adding review guidelines Change-Id: I61304856a4ecccbbf3aa06c30822494592a3b3d5 commit 29544a9e175b1ec9e0dbfd5288edc00a1402d5ca Author: Samuel Merritt Date: Wed Apr 27 16:59:00 2016 -0500 Use smaller quorum size in proxy for even numbers of replicas Requiring 2/2 backends for PUT requests means that the cluster can't tolerate a single failure. Likewise, if you have 4 replicas in 2 regions, requiring 3/4 on a POST request means you cannot POST with your inter-region link down or congested. This changes the (replication) quorum size in the proxy to be at least half the nodes instead of a majority of the nodes. Daemons that were looking for a majority remain unchanged. The container reconciler, replicator, and updater still require majorities so their functioning is unchanged. Odd numbers of replicas are unaffected by this commit. Change-Id: I3b07ff0222aba6293ad7d60afe1747acafbe6ce4 commit 5e420efc728b9f02a3f15eb312ebfa29e2e221ff Author: Tim Burke Date: Thu Jan 7 15:46:17 2016 -0800 Fix versioned_writes functional test skipping Previously, if object versioning was enabled via the old-style allow_versions container-server setting rather than the new-style allow_versioned_writes proxy-server setting, TestCrossPolicyObjectVersioning would skip tests while TestObjectVersioning and TestObjectVersioningUTF8 would run them. Additionally, if versioned_writes was explicitly included in the proxy-server's pipeline and allow_versioned_writes was disabled, the functional tests would fail with a 412. Now, all three will use the same logic to check whether versioning is enabled. Specifically, they will all try to set an X-Versions-Location header and skip if it doesn't stick. Additionally, the TestCrossPolicyObjectVersioningEnv will now properly clean up after itself. Change-Id: I4c788a0e18587ff17d3c6e346fd22b881495f06d commit 1a18079a15c2630e602c7c156a30e10e5f884d4b Author: Denis V. Meltsaykin Date: Thu Apr 7 23:07:13 2016 +0300 Change pids' type to set Previously, the pids type was list(), which raised a ValueError in case if pid was not in pids during the pids.remove(pid) call. The fix changes the type of pids to set() and substitutes remove() with discard(), which doesn't raise an Exception if a value is not in the set. Change-Id: I8d41cb2a8ec14d29e5c7411ddfe48ae7a41deb45 Closes-Bug: #1567638 commit 0da9da513101c172d8923469e40f9c15b3518e73 Author: Andy McCrae Date: Thu Mar 3 11:14:39 2016 +0000 Allow fallocate_reserve to be a percentage Add the ability to set the fallocate_reserve value as a percentage. This happens automatically when adding the '%' at the end of the value. Having the ability to set a % of free space rather than a byte value is useful especially when drive sizes are heterogenous. The default for fallocate_reserve has been adjusted to 1%, having the fallocate_reserve set seems sensible for all deploys and percentages are far safer to default than byte values (across drives of any size). Tests added for using fallocate_reserve as a percentage. Duplicate tests for fallocate_reserve have been removed. Docs updated to reflect the fallocate_reserve change. Change-Id: I4aea613a708205c917e81d6b2861396655e73238 commit 7d7eaab5afa4c36f0ac8467784138fc423f6ac4f Author: OpenStack Proposal Bot Date: Mon Apr 18 06:31:34 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I88be5c9dbb1fcc3a15592d42af7160478308b1f4 commit ca304cd08e9f8d37e4027f2f71dd77ebba3a30f9 Author: Samuel Merritt Date: Fri Apr 15 17:22:44 2016 -0700 Ignore negative suffix-byte-range requests. If the client asked for "Range: bytes=--123", Swift would respond with a 206 and a Content-Length of -123. Now that Range header is ignored just like all kinds of other invalid Range headers. Change-Id: I30d4522d223076ce342d20c52f57ff0eb2aea1f4 Closes-Bug: 1571106 commit b12222548de012ed496fba70c3d12aa31f003e3d Author: Tim Burke Date: Fri Apr 15 16:08:26 2016 -0700 Tighten up ResumingGetter.fast_forward When the current range is a "last N bytes" request, properly raise HTTPRequestedRangeNotSatisfiable/RangeAlreadyComplete errors. Also, add a couple tests for RangeAlreadyComplete with "normal" range requests. Change-Id: Icbfd621e7160747c5dfaf9f189da7d74e45a5347 commit 3bc33e59aaeefdb940794782a36517c2b4eb0642 Author: Samuel Merritt Date: Thu Apr 14 20:01:38 2016 -0700 Fix download resumption after getting no data. When the proxy is handling an object GET response and an object server fails to send data in a timely fashion, the proxy can pick up where it left off with another object server; other than a pause in the download, the client doesn't even know anything happened. However, if the proxy received the GET response headers but no data, it would resume at the wrong spot. In particular, for an N-byte object, it would ask the second object server for the last N-1 bytes (or equivalently, all but the first byte). For a replicated storage policy, this would result in the client getting an abbreviated download, while for an EC storage policy, the proxy would 500 after trying to decode a fragment set with a bogus fragment in it. This commit fixes the resumption logic to ask for all N bytes of the object from the second object server. Change-Id: Ib9e28c3dceaded1708e7a30844b534566c7a320c commit 746d928a875281a7154dcd438f46a58bbf656db9 Author: Dmitriy Ukhlov Date: Fri Apr 8 16:00:16 2016 +0300 Adds eventlet monkey patching of select module if thread is pathed Oslo.messaging pika driver requires patching of select module if thread is patched. Pika driver uses select call and if it is not patched onsuming messages blocks whole eventlet loop Closes-Bug: #1570242 Change-Id: I9756737309f401ebddb7475eb84725f65bca01bf commit c96d5c671db9c96f65067d93c0a307cf4b7d91b4 Author: oshritf Date: Thu Feb 18 14:50:08 2016 +0200 Per container stat. report In addition to the container sync stat. report, keeping per container statistics allows administrator with more control over bytes transfered over a specific time per user account: The per container stats are crucial for billing purposes and provides the operator a 'progress bar' equivalent on the container's replication status. Change-Id: Ia8abcdaf53e466e8d60a957c76e32c2b2c5dc3fa commit be84b03a07892f4972dd59309ad261fc72bd7ede Author: dharmendra Date: Thu Apr 14 16:41:09 2016 +0530 Removing unused clause Removing unused clause from common/middleware/dlo.py Change-Id: I7de9aaefd203c4f1be00ee74b89b5184fd419469 commit fb3692c9bb662d9cadc4238920f86676857a7f1f Author: Tim Burke Date: Wed Apr 13 11:07:44 2016 -0700 Don't include conditional headers in SLO HEAD requests Previously, attempting to PUT a SLO manifest with `If-None-Match: *` would include the header when validating the segments, causing the upload to fail. Now when SLO validates segments, no conditional headers will be included in the HEAD request. Change-Id: I03ad454092d3caa73d29e6d30d8033b45bc96136 Closes-Bug: #1569253 commit 91f980314f305c536e6e7c8607b8b52693c865f5 Author: John Dickinson Date: Wed Apr 13 11:57:45 2016 -0500 fix fallocate_reserve traceback Previously, fallocate_reserve could result in a traceback. The OSError being raised didn't have the proper errno set. This patch sets the errno to ENOSPC. Change-Id: I017b0584972ca8832f3b160bbcdff335ae9a1aa6 commit 0e7fca576cee81dd6ca8774760cb880c3fff9c1c Author: Tin Lam Date: Fri Apr 8 18:17:44 2016 -0500 Convert README.md to README.rst Change-Id: I223890bd4ffe469becc2127f9362243cdb52bc08 Closes-Bug: #1567026 commit 22933f5b55d3b8ba13b19c7004d24dfd28378f32 Author: Daisuke Morita Date: Mon Feb 22 18:03:48 2016 -0800 Fix bug expirer unexpectedly deletes object created after x-delete-at As reported at bug/1546067, expirer might accidentally deletes an object which is created after x-delete-at timestamp. This is because expirer sends a request with "X-Timestamp: " and tombstone is named as .ts so if object creation time is between x-delete-at and expirer's DELETE request x-timestamp, the object might be hidden by tombstone. This possibility can be simply removed if the value of x-timestamp which an expirer sends is the same timestamp as x-delete-at of an actual object. Namely, expirer pretends to delete an object at the time an user really wants to delete it. Change-Id: I53e343f4e73b0b1c4ced9a3bc054541473d26cf8 Closes-Bug: #1546067 commit b13a85367ef7f5fd64e285174859025d34b4e451 Author: Thiago da Silva Date: Mon Dec 21 16:25:45 2015 -0200 decouple versioned writes from COPY This change removes the use of the COPY request in the versioned writes middleware. It changes the COPY verb for GETs and PUTs requests. The main reasoning for this change is to remove any dependency that versioning had on copy, which will allow for the COPY functionality to be moved to middleware and to be to the left of the versioned writes middleware in the proxy pipeline. In this way, no COPY request will ever arrive at the versioned writes middleware. A side benefit of this change is that it removes a HEAD request from the PUT path. Instead of checking if a current version exists, a GET request is sent, in case of success, a PUT is sent to the versions container. A unit test was removed that tested non-default storage policies. This test is no longer necessary, since it was used to test specific policy handling code in the COPY method in the proxy object controller. Closes-Bug: #1365862 Change-Id: Idf34fa8d04ff292df7134b6d4aa94ff40887b3a4 Co-Authored-By: Alistair Coles Co-Authored-By: Janie Richling Co-Authored-By: Kota Tsuyuzaki Signed-off-by: Thiago da Silva commit 33f06dc48f7bec2e128b44427fb429ad640cd486 Author: Ondřej Nový Date: Sat Apr 9 18:47:58 2016 +0200 Fixed Sphinx errors doc/source/deployment_guide.rst:1372: ERROR: Malformed table. swift/obj/diskfile.py:docstring of swift.obj.diskfile.BaseDiskFileManager.yield_hashes:13: ERROR: Unexpected indentation. doc/source/ops_runbook/diagnose.rst:188: WARNING: Inline emphasis start-string without end-string. Change-Id: Id20eb62eb5baebb3814e7af5676badb94f17dee5 commit a829bd59770681f9d6c1ef02a6e1d5e441587a23 Author: Tin Lam Date: Fri Apr 8 18:40:30 2016 -0500 Convert CONTRIBUTING.md to CONTRIBUTING.rst Change-Id: I64c42c42db35a9f55a1df9d4ab6e97a2506b8c45 Closes-Bug: #1567027 commit a057c409ec8a23290bc72c4fa45d55a1178f4828 Author: OpenStack Proposal Bot Date: Fri Apr 8 07:02:33 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I9f4330ec20463e4d303e8ba3b67f86813a914ac5 commit d09ef0da62b64067b04a980c643f77526a9078ac Author: Alistair Coles Date: Wed Apr 6 15:40:42 2016 +0100 Assert that ChunkWriteTimouts are not raised Follow up for change Ibbc89449e7878fc4215e47e3f7dfe4ae58a2d638 to add a test assertion that the ChunkWriteTimeout contexts are exited without raising the timeout exception in iter_bytes_from_response_part(). Change-Id: I6d323cb26779e457fb5940093a81b349b333a0af commit 7c0f58ec2ed020186ca3f269153b184fc02bf37a Author: OpenStack Proposal Bot Date: Thu Apr 7 07:00:08 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: Ib80e3a759fa1e4a99576710607ad07fc5f259527 commit edc413b85ec2b703d7506be9c4801eb347611c58 Author: Nguyen Hung Phuong Date: Thu Apr 7 13:31:26 2016 +0700 Fix typos in Swift files Change-Id: I39dbf55c094c42347b57ef67520abff9e6fc24bc commit 95efd3f9035ec4141e1b182516f040a59a3e5aa6 Author: Samuel Merritt Date: Wed Mar 23 13:51:47 2016 -0700 Fix infinite recursion during logging when syslog is down Change-Id: Ia9ecffc88ce43616977e141498e5ee404f2c29c4 commit 0bf518e3b0eeaf66653db6972525701cacfe6333 Author: Thiago da Silva Date: Wed Apr 6 16:58:36 2016 -0400 remove unused current_status method Change-Id: I574919eaa14cadc800f3a1f6014221ee382ee7e0 Signed-off-by: Thiago da Silva commit 5d56f40f04fa9211a2e392d667fe395f38d2cca5 Author: Alistair Coles Date: Wed Apr 6 11:48:48 2016 +0100 Make DirectClientException report correct ip and port When direct_client.direct_get_suffix_hashes raises a DirectClientException the exception message and variables should report the replication_ip and replication_port, as opposed to the ip and port values reported for all other case when the exception is raised. Add option to override ip and port reported in DirectClientException. Also adds unit tests to verify both cases. Related-Bug: 1566395 Change-Id: If3d952847c7199f4e9f6164858085367266386d2 commit e15bceaa7e541c77f26a1f11ee2cbddbc871cbf1 Author: Kota Tsuyuzaki Date: Mon Dec 21 03:13:50 2015 -0800 Refactor CORS unit tests This is a follow-up patch for https://review.openstack.org/#/c/258392/ That one added good unit test cases for various kinds of allowe_origin like '*' or ''(empty). However, the result of handling in Swift proxy will depend on strict_cors_mode option configuration. This patch refactors the unit tests to split out for strict_cors_mode = on/off and add some missing unit tests for each case. Change-Id: I55f7cd279436b5c9f71d81fecf06021380e35579 commit 950b601a9c9e87661e35c6ed7a97ae9611560bc5 Author: Sivasathurappan Radhakrishnan Date: Tue Apr 5 22:45:17 2016 +0000 Modified REPLICATE request to use replication_ip direct_client.direct_get_suffix_hashes doesn't use replication ip and port for REPLICATE request. Since we have an option of doing replication in separate network, we can add replication_ip and port while creating rings if not it will get filled in with the regular node's ip. Change-Id: I34067df27042fc3146b795191ab8043ee1aed3ce Closes-Bug:1566395 commit a460e5affe7089437867afa1ed49ea677c5f4e34 Author: Thiago da Silva Date: Mon Apr 4 16:27:14 2016 -0400 added javaswift to associated projects Change-Id: I139ac3acb1c7d2498e87df554b81824ada2dbd00 Signed-off-by: Thiago da Silva commit 4e412e0cd877be0da2b2716a301d1ae2b4c29a13 Author: Cheng Li Date: Sun Apr 3 21:07:00 2016 +0800 Fix strings with positional arguments As descriped in develop guideline, Any message with more than one variable should use named interpolation instead of positional http://docs.openstack.org/developer/oslo.i18n/guidelines.html#adding-variables-to-translated-messages Change-Id: Ia25517cfb52037cf49d0a6ebf528344267fb22cd Closes-bug:#1559431 commit ee6af69af429e1a278581531e509be92774f668d Author: KATO Tomoyuki Date: Fri Apr 1 23:37:25 2016 +0900 [docs] Update Administrator Guide URL Change-Id: I7f6f4ffb7033d2a321362c91e05f1576847939b0 commit ebf0b220127b14bec7c05f1bc0286728f27f39d1 Author: Tim Burke Date: Wed Mar 30 14:19:00 2016 -0700 Fix upgrade bug in versioned_writes Previously, versioned_writes assumed that all container servers would always have the latest Swift code, allowing them to return reversed listings. This could cause the wrong version of a file to be restored during rolling upgrades. Now, versioned_writes will check that the listing returned is actually reversed. If it isn't, we will revert to getting the full (in-order) listing of versions and reversing it on the proxy. Change-Id: Ib53574ff71961592426cb386ef00a75eb5824def Closes-Bug: 1562083 commit 59bbe27fb0a40236108f09c9b3349e8faef0a95c Author: Nguyen Hung Phuong Date: Wed Mar 30 11:07:46 2016 +0700 Fix typos in Swift files Change-Id: I34e0c9a888127704ac1910e73ddd14e27ebade13 commit 7be55acf1bc4aa07d81b30fd93e144700889898d Author: Tim Burke Date: Thu Feb 11 16:00:38 2016 -0800 Simplify policy-name validation slightly _validate_policy_name always either returns True or raises an exception. Simplify it to just being a callable that may raise an exception. Also, move the check for blank/None names into _validate_policy_name, so it will be applied in more cases. Change-Id: I7832a0c9c895cd75ba4c6d0e8b5568a3c8a0ea25 commit 5902015fa8495ec0ef3c1ab92ae9a34c5bda4334 Author: OpenStack Proposal Bot Date: Sat Mar 26 06:35:18 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I3b5d401649fa3dea6dc43654516f7075bb06ee0d commit 7402d7d9cf91ed00a9c86fc1511592ac2eed5d6c Author: Clay Gerrard Date: Thu Mar 24 16:42:21 2016 -0700 Shutdown backend EC connection contexts on disconnect When eventlet.wsgi closes an ECAppIter on client disconnect we need to make sure our sub-iterators are also closed. We already kill the backend sockets, but the executing contexts are left running until they timeout. A slow client can result in needlessly holding queued backend fragments until the client_timeout (default 60s). Update associated test that exposed the problem to demonstrate the issue more quickly. Change-Id: Ibbc89449e7878fc4215e47e3f7dfe4ae58a2d638 commit 2f7d0f4a2ad2da7e6a35e5b054a47a2fafe5ed01 Author: Anh Tran Date: Fri Mar 25 11:44:26 2016 +0700 Removing some redundant words This patch removes some redundant words. Change-Id: Ia79717664b06ed9a41c3c5dcf1a25e9e49e21cf2 commit 925546ae8a211b50cf7fad6634d47fd1dbfeb58e Author: OpenStack Proposal Bot Date: Fri Mar 25 06:36:40 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I6ba2f35913e6ae83607b5e268645432d455d587c commit e97c4f794db7292a91135adb3fc3e1d6cce4dc84 Author: Janie Richling Date: Sat Jan 16 21:59:20 2016 -0600 swift-[account|container]-info when disk is full Extended the use of the DatabaseBroker "stale_reads_ok" flag to the AccountBroker and ContainerBroker. Now checks for an sqlite3 error from the _commit_puts call that processes the pending files. If this error is raised, then the stale_reads_ok flag will be checked to determine how to proceed as opposed to simply raising. The first time that print_info is attempted, the flag will be false, but swift-[account|container]-info will check for the raised exception. If it was raised, then a warning is reported that the data may be stale, and another attempt will be made using the stale_reads_ok=True flag. Change-Id: I761526eef62327888c865d87a9caafa3e7eabab6 Closes-Bug: 1531302 commit 3407d737c705a7afedeed0159588ab4433a601f3 Author: David Liu Date: Thu Mar 24 16:08:19 2016 +0800 Handle tempurl Content-Disposition header missing from HEAD Content-Disposition headers should make no difference between GET and HEAD according to HTTP rfc. Closes-Bug: #1539805 Change-Id: Ifa41a7cda2f321eb8e36420ede7912ed0a549712 commit 2f24fb9683a57b67348d65864d5af8c3a03dee67 Author: Alistair Coles Date: Wed Mar 23 20:49:50 2016 +0000 Check marker params in SimpleClient full listing requests Follow up for change [1] to add some assertions to check that marker param is included in sequential GET requests sent during a full listing. Extract multiple FakeConn class definitions to single class at module level and share between all classes. Also, explicitly unpack the return values from base request calls made in the full listing section of base_request, and explicitly return a list to make more consistent with rest of the method. [1] Change-Id: I6892390d72f70f1bc519b482d4f72603e1570163 Change-Id: Iad038709f46364b8324d25ac79be4317add79df5 commit a696c1e89ef1a1c7630c14bbdda3f6c8039bb05e Author: Christian Schwede Date: Mon Dec 14 16:03:43 2015 +0100 Fix full_listing in internal_client The internal_client is used in swift-dispersion-report, and in case one has more than 10000 containers or objects these are not queried. This patch adds support to the internal_client to iterate over all containers/objects if the listing exceeds the default of 10000 entries and the argument full_listing=True is used. Closes-Bug: 1314817 Closes-Bug: 1525995 Change-Id: I6892390d72f70f1bc519b482d4f72603e1570163 commit 5d00ce9e3a1f5e32ae91c78b6bdfd953658ab984 Author: John Dickinson Date: Mon Mar 21 22:03:34 2016 -0700 2.7.0 authors and changelog updates Change-Id: I16ad0c61b048921ca01fa96862ae7eea0eec6017 commit 51bea3943f9eadb179dc7327d42dcb9062a3a8e5 Author: Christian Schwede Date: Mon Mar 21 09:52:23 2016 +0000 Ignore files in the devices directory when auditing objects The object auditor raises an exception if there are some files in /srv/node (or any other defined "devices" directory). This change simply skips any file in the devices directory when generating locations for the object auditor. Change-Id: I934594994adc577799723edb6c5648685682a9e7 commit 1d03803a85ca50272071725518c7110e1b2dacb1 Author: Clay Gerrard Date: Tue Mar 15 17:09:21 2016 -0700 Auditor will clean up stale rsync tempfiles DiskFile already fills in the _ondisk_info attribute when it tries to open a diskfile - even if the DiskFile's fileset is not valid or deleted. During this process the rsync tempfiles would be discovered and logged, but no-one would attempt to clean them up - even if they were really old. Instead of logging and ignoring unexpected files when validate a DiskFile fileset we'll add unexpected files to the unexpected key in the _ondisk_info attribute. With a little bit of re-organization in the auditor's object_audit method to get things into a single return path we can add an unconditional check for unexpected files and remove those that are "old enough". Since the replicator will kill any rsync processes that are running longer than the configured rsync_timeout we know that any rsync tempfiles older than this can be deleted. Split unlink_older_than in common.utils into two functions to allow an explicit list of previously discovered paths to be passed in to avoid an extra listdir. Since the getmtime handling already ignores OSError there's less concern of race condition where a previous discovered unexpected file is reaped by rsync while we're attempting to clean it up. Update some doc on the new config option. Closes-Bug: #1554005 Change-Id: Id67681cb77f605e3491b8afcb9c69d769e154283 commit 125d18e0ffac572749da58fad1591d5099b5cda7 Author: OSHRITF Date: Wed Jan 20 15:55:30 2016 +0200 Container-Sync to perform HEAD before PUT object on remote This change adds a remote HEAD object request before each call to sync_row. Currently, container-sync-row attempts to replicate the object (using PUT) regardless of the existance of the object on the remote side, thus causing each object to be transferred on the wire several times (depending on the replication factor) An alternative to HEAD is to do a conditional PUT (using, 100-continue). However, this change is more involved and requires upgrade of both the client and server side clusters to work. In the Tokyo design summit it was decided to start with the HEAD approach. Change-Id: I60d982dd2cc79a0f13b0924507cd03d7f9c9d70b Closes-Bug: #1277223 commit eecb1f2d54b8737c784952072b7374ceb7647245 Author: OpenStack Proposal Bot Date: Wed Mar 23 06:16:13 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: Ia2c2819db372da46538d71a80888a4e27538bdcd commit 2afa3681cb9c8a74cbe23b1de8d4f2e1a06e8f3a Author: Alistair Coles Date: Tue Mar 22 11:51:11 2016 +0000 Add .eggs/* to .gitignore After running: python setup.py build_sphinx there is a .eggs directory left in the repo root directory which is not currently ignored by git. Change-Id: Id15811f94046fd8bb22153425bf5cafe6c045453 commit 736de613f1e2a6a460f4499cbd1e9f7f1fc3da68 Author: Alistair Coles Date: Tue Mar 22 11:36:32 2016 +0000 Docs: Container sync does not require POST-as-COPY Updates docs to remove warnings that container sync only works with object_post_as_copy=True. Since commit e91de49 container sync will also sync POST updates when using object_post_as_copy=False. Change-Id: I5cc3cc6e8f9ba2fef6f896f2b11d2a4e06825f7f commit 88b575895750c2c00f75a6193cc31bc930e980fd Author: Alistair Coles Date: Tue Mar 22 11:10:41 2016 +0000 Remove unused code from container sync Change-Id: Ia44138aadcd30c474f744a9c552220e18302ecc6 commit 365276464ce41070f14fee4314e665dd2934bfa6 Author: OpenStack Proposal Bot Date: Tue Mar 22 06:16:34 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I70db7d29a9859cb47144ac49df8c289d1c2ec3e6 commit fd86d5a95d73714365c07cb36bfd1404306142a7 Author: Christian Schwede Date: Mon Feb 15 19:17:01 2016 +0000 Skip already checked partitions when auditing objects after a restart The object auditor will save a short status file on each device, containing a list of remaining partitions for auditing. If the auditor is restarted, it will only audit partitions not yet checked. If all partitions on the current device have been checked, it will simply skip this device. Once all partitions on all disks are successfully audited, all status files are removed. Closes-Bug: #1183656 Change-Id: Icf1d920d0942ce48f1d3d374ea4d63dbc29ea464 commit b685e8513525569ec1e0e4c34377866b6f5c1f7e Author: OpenStack Proposal Bot Date: Mon Mar 21 06:16:52 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I3cae432fdcf311bc3654956a1f61ce22ecae0933 commit 0ad1b0cca1f2538ac8fe3ef27d9e06c102157000 Author: oshritf Date: Thu Mar 17 16:53:04 2016 +0200 Container sync nodes shuffle cleanup Since commit "Update container sync to use internal client" get_object is done using internal_client and not directly on nodes which makes the block of code to shuffle the nodes redundant. Change-Id: I45a6dab05f6f87510cf73102b1ed191238209efe commit f99b785f47f458baa6a5286e6f6e942a6e2745f3 Author: Andreas Jaeger Date: Fri Mar 18 21:30:58 2016 +0100 Avoid translation problem Today's import failed again for zh-TW since the translator of a string changed the order of the entries and then msgfmt complained with "format specifications in 'msgid' and 'msgstr' for argument 1 are not the same" Use named arguments to allow translators to switch the order of the entries. Logfile: https://jenkins.openstack.org/job/swift-propose-translation-update/179/console Change-Id: Ide7480a8293a8439ce89d0ad22be74f1f0ad930c commit e1f12960884f9ea52416891a8246a53f9673348b Author: Bill Huber Date: Tue Jan 5 14:40:50 2016 -0600 Re-format the SLO manifest file on new multipart-manifest GET call Currently, the multipart-manifest=get call returns output in json format that is inconsistent with the format that is used for the multipart-manifest=put. This in turn introduces a new call: ?multipart-manifest=get&format=raw Change-Id: I2242943a738f667cbda6363bcb6a017f341e834f Closes-Bug: 1252482 commit a85149c40e34b3ccd2aa5f3865bb85d2378d6086 Author: Paul Dardeau Date: Thu Mar 17 22:12:42 2016 +0000 Update ring overview for reuse of device ids Change-Id: Ied15706d4cc931b5314001a02457f83e175c5d24 commit c70abba52958a78d21c00a08488d1c2a7c3df554 Author: Clay Gerrard Date: Thu Mar 17 11:55:14 2016 -0700 Adjust replica count before pulling parts from failed devices When your device count falls below your replica count you can either add devices or reduce the replica count. Trying to reduce your replica count fails about half the time because removing parts from from failed devices temporarily invalidates your _replica2part2dev table with NONE_DEV which can result in an IndexError in _adjust_replica2part2dev_size. If you adjust the replica count first you won't have to worry about tracking unassigned parts from failed devices. Closes-Bug: #1558751 Change-Id: I99dc776fd260a2ba68ca77d7b5ed5120d10b06de commit 9f7e947dca83a8b7c5e81493b88669472ef2875d Author: OpenStack Proposal Bot Date: Thu Mar 17 14:45:42 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I95466266e314cf4a564c7f93cfae7df9df21b60c commit 12dd408823df158359e99fb01716f2059140c5c9 Author: Alistair Coles Date: Wed Mar 16 17:41:30 2016 +0000 Put correct Etag and Accept-Ranges in EC 304 and 416 responses When using an EC policy, 304 responses to conditional GETs are missing the Accept-Ranges header and have the wrong ETag value. 412 responses also have the wrong etag. 416 responses to ranged GETs also have the wrong ETag. This patch ensures behaviour with EC policy is consistent with replication policy: - 304 and 416 responses have correct etag and Accept-Ranges - 412 responses have correct Etag but no Accept-Ranges Co-Authored-By: Mahati Chamarthy Co-Authored-By: Thiago da Silva Closes-Bug: #1496234 Closes-Bug: #1558197 Closes-Bug: #1558193 Change-Id: Ic21317b9e4f632f0751133a3383eb5487379e11f commit 6ea7635f87ce480e712439bf20e0e4d9ca81b352 Author: Charles Hsu Date: Thu Dec 31 12:17:21 2015 +0800 Add a note for functional tests with Keystone. Change-Id: I9da8f6a56ffb06e89b8da8c06b26ebc8452b652d Closes-Bug: #1530254 commit 6efee0ebb12a8d0c36aad83d4fd592c4e51c5722 Author: Alistair Coles Date: Wed Mar 16 11:38:33 2016 +0000 Make keystone middleware options consistent in docs Bring overview_auth.rst and proxy server man page up to date with changes made in [1] [1] Change-Id: I373734933189c87c4094203b0752dd3762689034 Change-Id: Ia16f0c391e7c357ccb9c13945839dc5647e49a13 commit ecbcc94989a385efd1a7c60e7c91e3e8898c5b31 Author: Kota Tsuyuzaki Date: Thu Mar 10 06:42:57 2016 -0800 Fix ssync related object-server docs Swift now uses SSYNC verb instead of old REPLICATION verb for ssync protocol. This patch replaces all docs written as REPLICATION into SSYNC and fix a few words for explanation. Change-Id: I1253210d4f49749e7d425d6252dd262b650d9548 commit f595a7e70492c2751d8c6f5ab60b5512e63281cf Author: Matthew Oliver Date: Fri Aug 29 17:14:46 2014 +1000 Add concurrent reads option to proxy This change adds 2 new parameters to enable and control concurrent GETs in swift, these are 'concurrent_gets' and 'concurrency_timeout'. 'concurrent_gets' allows you to turn on or off concurrent GETs, when on it will set the GET/HEAD concurrency to replica count. And in the case of EC HEADs it will set it to ndata. The proxy will then serve only the first valid source to respond. This applies to all account, container and object GETs except for EC. For EC only HEAD requests are effected. It achieves this by changing the request sending mechanism to using GreenAsyncPile and green threads with a time out between each request. 'concurrency_timeout' is related to concurrent_gets. And is the amount of time to wait before firing the next thread. A value of 0 will fire at the same time (fully concurrent), setting another value will stagger the firing allowing you the ability to give a node a shorter chance to respond before firing the next. This value is a float and should be somewhere between 0 and node_timeout. The default is conn_timeout. Meaning by default it will stagger the firing. DocImpact Implements: blueprint concurrent-reads Change-Id: I789d39472ec48b22415ff9d9821b1eefab7da867 commit f87a5487b5224f77261d82a1087d31820c29e8f8 Author: Clay Gerrard Date: Mon Mar 14 16:52:50 2016 -0700 Make rsync ignore it's own temporary files In situations where rsync may inadvertently be unable to cleanup it's temporary files we shouldn't spread them around the cluster. By asking our rsync subexec to --exclude patterns that match it's own convention for temporary naming we'll only ever transfer real replicated artifacts and never temporary artifacts which should always be ignored until they are fully transfered. Cleanup of stale rsync droppings should be performed by the auditor and will be addressed in a separate change related to lp bug #1554005. Closes-Bug: #1553995 Change-Id: Ibe598b339af024d05e4d89c34d696e972d8189ff commit 9e90d309621584be173b9be7ac91c469a013ab96 Author: OpenStack Proposal Bot Date: Mon Mar 14 06:13:37 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I30d06a1d96d4bd0e2e4972cb8ede7ebc08beae2d commit a537684c77a2e91c0cc6e4c4953a7e993f6a7005 Author: Brian Cline Date: Mon Mar 14 00:17:47 2016 -0500 Don't report recon mount/usage status on files Today recon will include normal files in the payload it returns for /recon/unmounted and /recon/diskusage. As a result it can trigger bogus alarms on any operations-side monitoring checking for unmounted disks or disks that show up in diskusage with weird looking stats. This change adds an isdir check for the entries it finds in /srv/node. Change-Id: Iad72e03fdda11ff600b81b4c5d58020cc4b9048e Closes-bug: #1556747 commit 1cef9f0e3375a9113985fe60b80650395ffc96ad Author: zhangguoqing Date: Sun Mar 13 10:51:18 2016 +0000 Fix py34 error of indexing 'dict_keys' object In python3, keys() of a dict isn't a list and cannot be indexed. Change-Id: I8a5b8e66a69f7096b2e5388bdeb12bb800879158 Closes-Bug: #1550184 commit 87fe86bcf8f061d91a8587e90613eee4e83c41d2 Author: OpenStack Proposal Bot Date: Sat Mar 12 06:12:49 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: Iee254bef3513111d339cafb4cb752eb4ad4c3b03 commit 3ac7cab32bcd08e032ad587a5982bdc997fc20c1 Author: Alistair Coles Date: Wed Mar 9 10:17:12 2016 +0000 Remove reference to py26 in tox testing instructions Also change some rst formatting and clarify that tox is used to run more than just *unit* tests. Change-Id: I263af7b4f7a554da3a391cf5104af573c89e3825 commit 8aca4e8eb36c67038335462abdb9a58ee5263b5c Author: OpenStack Proposal Bot Date: Fri Mar 11 06:22:54 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: Ic39cc9ffb582909da3a827243b628206de5301a0 commit 0bc5a69d416004924bd17c14d8455f65433751bd Author: Alistair Coles Date: Thu Mar 10 18:02:44 2016 +0000 Wait on greenthreads before unmocking http_connect The fake_spawn() context manager wraps spawn and waits for greenthreads to complete (such as the async_update threads). The wait needs to be done before http_connect is un-mocked, so the fake_spawn context manager should exit *before* any context manager that mocks the http_connect method. Also add fake_spawn to _test_PUT_then_POST_async_pendings() Related-Bug: #1536376 Related-Bug: #1514111 Closes-Bug: #1555739 Change-Id: I15f36e191cfe3ee6c82b4be56e8618ec0230e328 commit e38b53393fee311ca4e351b6d9ee3d2bf02441a4 Author: Donagh McCabe Date: Wed Mar 9 14:28:17 2016 +0000 Cleanup of Swift Ops Runbook This patch cleans up some rough edges that were left (due to time constraints) in the original commit. Change-Id: Id4480be8dc1b5c920c19988cb89ca8b60ace91b4 Co-Authored-By: Gerry Drudy commit 994fa7b115f31da37d064a60a823d1c87545d292 Author: Alistair Coles Date: Wed Mar 9 16:49:20 2016 +0000 Fix object server test not using correct policy Without the correct request header, the test that should be using an EC policy was in fact using a replication policy. Change-Id: Id44d0d615a4fd09aadfb286425939aea6abdf7b0 commit f581fccf71034818d19062593eeb52a4347bb174 Author: Tim Burke Date: Tue Feb 9 15:59:52 2016 -0800 By default, disallow inbound X-Timestamp headers With the X-Timestamp validation added in commit e619411, end users could upload objects with X-Timestamp: 9999999999.99999_ffffffffffffffff (the maximum value) and Swift would be unable to delete them. Now, inbound X-Timestamp headers will be moved to X-Backend-Inbound-X-Timestamp, effectively rendering them harmless. The primary reason to allow X-Timestamp before was to prevent Last-Modified changes for objects coming from either: * container_sync or * a migration from another storage system. To enable the former use-case, the container_sync middleware will now translate X-Backend-Inbound-X-Timestamp headers back to X-Timestamp after verifying the request. Additionally, a new option is added to the gatekeeper filter config: # shunt_inbound_x_timestamp = true To enable the latter use-case (or any other use-case not mentioned), set this to false. Upgrade Consideration ===================== If your cluster workload requires that clients be allowed to specify objects' X-Timestamp values, disable the shunt_inbound_x_timestamp option before upgrading. UpgradeImpact Change-Id: I8799d5eb2ae9d795ba358bb422f69c70ee8ebd2c commit dd2fbcd4c75974859820f76c3aa78be445170202 Author: OpenStack Proposal Bot Date: Wed Mar 9 06:35:57 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I9196deac606a3d073fb1c400cbee778cbbbd1c1e commit 3eeff7c0fc79fab594e3bbb04be74d397c920253 Author: Samuel Merritt Date: Tue Mar 8 15:23:40 2016 -0800 Fix misleading comment Change-Id: I2e0e671a08fb855bf53c57987c08a7eefca7078a commit dc6db66d5e4423ee39e8bf423916c035e55732d0 Author: Clay Gerrard Date: Tue Mar 8 12:32:39 2016 -0800 Spot check more hashes for increase_part_power Just a touch of cleanup to the unittest to demonstrate the stability of the guarantee that objects will hash to the same nodes but have different parts. Change-Id: I4a24187755455366a6435816a138f6175ae713a4 commit 9db7391e55e069d82f780c4372ffa32ef4e79c35 Author: Alistair Coles Date: Mon Mar 7 18:42:01 2016 +0000 Object POST update gets distinct async pending file Each object update to a container server is saved in a pending file if the initial update attempt fails. Pending file names were derived from the update request's x-timestamp, which is equal to the object's data file timestamp. This meant that updates due to an object POST used the same async pending file as updates due to the object's PUT. This is not so bad because the object POST update has a superset of the metadata included in the PUT update. But there is a risk of a race condition causing an update to be lost: the updater may open an update file due to a PUT whuile the object server is writing an update due to a POST to the same file name. The updater could then unlink the file before the more recent update for the POST is sent. This patch changes the POST update pending file name to be derived from the object's metadata timestamp, thus making it distinct from the PUT update pending file name. There is no upgrade impact since existing pending files will continue to be processed. Change-Id: I1b093c837efe8c2a64e92075ebd5e1b93e30efb9 commit 82c8023b894aa7a1fe12fbf1efbfad47afdf86b3 Author: Mahati Chamarthy Date: Tue Mar 8 19:01:53 2016 +0530 remove hash_cleanup_listdir legacy translation method Change-Id: I0b96dfde32b4c666eebda6e88228516dd693ef92 closes-bug:#1550569 commit 9430f4c9f5e79026f7275f8ec32ef4a180c5e8fc Author: Samuel Merritt Date: Wed Mar 2 10:28:51 2016 +0000 Move HeaderKeyDict to avoid an inline import There was a function in swift.common.utils that was importing swob.HeaderKeyDict at call time. It couldn't import it at compilation time since utils can't import from swob or else it blows up with a circular import error. This commit just moves HeaderKeyDict into swift.common.header_key_dict so that we can remove the inline import. Change-Id: I656fde8cc2e125327c26c589cf1045cb81ffc7e5 commit fa0c5f244f907ed2d5bd6d90f0be44869d7d1de6 Author: Clay Gerrard Date: Mon Feb 29 13:14:59 2016 +0000 Additionally break container servers during fast-post probes The existing probetests were already doing brainsplitting on the object servers in order to introduce composite metdata timestamp reconciliation. But they were not introducing failures at the container server level, and therefore not covering functional testing of those servers replication consistency repair. This change adds a brain splitter for the container servers to additionally exercise the consistency engine at that layer under the existing probetest so that you can observe their correct behavior while exercising the probetests [1]. 1. I used a script like this to watch the object servers and container db's repair themselves -> https://gist.github.com/clayg/a8077d9d29ff68bb0edd Change-Id: Ic5bb03dffef7e30c58338cef969a105577b84620 commit be54d0c928528cdc1b12e1bcb1614ea8859fae2e Author: janonymous Date: Mon Dec 7 21:45:43 2015 +0530 clear pycache and remove all pyc/pyo before starting unit test Delete python bytecode before every test run. Because python creates pyc files during tox runs, certain changes in the tree, like deletes of files, or switching branches, can create spurious errors. Closes-Bug: #1368661 Change-Id: Iedcb400fa3b0417f5bb8e943b17758fcfb4070c6 commit 936881543e60fa437cd0f91f029f9874fd7437d9 Author: Alistair Coles Date: Thu Jan 14 18:31:21 2016 +0000 Reclaim isolated .meta files It is possible for an object dir to contain only a .meta file*, and currently this file will never be deleted. This patch changes the diskfile on disk file processing to identify any isolated meta files and mark them as ready to be reclaimed, so that the cleanup will reclaim them if they are older than reclaim age. *An isolated .meta can occur as follows: Consider two replicas of the same object whose ondisk files have diverged due to failures: A has t2.ts B has t1.data, t4.meta (The DELETE at t2 did not make it to B. The POST at t4 was rejected by A.) After ssync replication the two ondisk file sets will be: A has t2.ts B has t2.ts, t4.meta Once t2 becomes older than (current time - reclaim age) we are left with: B has t4.meta Closes-Bug: #1534685 Change-Id: I3a175fd948a331d08df1b28f9831d7529216c6f1 commit d456d9e934d8eb845d83b1e0c29247aac87e2e92 Author: Alistair Coles Date: Wed Oct 7 10:17:39 2015 +0100 Don't ssync data when only a durable is missing If an EC diskfile is missing its .durable file (for example due to a partial PUT failure) then the ssync missing check will fail to open the file and will consider it missing. This can result in possible reconstruction of the fragment archive (for a sync job) and definite transmission of the fragment archive (for sync and revert jobs), which is wasteful. This patch makes the ssync receiver inspect the diskfile state after attempting to open it, and if fragments exist at the timestamp of the sender's diskfile, but a .durable file is missing, then the receiver will commit the diskfile at the sender's timestamp. As a result, there is no longer any need to send a fragment archive. Change-Id: I4766864fcc0a3553976e8fd85bbb2fc782f04abd commit 3ff94cb785867382ff6c37cb256d1b0f5381abaa Author: Christian Schwede Date: Mon Feb 15 19:16:08 2016 +0000 Add internal method to increase ring partition power This method increases the partition power of an existing ring by one. It does not move any data nor does it exposes a CLI command yet; it is only intended to be used in a future version to do the actual ring modification itself. An existing object that is currently located on partition X will be placed either on partition 2*X or 2*X+1 after the partition power got increased. The reason for this is the Ring.get_part() method, that does a bitwise shift to the right. To avoid actual data movement to different disks or even nodes, the allocation of partitions to nodes needs to be changed. The allocation is pairwise due to the above mentioned new partition scheme. Therefore devices are allocated like this, with the partition being the index and the value being the device id: OLD: 0, 3, 7, 5, 2, 1, ... NEW: 0, 0, 3, 3, 7, 7, 5, 5, 2, 2, 1, 1, ... If an operator stops the cluster, increases the partition power and renames & hardlinks the existing data it is possible to do a power shift without actually moving data. Please see the partition power spec for further details on this. Change-Id: I063fd8077497ee8c14d9065f07b4ec0fb5cbe180 Partially-Implements: spec increasing_partition_power commit 6b4e73bb15c680ada8b30279b090c6123ce5731f Author: Charles Hsu Date: Thu Feb 25 13:31:51 2016 +0800 Fix account-reaper unable to delete all containers. When reaper try to find the correct container_shard number, but it doesn't compare local device's name, so it always return first entry's index of local devices from the ring to container_shard. That causes some containers will be skip and objects under these containers won't be delete. Change-Id: I0e2bbdd99add86ee46e856920c0740c4aa13c77d Closes-Bug: #1549615 commit 22685d6231003bd236e7fc363b58e86f35ab80bb Author: Samuel Merritt Date: Tue Mar 1 13:54:09 2016 +0000 Faster suffix invalidations on object PUT/DELETE Before this commit, we performed a full read-modify-write cycle on a partition's hashes.pkl to invalidate the relevant suffix whenever an object PUT or DELETE completes. Now we append invalid suffixes to a new file, "hashes.invalid", alongside hashes.pkl. When we actually get a REPLICATE request and need to compute the hashes, *then* we perform a full read-modify-write on hashes.pkl and clear out hashes.invalid. Change-Id: Ia7add438e25688b4b286f0110a4e43490e11ad75 commit 6d8be59fce17008369656628615efcd2677dfe36 Author: Kota Tsuyuzaki Date: Mon Feb 22 22:41:51 2016 -0800 Fix ringbuilder tests Some of tests in test/unit/cli/test_ringbuilder doesn't assert the exit code and unfortunately some of these passed even if the statement fails for the assertion actually. This patch enables to assert the exit code from ringbuider and fixes some code/test bugs I noticed. Change-Id: I18fa675ba8a90678e2b5ccb5f90eafab01d22787 commit fe70898daced5f5fd698dd27098451439fdd8d08 Author: Tim Burke Date: Sun Feb 28 01:18:07 2016 +0000 Require account/container metadata be UTF-8 Otherwise, we get a UnicodeDecodeError when we call json.dumps() Change-Id: Ie200d029e1fd7f0ff0956c8ced98207e11ef9080 commit 0628f1268c804aa555ff3873d11831266c1b1cb1 Author: Bill Huber Date: Fri Oct 23 15:50:21 2015 -0500 Add unit tests for swift.account.reaper This patch adds more unit tests to diminish missing pieces of the coverage in the account_reaper unit test. Change-Id: Ib9e875ddd1334a4a67037dcfbd42d3b008ccd4e7 commit 2d73dd9e660b50ef04a20012963a15376983923f Author: Cheng Li Date: Sat Feb 27 23:26:47 2016 +0800 keep etag of target source instead of possible_source In proxy/controllers/base.py : ResumingGetter._get_source_and_node self.used_source_etag should be etag of target source instead of possible_source. Change-Id: Id6c33e79f65cb4ad9e3249957dd57dc2da8cad98 Closes-bug: #1550734 commit 75ab3cb78862647644af7eaf5e0fb590e73ae341 Author: Alistair Coles Date: Wed Oct 14 15:49:35 2015 +0100 Stop staticweb revealing container existence to unauth'd requests When a container has `X-Container-Meta-Web-Listings: false` then staticweb will return a 404 in response to a GET or HEAD on the container, regardless of whether the request is auth'd. That provides a way to probe for container existence. It should return a 401 if the request is not auth'd. This patch adds a call to swift.authorize before returning the 404. Closes-Bug: 1506116 Change-Id: I382323b49dc8f6d67bf4494db7084a860a10db59 commit b5a243e75a8033988063d2c1c90ac373bc0050d2 Author: Janie Richling Date: Fri Feb 19 18:07:18 2016 -0600 Set backend content length for fallocate - EC Policy Currently, the ECObjectController removes the 'content-length' header. This part is ok, except that value is being used to set 'X-Backend-Obj-Content-Length', so it is always 0. This leads to not calling fallocate (details on bug) on a PUT since the size is 0. This change makes use of some numbers returned from the EC Driver get_segment_info method in order to calculate the expected on-disk size that should be allocated. The EC controller will now set the 'X-Backend-Obj-Content-Length' value appropriately. Co-Authored-By: Kota Tsuyuzaki Co-Authored-By: John Dickinson Co-Authored-By: Tim Burke Change-Id: Ifd16c1438539e6fd9bb2dbcd053d11bea2e09fee Fixes: bug 1532008 commit 7aaca029175661d45d05d41cce367c0ae19cab41 Author: Christian Schwede Date: Thu Feb 11 13:18:07 2016 +0100 Add comments to role-based access control functional tests Change-Id: Ic18c8a2abc33e2f175f948991af0189935b8702f commit 335d58611d0daf87783e565b521974b456f5c636 Author: Ondřej Nový Date: Tue Dec 15 22:42:18 2015 +0100 Keystone middleware deprecated option is_admin removed It has been deprecated from Swift 1.8.0 (Grizzly) Change-Id: Id6bc10c3e84262c0a9e6160a76af03c0ad363e9c commit ceaaedbe4057542e374200673dd323ba64bb42eb Author: Alistair Coles Date: Wed Mar 11 11:22:53 2015 +0000 Make functional tests tolerate pre-existing versioned container test/functional/tests.py:TestAccount.testAccountHead relies on the account having a known number of containers. The test setup attempts to delete all existing containers but this fails a container has versions (and so is not emptied, and cannot be deleted). The tests then fails because the expected number of containers does not match the actual. 'bin/resetswift' before running tests will obviously clear all state but is not always convenient. This change removes any x-versions-location header before deleting containers during test setUp. Steps to recreate the pre-condition for failure on master: (based on original work by clayg) swift post target -r '.r:*, .rlistings' swift post source -H 'x-versions-location: target' for i in {1..4}; do echo "junk${i}" > junk swift upload source junk done Co-Authored-By: Clay Gerrard