Reviewed: https://review.openstack.org/221410 Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=eb8f1f83f1cfc63d8452bc30096fd1c145781527 Submitter: Jenkins Branch: feature/hummingbird commit cb683d391cb66d0f52830de16760c80fd2afedf9 Author: OpenStack Proposal Bot Date: Sat Sep 5 06:17:51 2015 +0000 Imported Translations from Transifex For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I2d92b8e34a665fb0bb4c048cfb0c59de295dfce6 commit e4542455c8a07b7981c247df8b737816062c1655 Author: Emett Speer Date: Wed Sep 2 17:18:03 2015 -0700 [Labs] Update links to Cloud Admin Guide Update links to the Cloud Admin Guide after the RST conversion of that book altered URLs. Change-Id: I899f8938498b744e62887968a65e58c00ef27f1b commit 58fcc07523978306cd3889ada73af5d9e664cf59 Author: Christian Schwede Date: Wed Sep 2 10:52:34 2015 +0000 Test if container_sweep is executed on unmounted devices This change ensures that container_sweep is not run if a device is not mounted and mount_check is set to True. Change-Id: I823083c8431d9e61fd426508033ec9188503957b commit e02609c66a804845672413b06830b87395afef31 Author: Samuel Merritt Date: Tue Sep 1 15:19:50 2015 -0700 Preserve traceback in swift-dispersion-report Commit c690bcb fixed a bug in the dispersion report, but changed this from a bare "raise" to "raise err", which loses the traceback. Not a big deal, but worth putting back IMO. Change-Id: Id5b72153a4b8df8e3faaf1fa3fb2040e28ba85cc commit d06d4ad0fd2dfe69da8008e729651264522c6c06 Author: Minwoo Bae Date: Tue Sep 1 15:08:44 2015 -0500 Included reference in swift.obj.diskfile to enumerate the string used for data file paths. Change-Id: Ie22caa678bc00dfc43fabec7efbbb9f34490f1b5 commit 615c7a204b9386e05c5bab658bfe96766ad1e680 Author: Brian Cline Date: Tue Sep 1 10:51:20 2015 -0500 Adds useful dispersion info from changelog Change-Id: I1a45088fc32620b02ff9a754b02ec1eb75a59d6e commit 3b8755098a1786c5447abf158bd686293a82977c Author: janonymous Date: Sun Aug 2 21:29:13 2015 +0530 Replace a / b with a // b to use integer division where needed Change-Id: I72c81faa62786e140b0de00e3a04934bf1b5adbd commit 524c89b7eeff037b8a6b421888771e15f98c2da2 Author: John Dickinson Date: Fri Aug 21 13:39:41 2015 -0700 Updated CHANGELOG, AUTHORS, and .mailmap for 2.4.0 release. Change-Id: Ic6301146b839c9921bb85c4f4c1e585c9ab66661 commit 05de1305a903ee4ce9c8c50fde53c552d5b90d51 Author: Clay Gerrard Date: Thu Aug 27 18:35:09 2015 -0700 Make ssync_sender send valid chunked requests The connect method of ssync_sender tells the remote connection that it's going to send a valid HTTP chunked request, but if the remote end needs to respond with an error of any kind sender throws HTTP right out the window, picks up his ball, and closes the socket down hard - much to the surprise of the eventlet.wsgi server who up to this point had been playing along quite nicely with this 'SSYNC' nonsense assuming that everyone here is consenting mature adults. If you're going to make a "Transfer-Encoding: chunked" request have the good decency to finish the job with a proper '0\r\n\r\n'. [1] N.B. It might be possible to handle an error status during the initialize_request phase with some sort of 100-continue support, but honestly it's not entirely clear to me when the server isn't going to close the connection if the client is still expected to send the body [2] - further if the error comes later during missing_check or updates we'll for sure want to send the chunk transfer termination line before we close down the socket and this way we cover both. 1. Really, eventlet.wsgi shouldn't be so blasted brittle about this [3] 2. https://lists.w3.org/Archives/Public/ietf-http-wg/2005AprJun/0007.html 3. https://github.com/eventlet/eventlet/commit/c3ce3eef0b4d0dfdbfb1ec0186d4bb204fb8ecd5 Closes-Bug #1489587 Change-Id: Ic17c6c3075553f8cf6ef6213e62a00282f0d01cf commit 993ee4e37af1961adba2047d5aa2eb210e423eb3 Author: nakagawamsa Date: Fri Aug 28 11:49:43 2015 +0900 Remove duplicate X-Backend-Storage-Policy-Index key There is duplicate 'X-Backend-Storage-Policy-Index' dictionary key in unit.obj.test_server.py. One key has fixed policy index value, and another has random value. Unittest should done with random policy index, so remove key which is set fixed value. Change-Id: Ic91fcf44d48297d0feee33c928ca682def9790a3 commit 893f30c61d280804e417790dd34ba7bc3fb4f6fc Author: paul luse Date: Wed Aug 12 13:32:50 2015 -0700 EC GET path: require fragments to be of same set And if they are not, exhaust the node iter to go get more. The problem without this implementation is a simple overwrite where a GET follows before the handoff has put the newer obj back on the 'alive again' node such that the proxy gets n-1 fragments of the newest set and 1 of the older. This patch bucketizes the fragments by etag and if it doesn't have enough continues to exhaust the node iterator until it has a large enough matching set. Change-Id: Ib710a133ce1be278365067fd0d6610d80f1f7372 Co-Authored-By: Clay Gerrard Co-Authored-By: Alistair Coles Closes-Bug: 1457691 commit 58a10a5fffec69304d7bcce0f1c43bd2a9f9ff52 Author: Alistair Coles Date: Wed Aug 26 16:30:23 2015 +0100 Add test that a tempurl POST cannot set a DLO manifest header Follow up to [1] to add tests for tempurl POSTs not being allowed to set a DLO manifest header. [1] I11e68830009d3f6bff44ae4011a41b67139146f6 Change-Id: I7c0ad5a936f71e56c599b8495a586913d3334422 Related-Bug: 1453948 commit d4409c0a046c6ce0e14e18c95efe2cd16cf120e8 Author: Samuel Merritt Date: Tue Aug 11 09:10:13 2015 -0500 Better scoping for tempurls, especially container tempurls It used to be that a GET of a tempurl referencing a large object would let you download that large object regardless of where its segments lived. However, this led to some violated user expectations around container tempurls. (Note on shorthand: all tempurls reference objects. However, "account tempurl" and "container tempurl" are shorthand meaning tempurls generated using a key on the account or container, respectively.) Let's say an application is given tempurl keys to a particular container, and it does all its work therein using those keys. The user expects that, if the application is compromised, then the attacker only gains access to the "compromised-container". However, with the old behavior, the attacker could read data from *any* container like so: 1) Choose a "victim-container" to download 2) Create PUT and GET tempurl for any object name within the "compromised-container". The object doesn't need to exist; we'll create it. 3) Using the PUT tempurl, upload a DLO manifest with "X-Object-Manifest: /victim-container/" 4) Using the GET tempurl, download the object created in step 3. The result will be the concatenation of all objects in the "victim-container". Step 3 need not be for all objects in the "victim-container"; for example, a value "X-Object-Manifest: /victim-container/abc" would only be the concatenation of all objects whose names begin with "abc". By probing for object names in this way, individual objects may be found and extracted. A similar bug would exist for manifests referencing other accounts except that neither the X-Object-Manifest (DLO) nor the JSON manifest document (SLO) have a way of specifying a different account. This change makes it so that a container tempurl only grants access to objects within its container, *including* large-object segments. This breaks backward compatibility for container tempurls that may have pointed to cross container *LO's, but (a) there are security implications, and (b) container tempurls are a relatively new feature. This works by having the tempurl middleware install an authorization callback ('swift.authorize' in the WSGI environment) that limits the scope of any requests to the account or container from which the key came. This requires swift.authorize to persist for both the manifest request and all segment requests; this is done by having the proxy server restore it to the WSGI environment prior to returning from __call__. [CVE-2015-5223] Co-Authored-By: Clay Gerrard Co-Authored-By: Alistair Coles Co-Authored-By: Christian Schwede Co-Authored-By: Matthew Oliver Change-Id: Ie6d52f7a07e87f6fec21ed8b0ec1d84be8b2b11c Closes-Bug: 1449212 commit 10b2939b433a4a79b4f7b97640b3d208cacfeffb Author: Clay Gerrard Date: Thu Jul 23 22:36:21 2015 -0700 Disallow unsafe tempurl operations to point to unauthorized data Do not allow PUT tempurls to create pointers to other data. Specifically disallow the creation of DLO object manifests by returning an error if a non-safe tempurl request includes an X-Object-Manifest header regardless of the value of the header. This prevents discoverability attacks which can use any PUT tempurl to probe for private data by creating a DLO object manifest and then using the PUT tempurl to head the object which would 404 if the prefix does not match any object data or form a valid DLO HEAD response if it does. This also prevents a tricky and potentially unexpected consequence of PUT tempurls which would make it unsafe to allow a user to download objects created by tempurl (even if they just created them) because the result of reading the object created via tempurl may not be the data which was uploaded. [CVE-2015-5223] Co-Authored-By: Kota Tsuyuzaki Change-Id: I11e68830009d3f6bff44ae4011a41b67139146f6 Closes-Bug: 1453948 commit c690bcb68331818a04c94741b25f40cc40f7b3c4 Author: Kazuhiro MIYAHARA Date: Mon Aug 17 16:50:56 2015 +0900 Fix dispersion-reports error message This patch fixes Swift to show message "No objects to query. Has swift-dispersion-populate been run?" for "swift-dispersion-report —object-only” with no container for object dispersion. Change-Id: I82da56709cfc296a27f5180681709bc56adbc13d Closes-Bug: #1468120 commit 8086a0e53406dc95856df2d3e015d3eaaf81380a Author: Tushar Gohad Date: Tue Aug 18 07:24:19 2015 +0000 Restrict PyECLib version to 1.0.7 v1.0.9 rev of PyECLib replaces Jerasure with a native EC implementation (liberasurecode_rs_vand) as the default EC scheme. Going forward, Jerasure will not be bundled with PyPI version of PyECLib as it used to be, until v1.0.7. This is an interim change to Swift requirements until we get v1.0.9 PyECLib included into global-requirements and ready patches that change Swift default ec_type (for doc, config samples and unit tests) from "jerasure_rs_vand" to "liberasurecode_rs_vand." Change-Id: Ica4fee2cdea2bc7f5edd0e51ad637a4457faf3b4 commit 17efa343c605d0361b3f423696babbab3f3d972d Author: Kota Tsuyuzaki Date: Mon Jul 6 13:21:40 2015 -0700 Fix EC GET backend stream iteration state In EC case, When GET object requested, proxy-server always makes a log line "Client disconnected on read" even though the request succeeded. That is because ECAppIter class doesn't maintain a bunch of backend stream when closing the app_iter. It will cause unfortunately GeneratorExit on backend stream ResumingGetter. This patch fixes to set non_client_disconnected to propagate the state to the backend streams when the range iteration stopped successful. Co-Authored-By: Clay Gerrard Change-Id: I77af9807816bea1444d66534a17e2a210bcf09f8 Closes-Bug: #1472201 commit 25dc7224b63b039cb3778cca7f9f81f954c6d94e Author: Alistair Coles Date: Tue Aug 25 11:05:41 2015 +0100 Fix swob.Range docstring Bad ranges cause a ValueError to be raised, not an empty ranges list. Change-Id: I118bd2f7dc08ff5198870f4093c6eb350506c8ed commit a7b84f4c51207608adc3a5de7f4f9a629053c0ab Author: Samuel Merritt Date: Mon Aug 24 23:11:34 2015 -0700 Allow pep8 of a single file Now you can run $ tox -e pep8 path/to/file.py [path/to/file2.py [...]] to run pep8 against just those files[1]. This is quite a bit faster than a full pep8 run, and the faster feedback is nice when you're fiddling with some formatting to placate pep8. Of course, you can still run "tox -e pep8" to check the whole source tree, just as before this commit. [1] It'll still run against bin/swift* as well, but that's still a lot faster than running against all our .py files. Change-Id: I81b4363fb95a34ff0f5c346b2b24f2047154f502 commit 6a35d479e8952c854fee6fbef9fe0397a289a6e5 Author: Matthew Oliver Date: Tue Aug 25 11:24:49 2015 +1000 Follow up patch to fix a multiline import NITPIC This change cleans up test/unit/obj/test_replicator.py's imports to use only 1 version of multiline import syntaxes (' \' vs '()'). I don't really mind which, but we should be consistant, at least in the same file. This is a follow up for patch 215857. Change-Id: Ie2d328c25865b19092c493981a803ee246a9d7a5 commit edde5584affaa983d8db1d294bf1e20a2d4bbb50 Author: Hisashi Osanai Date: Tue Aug 25 07:52:18 2015 +0900 Fix typo of a comment in replicator The typo was introduced by patch 138342 (sorry) so I fix it. Change-Id: Id5126802d281ef7ee9be128bd2152c0d2584160e commit 6151c6c45da481c0ea9a4fb876d14b39c1e347bf Author: Tim Burke Date: Mon Aug 24 20:36:22 2015 +0000 Fix use of delimiter in account listings Previously, account listings that used the delimiter query param could omit some containers if they ended with the character that follows the delimiter. See If196e3075612b121ef8da4a9128167d00a248c27 for the corresponding fix for container listings. Change-Id: I57fcb97e51f653f5f4e306a632fcb3a0fb148c4e commit a38f63e1c6b8b85b1675aa900e239a2e9906811e Author: Clay Gerrard Date: Fri Aug 21 18:14:55 2015 -0700 Use correct Storage-Policy header for REPLICATE requests Under some concurrency the object-replicator could potentially send the wrong X-Backed-Storage-Policy-Index header to it's partner nodes during replication if there were multiple storage policies on the same node because of a race where multiple jobs being processed concurrently would mutate some shared state on the ObjectReplicator instance. Instead of using shared stated on the ObjectReplicator instance when mutating the default headers send with REPLICATION requests each job will copy them into a local where they can safely be updated. Change-Id: I5522db57af7e308b1f9d4181f14ea14e386a71fd commit ae129bf5e8c918efd8ee66279f98ee85e7c97b24 Author: Kota Tsuyuzaki Date: Fri May 15 06:44:12 2015 -0700 eventlet.posthook related code needn't anymore In the past, older Swift seemed to use eventlet.posthooks scheme for the logging on proxy with posthooklogger method. However, for now, Swift attaches no method to eventlet.posthooks everywhere so we don't have to maintain the posthooks anymore. Change-Id: Ie63941a202d448532d980252a4d25575f8edab9c commit 4500ff340f021e4e5f6fd53960de03cb3c207925 Author: Kota Tsuyuzaki Date: Wed May 13 00:43:59 2015 -0700 Fix the missing SLO state on fast-post When using fast-post and POST (i.e. metadata update) is requested to a SLO manifest files, current Swift drops the 'X-Static-Large-Object' header from the existing metadata. It results in breaking the SLO state because the manifest missing the 'X-Static-Large-Object' metadata will be maintained as a normal files. This patch fixes object-server to keep the existing 'X-Static-Large-Object' flag and then keep the SLO state. Change-Id: Ib1eb569071372c322dd105c52baeeb094003291e Closes-bug: #1453807 commit 69e7424d3cb6c59341d739c3d41f42397031d96a Author: Eran Rom Date: Tue Jul 28 09:41:12 2015 +0300 Add container sync probe test to SAIO default set SAIO Configuration and documentation changes enabling to run the container sync probe test by default Change-Id: Iccf59533d0d4fe72549d318339ab125d04dde006 Related-Bug: #1476623 commit 8b1df9918bf6c22b49b161a402199b532bfd8266 Author: Clay Gerrard Date: Fri Aug 21 14:04:46 2015 -0700 Minor cleanup handoff mode warnings * message is a little clearer * test is a little stronger Change-Id: I745cde7f4a46dafc80ab42d39e6ccc92aa3b746e commit a1ceab5a92dfdafd5abffed272ac2cece5302f32 Author: Carlos Cavanna Date: Fri Aug 21 14:14:31 2015 -0400 New troubleshooting case in documentation. Added a new troubleshooting case for the "First Contribution to Swift" documentation page. Change-Id: I182ba702b49b28409fe56becae93326e5f63dcd0 commit ab163702de733be39ba5e7024c7a8dd4c86bc29b Author: Pradeep Kumar Singh Date: Tue Jul 14 10:03:18 2015 +0530 Emit warning log in object replicator When the object-replicator encounters handoffs_first and handoff_delete options as enabled it should emit a log warning indicating that it should be changed back to the default before the next "normal" rebalance. Closes-Bug: #1457262 Change-Id: If9dc2796c18ed3cf13da920831e2d5c2ae9f12a0 commit be66aa8e76d2994eb8a0a944e0dea26ffbb3c159 Author: Samuel Merritt Date: Thu Aug 20 13:24:38 2015 -0700 Fix 500 for bogus Range request to 0-byte object. The proxy was trying to pop a byterange off a Range header that didn't contain syntactically-valid byteranges. This worked about as well as you'd expect. Now we detect the bogus value and remove the header entirely. Change-Id: I24b92f900d33ec79880c7db2870378489d5a6810 commit 4b6836b3bc465e9c2c6d2974c193a49b973c4e4e Author: Akihito Takai Date: Thu Aug 20 18:40:52 2015 +0900 Fix the comment in [1]. (line 259) parameter(op) of object_update method is 'PUT' or 'DELETE' not 'POST' or 'DELETE'. [1]: swift/obj/updater.py Change-Id: I876a620ba8e09e69fba7156b12e69445c229e160 commit 923238aa1ba0963c414ba5321cd3910b2910f4ed Author: janonymous Date: Tue Jul 28 20:35:25 2015 +0530 test/(functional/probe):Replace python print operator with print function (pep H233, py33) 'print' function is compatible with 2.x and 3.x python versions Link : https://www.python.org/dev/peps/pep-3105/ Python 2.6 has a __future__ import that removes print as language syntax, letting you use the functional form instead Change-Id: I416c6ac21ccbfb91ec328ffb1ed21e492ef52d58 commit 47dc31940d64b0c87e99cc73eddcdd01dd8b3ad4 Author: Samuel Merritt Date: Wed Aug 19 10:42:28 2015 -0700 Add OpenStack release names to changelog I can never remember the mapping of real version numbers to the OpenStack names. Change-Id: Ib7c5ae4ff2a33018364698eb34f8df1622719fab commit 183508710d011ec30ebba2186c58fd28d449e4bb Author: Victor Stinner Date: Sun Aug 16 11:18:49 2015 +0200 pep8: Don't override '_' symbol Fix pep8 warning F402 "import '_' from line 51 shadowed by loop variable". Change-Id: I139060ff5d298a8b0f0f8e529a5737478fb5daf5 commit c7eb589c6ca2fe4f2950cae22a18513e4dec6c46 Author: Victor Stinner Date: Sun Aug 16 11:06:52 2015 +0200 pep8: Fix usage of the l10n _('...') function Fix the pep8 warning H702 "Formatting operation should be outside of localization method call". For the logger, pass parameters as indexed parameters instead of using the string str%args operator, the logger is more reliable in case of formatting error. Change-Id: If418bc155f6a6c0a00f63e3d87ebe4addf4aae55 commit 7bea148d2fe22daa9d1fbbc5cd16ebd390b64b1b Author: Victor Stinner Date: Sun Aug 16 10:34:26 2015 +0200 pep8: replace deprecated calls to assert_() The TestCase.assert_() has been deprecated in Python 2.7. Replace it with assertTrue() or even better methods (assertIn, assertNotIn, assertIsInstance) which provide better error messages. Change-Id: I21c730351470031a2dabe5238693095eabdb8964 commit 8aaacbf88d9ea4432e58dd261b98ff95d7c69be6 Author: Victor Stinner Date: Mon Jul 27 18:55:01 2015 +0200 pep8: Fix hacking H232 warnings (octal) Fix warnings "H232: Python 3.x incompatible octal 000001234 should be written as 0o1234". Change-Id: I9a7bbb034357783885ac3e18fe1e9e32a5951616 commit 5b01c5eadc3a13607d2581a55007700fca0b00af Author: Bill Huber Date: Fri Aug 7 16:24:28 2015 -0500 Add unit test for container_update for unmounted device This unit test case gains coverage for container/update.py where a device (i.e. sda1) is not mounted and the code responds with a warning logger that the device is not mounted and continues. Coverage increases 2% as a result. Change-Id: I33d247a930b28604093df4ade1ce7dbbd24aac54 commit d6267c1f95a7fcd2b0e4f15d5efc6bcfe2114de0 Author: Clément Contini Date: Wed Aug 12 15:00:45 2015 -0400 Keep user id and project id in subrequests env Keep HTTP_X_USER_ID and HTTP_X_PROJECT_ID to be available as user_id and project_id in storage.objects.outgoing.bytes in ceilometer when downloading a multipart object. Change-Id: I0f4734f021e5d6e84d48ed9bebeb321d7a9590ad Closes-Bug: #1477283 commit eaa006464cb98a2f36809edfa13bd3dcaebc9952 Author: John Dickinson Date: Mon Aug 17 22:13:42 2015 -0700 move global statement up a few lines Change-Id: I190d2d530c6c0525d988cc88b0965b48a291fffb commit 1a81cda8b7eba0d25ea7341da756bb588cae8d73 Author: Hamdi Roumani Date: Mon Aug 3 17:45:56 2015 -0400 Doc instructions for post rebase steps Improve the 'first_contribution_swift' by adding instructions for how to rebuild swift following a rebase. Change-Id: If5c91dc4e1e8d1712bbd8b326c675967fb4b8c15 commit 79ba4a85983641e539b620bd143e62673c98416e Author: Hisashi Osanai Date: Wed Dec 3 06:15:16 2014 +0900 Enable Object Replicator's failure count in recon This patch makes the count of object replication failure in recon. And "failure_nodes" is added to Account Replicator and Container Replicator. Recon shows the count of object repliction failure as follows: $ curl http://:/recon/replication/object { "replication_last": 1416334368.60865, "replication_stats": { "attempted": 13346, "failure": 870, "failure_nodes": { "192.168.0.1": {"sdb1": 3}, "192.168.0.2": {"sdb1": 851, "sdc1": 1, "sdd1": 8}, "192.168.0.3": {"sdb1": 3, "sdc1": 4} }, "hashmatch": 0, "remove": 0, "rsync": 0, "start": 1416354240.9761429, "success": 1908 }, "replication_time": 2316.5563162644703, "object_replication_last": 1416334368.60865, "object_replication_time": 2316.5563162644703 } Note that 'object_replication_last' and 'object_replication_time' are considered to be transitional and will be removed in the subsequent releases. Use 'replication_last' and 'replication_time' instead. Additionaly this patch adds the count in swift-recon and it will be showed as follows: $ swift-recon object -r ======================================================================== ======= --> Starting reconnaissance on 4 hosts ======================================================================== ======= [2014-11-27 16:14:09] Checking on replication [replication_failure] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 4 [replication_success] low: 3, high: 3, avg: 3.0, total: 12, Failed: 0.0%, no_result: 0, reported: 4 [replication_time] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 4 [replication_attempted] low: 1, high: 1, avg: 1.0, total: 4, Failed: 0.0%, no_result: 0, reported: 4 Oldest completion was 2014-11-27 16:09:45 (4 minutes ago) by 192.168.0.4:6002. Most recent completion was 2014-11-27 16:14:19 (-10 seconds ago) by 192.168.0.1:6002. ======================================================================== ======= In case there is a cluster which has servers, a server runs with this patch and the other servers run without this patch. If swift-recon executes on the server which runs with this patch, there are unnecessary information on the output such as [failure], [success] and [attempted]. Because other servers which run without this patch are not able to send a response with information that this patch needs. Therefore once you apply this patch, you also apply this patch to other servers before you execute swift-recon. DocImpact Change-Id: Iecd33655ae2568482833131f422679996c374d78 Co-Authored-By: Kenichiro Matsuda Co-Authored-By: Brian Cline Implements: blueprint enable-object-replication-failure-in-recon commit eeb0fa40a19917e6548f95f0bd3c08736928449b Author: Christian Schwede Date: Thu Aug 6 07:21:15 2015 +0000 Make swift-ring-builder filename usage more consistent Sometimes the given argument is internally altered and another filename is used without a note to the operator. Even worse, a given .ring.gz filename is sometimes written out as builder file, without updating the corresponding .builder file. There is already a method to parse the given argv and return the name of the builder and ring file. However, it's rarely used and no warning is given to the user if it is altered. This patch uses the already parsed builder and ring file name instead of argv[1], and also adds a note to the user if the used filename is differently to the one given as argument. Closes-Bug: 1482096 Change-Id: I2f8ef23aeab8b07caaa799f7dcd57e684b4b2ad2 commit b75d2a4e37d4c86763a2cc56c6dd53ebe2e0de19 Author: Bill Huber Date: Mon Aug 17 13:54:44 2015 -0500 Quorum on durable response is too low Increase the .durable quorum from 2 to "parity + 1" to guarantee that we will never fail to rebuild an object. Otherwise, with low durable responses back (< parity + 1), the putter objects return with failed attribute set to true, thereby failing the rebuild of fragments for an object. Change-Id: I80d666f61273e589d0990baa78fd657b3470785d Closes-Bug: 1484565 commit fa35e38c9f83f704eb087f80157cbe33b23b9db2 Author: Kai Zhang Date: Fri Aug 14 16:46:35 2015 -0700 Fix some minor typos Fixed some typos in function name and comments. Change-Id: Ida76ab4b331a51b71e57650702acc136e66ba4b2 commit 5b246e875fe9ac3d764ea581ad52b04238f5bcc8 Author: Kota Tsuyuzaki Date: Wed Jul 22 10:39:22 2015 -0700 Fix EC range GET/COPY handling When range GET (or COPY) for an EC object requested, if the requested range starts from more than last segments alignment (i.e. ceil(object_size/segment_size) * segment_size), proxy server will return the original content length w/o body, though Swift should return an error massage as a body and the length of message as the content length. The current behavior will cause stuck on some client. (e.g. curl) This patch fixes that proxy enables to return correct response, even if such an over range requested. Co-Authored-By: Clay Gerrard Change-Id: I21f81c842f563ac4dddc69011ed759b744bb20bd Closes-Bug: #1475499 commit 7064706b2721c36c5181267f58704b9344ad4ac0 Author: Samuel Merritt Date: Tue Aug 11 10:17:20 2015 -0500 Fix 500 in versioned writes with bad Destination When this code lived in the proxy, it was protected by an "except HTTPException" clause in proxy.Application.handle_request(). When it moved to middleware, it lost that, and then things like constraints.check_name_format that raised HTTPException would cause 500s. The HTTPException would make it all the way out to catch_errors and get translated to a 500. This commit just wraps a couple try/excepts around the bits in versioned writes that can raise HTTPException. I tried to make it use wsgify so I could get that for free, but that wound up being a real pain because env/start_response are plumbed through pretty much the whole versioned-writes middleware. Closes-Bug: 1483705 Change-Id: Ife165bf709e64f313ed07c779b21914045e51f25 commit 035a411660ca02983cd486312266c67d78a2359c Author: Thiago da Silva Date: Sun Nov 9 13:13:27 2014 -0500 versioned writes middleware Rewrite object versioning as middleware to simplify the PUT method in the object controller. The functionality remains basically the same with the only major difference being the ability to now version slo manifest files. dlo manifests are still not supported as part of this patch. Co-Authored-By: Clay Gerrard DocImpact Change-Id: Ie899290b3312e201979eafefb253d1a60b65b837 Signed-off-by: Thiago da Silva Signed-off-by: Prashanth Pai commit 968c91a465627bb9317ed6e2ae9ccd818c83c0c4 Author: Alistair Coles Date: Tue Jul 28 10:31:54 2015 +0100 Replace assertTrue(not ) with assertFalse() The replacement of assert_ with assertTrue [1] resulted in a number of tests using calls of the form assertTrue(not ). This patch replaces those with assertFalse(). [1] change I74705c6498249337bfdf955d62e0ad972035bc1f Change-Id: I78b49558f4425c2335df187b1793d1e4b3c514b1 commit 4ac1fea5d111c669ff827f4eb29c0735cbad6ba5 Author: Zhao Lei Date: Fri Aug 7 22:07:01 2015 +0800 Fix some spelling typo in comments s/overide/override for object-expirer.conf and sample. s/automaticaly/automatically for swift/proxy/controllers/obj.py Change-Id: Ife107c7a1005a5d4959288db50a7f8f33c522dd4 Signed-off-by: Zhao Lei commit 9456af35a2832aeafc4a4e78ebd0d4142ead71cd Author: janonymous Date: Thu Aug 6 00:55:36 2015 +0530 pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3,changes in dir: *test/unit/obj/* *test/unit/test_locale/* Change-Id: I3dd0c1107165ac529f1cd967363e5cf408a1d02b commit 1952451ed7e97e2bfa3b11dfa5000cc39f712084 Author: Zhao Lei Date: Fri Aug 7 21:57:08 2015 +0800 Fix a spelling typo in comment s/automaticaly/automatically/ for swift/proxy/controllers/obj.py Change-Id: I405441383739637e0b13746d6f4e2bad1c874b4d Signed-off-by: Zhao Lei commit c35cc13b8abeb97400632cbe8ec56fc1040d8210 Author: Bill Huber Date: Thu Aug 6 10:01:17 2015 -0500 pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3 in the following dir: test/unit/proxy/* Change-Id: Ie2c7e73e1096233a10ee7fbf6f88386fa4d469d6 commit 239e94e62518f36601fb18119700269753d02a01 Author: Bill Huber Date: Thu Aug 6 09:28:51 2015 -0500 pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3 in the following dir: test/probe/* Change-Id: Ie08dd7a8a6c48e3452dfe4f2b41676330ce455d5 commit 49b9ba37ac522502ccad66fba7bfef375eab145d Author: Bill Huber Date: Wed Aug 5 14:51:32 2015 -0500 pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3 in the following dir: test/functional/* Change-Id: Iee7f8ffca9838ccc521107180697d91ac9559405 commit 81816bebe6b3506af8cabaa7ebe265c70f3499cd Author: kenichiro matsuda Date: Thu Aug 6 10:13:02 2015 +0900 Fix shebang of commands Fix shebang of following commands. $ grep '#!/usr/bin/python' swift/bin/* swift/bin/swift-account-info:#!/usr/bin/python swift/bin/swift-container-info:#!/usr/bin/python swift/bin/swift-container-sync:#!/usr/bin/python swift/bin/swift-recon:#!/usr/bin/python swift/bin/swift-ring-builder:#!/usr/bin/python swift/bin/swift-ring-builder-analyzer:#!/usr/bin/python Change-Id: I564d1d8abd76eba57730fc2f30263b0a0f809867 Closes-Bug: #1481623 commit 652f0f9da408e955843f87a504b3dab5be3cff80 Author: Pete Zaitcev Date: Wed Aug 5 15:33:29 2015 -0600 Having said H, I, J, we ought to say K In the long run, we might want to stop these updates. But this decision is above my pay grade. Change-Id: I335558e1da8052be1b215fbad51244b47af3d81b commit ed3aec2146b114c85bb3b196c999bf3f44fe159e Author: janonymous Date: Thu Aug 6 00:18:52 2015 +0530 pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3 in dir: test/unit/container/* Change-Id: I3333022ed63ce03198bc73147246d91d2442a440 commit f449e914728fdcaf0870c15b9fc5ebd16a9df916 Author: janonymous Date: Wed Aug 5 22:32:02 2015 +0530 pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, fixing in: dir: test/unit/cli/* Change-Id: I9a2fc1f717beafd5fa8408942046e310e8de0318 commit 6594bbebb582801c069c4966352fee19eadef2f8 Author: janonymous Date: Wed Aug 5 22:25:46 2015 +0530 pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3 in dir : test/unit/account/* Change-Id: I70415197df1b0d58d4d6e219733c13a9c9cdfff7 commit 89f59062864e5cbfc839a6084c323ce35438aa57 Author: Ben Martin Date: Mon Jul 27 14:19:09 2015 -0500 +Document method to avoid rsync filling root drive When rsync pushes to a remote node with an unmounted drive and if certain steps are not taken, rsync may attempt to write files to the local drive at the location where the drive was mounted. There are two suggested solutions for this issue: 1) Set the permissions for all mount points in /srv/node/ to root:root 755 2) Mount the drives elsewhere and symlink the drives to /srv/.../ The first method ensures that only root and not the swift user can write in the /srv/.../ directories. The second method will prompt a broken link issue if rsync attempts to write to an unmounted drive. Change-Id: I60ce4ed9ef8401768d5f78b6806cbb2e2a65303e Closes-Bug: #1470576 commit f0d51882b921075f5309ed6ff07385dc6103d4dc Author: Charles Hsu Date: Wed Aug 5 22:09:40 2015 +0800 Add extra_header_count to document and config. Change-Id: Iec86b488d71553c295afe7098822ce2046df9546 commit 7071762d3698c59fabdee76890e300f81417be74 Author: Clay Gerrard Date: Tue Aug 4 23:15:37 2015 -0700 Fix TypeError if backend response doesn't have expected headers There was some debug logging mixed in with some error handling on PUTs that was relying on a very specific edge would only encounter a set of backend responses that included the expected set of headers to diagnoise the failure. But the backend responses may not always have the expected headers. The proxy debug logging should be more robust to missing headers. It's a little hard to follow, but if you look `_connect_put_node` in swift.proxy.controller.obj - you'll see that only a few connections can make their way out of the initial put connection handling with a "resp" attribute that is not None. In the happy path (e.g. 100-Continue) it's explictly set to None, and in most errors (Timeout, 503, 413, etc) a new connection will be established to the next node in the node iter. Some status code will however allow a conn to be returned for validation in `_check_failure_put_connections`, i.e. * 2XX (e.g. 0-byte PUT would not send Expect 100-Continue) * 409 - Conflict with another timestamp * 412 - If-None-Match that encounters another object ... so I added tests for those - fixing a TypeError along the way. Change-Id: Ibdad5a90fa14ce62d081e6aaf40aacfca31b94d2 commit 01848c565e5821d2500cb234261c70871e562ebe Author: David Goetz Date: Fri Jun 26 17:27:31 2015 -0700 Speed up reaper for a big account delete and some better error handling In run_forever mode, run the reaper on all three copies of the account DB instead of just the first one and shard the containers between them. Change-Id: Id182e89dad70df9d6d9988523861b8085d74c765 commit 5b24b2249878e25ee702ee4237227c984036c6ec Author: OpenStack Proposal Bot Date: Tue Aug 4 06:29:11 2015 +0000 Imported Translations from Transifex For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I9f0e5fd25ec143fb87de1895794764dea4587f93 commit 8d06f56c1fac23bca463d3a93353da6b50ad5bed Author: Takashi NATSUME Date: Tue Aug 4 10:14:25 2015 +0900 Fix typo of 'receive' Change-Id: Ic01defa8d48ba4b7f4bd6cbe4d59b9bab468a36c commit 310ac18d61b001bdf297505243d8aa62e8fde8fe Author: Brian Reitz Date: Mon Aug 3 16:28:31 2015 -0500 Convert docstring to a comment in test_denied_DELETE_of_versioned_object unit test. Docstrings cause issues for nose and created non standard test output for this test. Change-Id: I91e6450480567de0ee0b18d3efae01a4b94f0a86 commit 21a7b4aaa6f991c0eeb8f74876b14fe22713586b Author: Alistair Coles Date: Mon Aug 3 14:58:24 2015 +0100 Test that get_hashes ignores only removed hash dir Add test for case not yet covered by unit tests: suffix dir has two hash dirs, one with expired tombstone. That hash dir gets removed and its hash is not included in the suffix hash, but the remaining hash dir's hash is reported. Change-Id: I031a022daed6b8a66dfd04bea1b4d5eebcb882b3 commit 44917c8a90e79a85cd6c6821d8b6d043f83b322d Author: Clay Gerrard Date: Mon Jul 20 13:06:48 2015 -0700 Handle removed suffix dirs the same as empty suffix dirs When hashes suffix dirs, the directory might have gotten cleaned up while still appearing in hashes.pkl. It would even get cleaned up the next time. For example, given this really old tombstone: objects/846/3d0/d3a20154d0a828a032aba6860397c3d0/1432267961.41808.ts Prior to this commit, a call to get_hashes() would reap the old tombstone and any empty containing dirs, but the resulting hashes.pkl would still contain {'3d0': 'd41d8cd98f00b204e9800998ecf8427e'} even though there's no such suffix dir any more. ("d41d8cd98f00b204e9800998ecf8427e" is the MD5 of the empty string.) Then, a *subsequent* get_hashes() call would omit 3d0 from the resulting hash, so then hashes.pkl would no longer contain 3d0. This difference would result in a little useless replication traffic while nodes without a particular part/suffix pair, but who disagreed on how that showed up in hashes.pkl, tried to push their version of nothing to one another. Now, an empty suffix dir doesn't appear in hashes.pkl at all, whether it's for replication or EC, or whether it's for the get_hashes() call that reaped the suffix dirs or not. Co-Author: Samuel Merritt Change-Id: Ie1bfb1cc56d0fc030c6db42f97b55d140695cf1f commit cf4d50bd68b658d35759fb56446557858dcd9572 Author: Alistair Coles Date: Fri Jun 5 13:06:29 2015 +0100 Make test_proxy work independent of env vars test.unit.common.test_internal_client.TestSimpleClient.test_proxy will fail if the environment has http_proxy set and does not have no_proxy=127.0.0.1 set. This seems to be because urllib overrides any proxy arg passed to the Request with the env http_proxy var, unless the Request host is in no_proxy. We don't need to test urllib, so this patch changes the test to simply ensure that swift code does pass the correct proxy arg to urllib2.urlopen. That avoids testers having to make sure their env vars are compliant to successfully run unit tests. While it is reasonable to think that 127.0.0.1 is in the no_proxy list, it shouldn't be a requirement pass swift tests. Change-Id: Iff91acdb76fabe7056b3e90e2756e27fe5105817 commit 67513fc17c23066f6547b5e3c4ab2dc852e4f7f0 Author: janonymous Date: Sat Jun 27 23:26:47 2015 +0530 Adding bandit for security static analysis testing in swift This change adds a basic bandit config for Swift. It can be invoked by running the tox environment for bandit; tox -e bandit This is an initial step for using bandit with Swift and it should be revisited to improve the testing as more is learned about the specific needs of the Swift code base.As per now some tests are excluded as they were used on purpose. https://wiki.openstack.org/wiki/Security/Projects/Bandit Implements: blueprint swift-bandit Change-Id: I621be9a68ae9311f3a6eadd1636b05e646260cf2 commit 768d7ab074807175471d589de96d091e3239052e Author: Clay Gerrard Date: Thu Jul 30 15:30:35 2015 -0700 Add a probetest for HUP/reload This would have been enough to catch the regression, and we can extend them as we work on any future ehancements to our process management. Change-Id: I9a1b57aa15663380c45cf783afc8212ab4ffbace commit 207dd9b49d7d53a9faa4849af2c40bb875416fce Author: Darrell Bishop Date: Thu Jul 30 14:32:08 2015 -0700 Fix regression in WSGI server SIGHUP behavior The SIGHUP receipt used to pop us out of an os.wait() where now, it's in a "green" wait() and Timeout() combo, some part of which eats the signal receipt. This causes the while loop condition to never get checked and SIGHUP no longer works as a server reload command. The fix is to loop at least every 0.5 seconds, as a trade-off between not busy-waiting and checking the "keep running" condition often enough to feel responsive. Change-Id: I95283b8b7cfc2998ab5813e0ad3ca1fa231696c8 Closes-Bug: #1479972 commit bcd00d9461603db1477c5f1e9f8dd6405a319eb9 Author: Alistair Coles Date: Mon Jun 8 19:40:56 2015 +0100 Refactor diskfile This patch mostly eliminates the duplicate code that was deliberately left in place during EC review to avoid major churn of the diskfile module prior to the kilo release. This focuses on obvious de-duplication and shuffling code between classes. It deliberately does not attempt to hammer out every last piece of de-duplication where that would introduce more complex changes - that can come later. Code is moved from the module level and from ECDiskFile* classes into new BaseDiskFile* classes. Concrete classes for replication and EC policy retain their existing names i.e. DiskFile[Manager|Writer|Reader|] and ECDiskFile[Manager|Writer|Reader|] respectively. Knock-on changes: - fix bug whereby get_hashes was ignoring self.reclaim_age and always using the default arg value. - replication diskfile manager now deletes a tombstone that is older than reclaim_age even when there is a newer .meta file. - replication diskfile manager will no longer raise an AssertionError if only a .meta file is found during hash_cleanup_listdir. - fix stale test in test_auditor.py: test_with_tombstone test setup was convoluted (probably dates back to when object puts did not clean up the object dir). Now that they do you have to try harder to create a dir with a tombstone and a data file. Change-Id: I963e0d0ae0d6569ad1de605034c529529cbb4f9a commit 9cb7eb4a4b6cdab8a5f16b3dc800b39ab4068522 Author: Victor Stinner Date: Mon Jul 27 11:34:07 2015 +0200 Update hacking to 0.10.0 Replace the whitelist of flake8 checks (select) with a blacklist (ignore). It makes possible to disable a single check, which was not possible before. This new approach permits to enable new tests more easily and see which checks are currently disabled. Only new checks are disabled, this change doesn't run less checks than before. Currently, many checks are disabled, but following changes will fix these new warnings. When the "select" option of flake8 is used in tox.ini, the "ignore" option is ignored. It is not possible to use "select" and "ignore" options at the same time. See the code in pep8 (flake8 is based on pep8): https://github.com/PyCQA/pep8/blob/d48eef071003952ed0f98445dc3dc6bc29618f9c/pep8.py#L1808 The purpose of flake8 is to enhance the coding style, but also to help to detect common bugs. Blacklisting checks permits to detect new bugs when upgrading the tools. For example, new checks were recently added to help porting code to Python 3. This change permits to use pbr 1.0 and newer which is required to support Python 3. Previously, hacking 0.8.1 was used, but hacking 0.8 is incompatible with pbr 1.0 and newer (hacking 0.8 requires pbr<1.0). pbr 1.0 is required to use environment markers on requirements to install dnspython on Python 2 and dnspython3 on Python 3. Change-Id: I41d63b9467083d7606ad18aaa398ca7738b27fae commit 389a97e3526df68121cecfaf1909343b5d381cbc Author: Victor Stinner Date: Wed Jul 29 23:46:58 2015 +0200 Fix pep8 E warning for hacking 0.10 Fix pep8 warnings of the E category of hacking 0.10: * E113: unexpected indentation * E121: continuation line under-indented for hanging indent * E122: continuation line missing indentation or outdented * E123: closing bracket does not match indentation of opening bracket's line * E126: continuation line over-indented for hanging indent * E251: unexpected spaces around keyword / parameter equals Change-Id: I0b24eebdf1a37dc1b572b6c9a3d3d4832d050237 commit d719064e788ec5fa70305fadaa46affcb603a9bb Author: Victor Stinner Date: Thu Jul 30 00:28:44 2015 +0200 Fix warning pep8 E128 warning of hacking 0.10 Fix the warning E128: "continuation line under-indented for visual indent" of pep8. Change-Id: Ie6c6ae341fe3d6281f2095c1d756d552fa5937f9 commit a0db56dcde83096708e6fe64ae9338098e0ba98a Author: Victor Stinner Date: Thu Jul 30 00:16:06 2015 +0200 Fix pep8 E265 warning of hacking 0.10 Fix the warning E265 "block comment should start with '# '" added in pep 1.5. Change-Id: Ib57282e958be9c7cddffc7bca34fbbf1d4c460fd commit 345785837f1caa01fec5bca091c70481d7069bea Author: Charles Hsu Date: Thu Jul 30 13:11:00 2015 +0800 Remove error_suppression_interval, error_suppression_limit options. These two options is belong to proxy-server, not account-replicator. Change-Id: Ie030ecffd213e56db32df77c69b847479d96308f commit 5a7383313b1c555df5fd47050f7ee3810b5e31fc Author: Carlos Cavanna Date: Tue Jul 28 19:15:25 2015 -0400 Swift documentation for first-time contributors. In this change, there are proposed improvement to Swift documentation aimed at first-time contributors. They include a simplification of the Getting Started page and a new page with some basic instructions/commands that the first-time contributor should know. In addition, it shows some common errors that the first-time contributor may find when executing git rebase and information on how to track your changes. Change-Id: I704202955093736b2f3b4102a649690a0392c6b0 commit 5b5c8820024141bc6ae49d7646a92fb53667ea9b Author: Alistair Coles Date: Wed Jul 29 12:36:35 2015 +0100 Remove unused code in swift/obj/server.py keep_cache was defined twice Change-Id: I5f775f618669e3dabfc3bc54dd7bd2cd7f00b2f0 commit 0c27cb07caf1027cd9734e8bd34d9759da33e667 Author: OpenStack Proposal Bot Date: Wed Jul 29 06:36:01 2015 +0000 Imported Translations from Transifex For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: Ic30cb1dc8f12280a52cc1f07a1cf516397ed680e commit c5b5cf91a984f80cc6cbe42735b242083d700542 Author: janonymous Date: Tue Jul 28 21:03:05 2015 +0530 test/unit: Replace python print operator with print function (pep H233, py33) 'print' function is compatible with 2.x and 3.x python versions Link : https://www.python.org/dev/peps/pep-3105/ Python 2.6 has a __future__ import that removes print as language syntax, letting you use the functional form instead Change-Id: I94e1bc6bd83ad6b05695c7ebdf7cbfd8f6d9f9af commit 117b61976fbf14d8a0d2157256f9a3230421ee72 Author: Samuel Merritt Date: Wed Jul 22 15:40:55 2015 -0700 functests: use assertIn and assertNotIn We have a bunch of assertions like self.assertTrue(resp.status in (200, 204)) Sometimes we get smart about failure messages and have something like self.assertTrue(resp.status in (200, 204), resp.status) so we can see what the status was when it failed. Since we don't have to support Python 2.6 any more, we can use assertIn/assertNotIn and get nice failure messages for free. Change-Id: I2d46c9969d41207a89e01017b4c2bc533c3d744f commit 8fe8bee7e0c5c9d34d7a8e6ab090c8d65c5ca78e Author: janonymous Date: Tue Jul 21 18:06:32 2015 +0530 test/(functional, probe): Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not generate warnings all over the screen. Change-Id: I74705c6498249337bfdf955d62e0ad972035bc1f commit 4a6e0ccc69302a5384b988149e294656323d7f20 Author: Samuel Merritt Date: Mon Jul 20 11:58:35 2015 -0700 Rename WsgiStringIO -> WsgiBytesIO. If we're going to have a subclass of BytesIO, having "StringIO" in its name is just asking for confusion. Change-Id: I695ab3105b1a02eb158dcf0399ae91888bc1c0ac commit 5ee742c17d9ea92ca7332cb67db0cce64b7a0d8f Author: janonymous Date: Fri Jul 24 21:19:47 2015 +0530 Modifying Lambda function tuple usage for python3 Current tuple unpacking is not supported in python3 Change-Id: I15a8e977684d19154cf2233caf1174f6983ac9e2 commit 6f89f71f9ba11f7b4126b80076a0f13792332e1e Author: Alistair Coles Date: Wed Apr 15 16:17:23 2015 +0100 Filter Etag key from ssync replication-headers ssync rx sends a header X-Backend-Replication-Headers whose value is a list of headers that the source object has. This list extends the list of allowed headers on the target object server, so that the target object metadata is faithfully reconstructed to match the source. Unfortunately the combination of lower() and title() operations on header keys results in the source 'ETag' value being added to the target metadata under the key 'Etag' in addition to the 'ETag' key that the receiving server adds (note different capitilization), both having the same value. The spurious 'Etag' metadata is potentially confusing for humans inspecting the object metadata and complicates tests that wish to assert the equality of two object metadata dicts. See for example the test in test_ssync_sender.py that this patch cleans up. Furthermore, the possibility of having both Etag and ETag keys has required a workaround in the EC reconstructor [1]. [1] reconstructor fix change id: Ie59ad93a67a7f439c9a84cd9cff31540f97f334a Change-Id: I0c89cf7924a4471bb6d268b5ef3884e2d2cb4286 commit e2f69138bf5fcdd84ac23be6b0bec1bb006b41f5 Author: Clay Gerrard Date: Thu Jul 23 16:46:48 2015 -0700 Fix string formatting in dispersion cli command ... and add a basic test that would have prevented the regression Change-Id: I4c5f643ee291dcc1397ca951450459d8b8ad0bbd commit 83025ff38dfea3d4e3f27dc045128a7aa4ba11e4 Author: Clay Gerrard Date: Fri Jul 24 11:09:10 2015 -0700 Use right most occurrence object dir in diskfile.extract_policy I like using the rightmost one more; it's basically /operator-defined/mountpoint/objects/part/suffix/hash/ts.data, so I don't see any opportunity for other things named "objects" to creep in on the right of the real objects-N dir; but I could see some admin using /srv/object-storage/ or something -- Torgomatic The Wise Change-Id: I0a63a3e02df091a5ee2e110a345183012e357a2f commit 363a256e58ee0f3e9f059352c2de968e3dc398e3 Author: Falk Reimann Date: Wed Jun 24 16:54:02 2015 +0200 Support keystone v3 domains in swift-dispersion This provides the capability to specify a project_name, project_domain_name and user_domain_name in /etc/swift/dispersion.conf. If this values are set in dispersion.conf they get populated to the swift-client. With this it is possible to have a specific dispersion project specified, which is not the keystone default domain. Changes were applied to swift-dispersion-populate and swift-dispersion-report. Relevant man pages, the example dispersion.conf and the admin guide were updated accordingly. DocImpact Closes-Bug: #1468374 Change-Id: I0e716f8d281b4d0f510bc568bcee4a13fc480ff7 commit 3d9b2d077a157279fb6cd107a0c83727106af3ef Author: Daisuke Morita Date: Fri Jul 24 10:42:10 2015 +0900 Fix swift-object-info to show policy index at deep working directory If swift-object-info command is executed at deeper working directory than 'objects-*' directory, it cannot parse policy index from file path so it does not show appropriate policy index. This patch fixes this problem by simply extracting the full path of a target object file. Change-Id: Idb734106a44b6121119c9b1dc8cdaaf4c6c28c31 Closes-Bug: 1469951 commit e24d7c36fa9758618ef88ed1a8e19f4d7bb0404f Author: Victor Stinner Date: Mon May 25 18:26:38 2015 +0200 Use six to fix imports on Python 3 Get configparser, queue, http_client modules from six.moves. Patch generated by the six_moves operation of the sixer tool: https://pypi.python.org/pypi/sixer Change-Id: I666241ab50101b8cc6f992dd80134ce27327bd7d commit 964869accce8836bd2587c4b07559fb05fe580b2 Author: Atsushi SAKAI Date: Fri Jul 24 17:09:48 2015 +0900 Fix six typos on swift documentation mechanisim => mechanism http://docs.openstack.org/developer/swift/cors.html overridde => override http://docs.openstack.org/developer/swift/deployment_guide.html extentsions => extensions http://docs.openstack.org/developer/swift/development_ondisk_backends.html reuqest => request http://docs.openstack.org/developer/swift/logs.html suport => support http://docs.openstack.org/developer/swift/overview_architecture.html mininum => minimum http://docs.openstack.org/developer/swift/overview_erasure_code.html $ git diff | diffstat cors.rst | 2 +- deployment_guide.rst | 2 +- development_ondisk_backends.rst | 2 +- logs.rst | 2 +- overview_architecture.rst | 2 +- overview_erasure_code.rst | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) Change-Id: I8e095f4c216b2cfae48dff1e17d387048349f73c Closes-Bug: #1477877 commit dd2f1be3b124d3901ebbc176a7adc462b6449667 Author: Ondrej Novy Date: Tue Jun 16 17:42:58 2015 +0200 Time synchronization check in recon. This change add call time to recon middleware and param --time to recon CLI. This is usefull for checking if time in cluster is synchronized. Change-Id: I62373e681f64d0bd71f4aeb287953dd3b2ea5662 commit 89f705e8aab144092d40a13fc4ef19ffef5f3eba Author: OpenStack Proposal Bot Date: Thu Jul 23 06:11:27 2015 +0000 Imported Translations from Transifex For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I94cf347564cb33977f33b1e64259bcb39a8cf809 commit cbcfb74132aba4d5834dc9cf33dc4f664b82d39a Author: Clay Gerrard Date: Wed Jul 22 19:32:46 2015 -0700 Cleanup nits from container_update_timeout Change-Id: Ib17fc9a027db267350abc5a7db4e3172dfff7913 commit f563a423bf2534268c2e785094d58dbbe0ebd390 Author: Hamdi Roumani Date: Wed Jul 22 15:27:26 2015 -0400 Updated tox setup instructions on Development Guidelines The instructions suggest updating the ownership of the swift.egg-info directory may be required and provide a cmd to do so. Said cmd assumes user 'swift' exists, but tox will require the current user to have access to the directory to perform updates, moreover user swift may not exist at all. Change-Id: I63951970553c086ff4f1ac65a81559ba69166494 commit a65e9db8752793ec37b594dc9eca5066171825db Author: Christian Schwede Date: Wed Jul 22 10:43:17 2015 +0000 Removing commented out code in test/unit/account/test_backend.py Noticed this while reviewing another change. Looks like the test itself already ensures correct functionality of the reclaim() method in AccountBroker without the commented code, thus removing this stale code. Change-Id: I6a26a7591adef9fd794ca68a4e9c493d1127f93c commit 99d052772a9585e0befdfd292fd03aefde77180a Author: Kota Tsuyuzaki Date: Mon Jul 13 01:12:43 2015 -0700 Fix 499 client disconnected on COPY EC object Currently, a COPY request for an EC object might go to fail as 499 Client disconnected because of the difference between destination request content length and actual transferred bytes. That is because the conditional response status and content length for an EC object range GET is handled at calling the response instance on proxy server. Therefore the calling response instance (resp()) will change the conditional status from 200 (HTTP_OK) to 206 (PartialContent) and will change the content length for the range GET. In EC case, sometimes Swift needs whole stored contents to decode a segment. It will make 200 HTTP OK response from object-server and proxy-server will unfortunately set whole content length to the destination content length and it makes the bug 1467677. This patch introduces a new method "fix_conditional_response" for swift.common.swob.Response that calling _response_iter() and cached the iter in the Response instance. By calling it, Swift can set correct condtional response any time after setting whole content_length to the response instance like EC case. Change-Id: If85826243f955d2f03c6ad395215c73daab509b1 Closes-Bug: #1467677 commit 2289137164231d7872731c2cf3d81b86f34f01a4 Author: John Dickinson Date: Sat May 23 15:40:03 2015 -0700 do container listing updates in another (green)thread The actual server-side changes are simple. The tests are a different matter. Many changes were needed to the object server tests to handle the now-async calls to the container server. In an effort to test this properly, some drive-by changes were made to improve tests. I tested this patch by doing zero-byte object writes to one container as fast as possible. Then I did it again while also saturating 2 of the container replica's disks. The results are linked below. https://gist.github.com/notmyname/2bb85acfd8fbc7fc312a DocImpact Change-Id: I737bd0af3f124a4ce3e0862a155e97c1f0ac3e52 commit 62ed4f81ef80440550633eaaaa962a4f9383c2d3 Author: Timur Alperovich Date: Tue Jul 14 16:56:44 2015 -0700 Add two functional tests for delimiter. The first test verifies that a delimiter will trim entries beyond the first matching instance of delimiter (after the given matching prefix, if any) and squash duplicates. So, when setting the delimiter to "-", given blobs "test", "test-foo" and "test-bar-baz", we expect only "test" (no matching delim) and "test-" (trim all characters after the first "-", and squash duplicates). The second test verifies that when a prefix is provided, the delimiter will trim entries beyond the first matching instance of the delimiter *after the given prefix*. So "bar, "bazar" which both match the prefix "ba" will be returned as "bar" (no delimiter after the matching prefix) and "baza" (after matching the prefix the remainder after the the *next* matching delimiter "a" is trimmed). Change-Id: I49a2aa8722f83e87b7d211e5c26827e93963d92a commit 2d6c5fe2afde382edb8f04c91fca1911601b61d9 Author: Clay Gerrard Date: Tue Jul 21 09:56:13 2015 -0700 Fix Associated Projects link on Getting Started Change-Id: I50be656bc5868242489f97c96a1feaef171933e0 commit cd7b2db550e41412281da0256b468c90ebcf3e8b Author: janonymous Date: Tue Jul 21 19:23:00 2015 +0530 unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71 commit c3cc98b2c918476f1b54924e0c54e4c746eba4ac Author: Samuel Merritt Date: Mon Jul 20 17:03:20 2015 -0700 Add comment about ResumingGetter.used_source_etag This confused a couple developers and took about ten minutes to unravel in IRC; let's leave a clue for the next person. Change-Id: I356c8c7a44de23f02eaf68d23a39c9eb4c203ff1 commit 06326d74657a8796439988115002298e4a5f4630 Author: janonymous Date: Mon Jul 20 21:41:58 2015 +0530 Metaclass Python 3.x Compatibility use "six.add_metaclass" instead of "__metaclass__" Change-Id: I57390fec40afe3a965be3215db8f79b9cf862a70 commit 317fa1af49d72ecaf7a9a499d8cdf00a086b4c6f Author: Azhagu Selvan SP Date: Sat Jul 18 14:37:16 2015 +0530 Minor change in development saio setup doc Make the test-requirements install command uniform with the rest of the setup commands in the page by using the $HOME, instead of hardcoding the folder path. Change-Id: I8da7a191a301601e27f894f8268238d77b150ede commit 39621b8629c7f378fc197f6ab1ddc8e2e797d1fa Author: janonymous Date: Sat Jul 18 14:12:58 2015 +0530 Deprecated xreadlines() usage fixed Change-Id: Iadf42c2f86f78c11259e21c88b4aef51db3441ad commit dde34c4b442a27761f2e6eca46ed9ec47a110392 Author: janonymous Date: Sat Jul 11 22:46:41 2015 +0530 Modify zip usage for python3 where necessary. py2 zip() is eager but py3 zip() and six.moves.zip() are lazy, changing ones that require eager evaluation. Change-Id: Ic9f6bccd7f57772158581905794f8d23b05f4223 commit faff98efd47fd59e28b1adb3e7ad4bc3dc6dd781 Author: janonymous Date: Thu Jul 9 07:35:47 2015 +0530 Replace python print operator with print function (pep H233, py33) 'print' function is compatible with 2.x and 3.x python versions Link : https://www.python.org/dev/peps/pep-3105/ Python 2.6 has a __future__ import that removes print as language syntax, letting you use the functional form instead Change-Id: I9d9ca5766a2773a7b4071e280f00361d16b85383 commit 8753e452b04b0af6d6b444184b628724e14fbbb1 Author: Victor Stinner Date: Wed May 27 18:01:37 2015 +0200 Replace StringIO with BytesIO for WSGI input wsgi.input is a binary stream (bytes), not a text stream (unicode). * Replace StringIO with BytesIO for WSGI input * Replace StringIO('') with StringIO() and replace WsgiStringIO('') with WsgiStringIO(): an empty string is already the default value Change-Id: I09c9527be2265a6847189aeeb74a17261ddc781a commit 6e70f3fa322da70c2a65fb919d200e52fc5add8a Author: Victor Stinner Date: Wed May 27 17:27:47 2015 +0200 Get StringIO and cStringIO from six.moves * replace "from cStringIO import StringIO" with "from six.moves import cStringIO as StringIO" * replace "from StringIO import StringIO" with "from six import StringIO" * replace "import cStringIO" and "cStringIO.StringIO()" with "from six import moves" and "moves.cStringIO()" * replace "import StringIO" and "StringIO.StringIO()" with "import six" and "six.StringIO()" This patch was generated by the stringio operation of the sixer tool: https://pypi.python.org/pypi/sixer Change-Id: Iacba77fec3045f96773d1090c0bd48613729a561 commit c366ecf5c85c2797582bf79f7faab1d75b99a70e Author: janonymous Date: Mon Jul 13 00:07:13 2015 +0530 Fix a minor python issue related with string.split Replacing string.split(value, ',') with value.split(','): the string.split() function was removed in Python 3 Change-Id: I9ed241a894ac698da33df75a774a56ae2824fadf commit c8131c6abb17b00b043a9b95e65be819e1fd3c9d Author: janonymous Date: Sun Jul 12 14:48:35 2015 +0530 sys.exc_type/exc_value/exc_traceback are Deprecated sys.exc_info() contains a tuple of these three. Change-Id: I530cbeb37c43da98b4924db41f6604871077bd47 commit 39b6ef6e4fd515d81e688ef365d26d5e0499be7c Author: Charles Hsu Date: Thu Jun 25 02:06:54 2015 +0800 Fix reconstructor stats mssage. Calculate reconstruction job count and remaining time that would be inappropriate for user. Use real partition count would be suitable for user. Change-Id: I6b025854baf4757dddf9d7fe7bc2cece58a49157 Closes-Bug: #1468298 commit bfbda38db999ccd881509c68f0319c3d3dae4274 Author: Clay Gerrard