Reviewed: https://review.openstack.org/264517 Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=93ddaffaebb620bc712d52d0267194562dce02be Submitter: Jenkins Branch: feature/hummingbird commit f53cf1043d078451c4b9957027bf3af378aa0166 Author: Ondřej Nový Date: Tue Jan 5 20:20:15 2016 +0100 Fixed few misspellings in comments Change-Id: I8479c85cb8821c48b5da197cac37c80e5c1c7f05 commit 79222e327f9df6335b58e17a6c8dd0dc44b86c17 Author: ChangBo Guo(gcb) Date: Sat Dec 26 13:13:37 2015 +0800 Fix AttributeError for LogAdapter LogAdapter object has no attribute 'warn' but has attribute 'warning'. Closes-Bug: #1529321 Change-Id: I0e0bd0a3dbc4bb5c1f0b343a8809e53491a1da5f commit 684c4c04592278a280032002b5313b171ee7a4c0 Author: janonymous Date: Sun Aug 2 22:47:42 2015 +0530 Python 3 deprecated the logger.warn method in favor of warning DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead Change-Id: I35df44374c4521b1f06be7a96c0b873e8c3674d8 commit d0a026fcb8e8a9f5475699cc56e1998bdc4cd5ca Author: Hisashi Osanai Date: Wed Dec 16 18:50:37 2015 +0900 Fix duplication for headers in Access-Control-Expose-Headers There are following problems with Access-Control-Expose-Headers. * If headers in X-Container-Meta-Access-Control-Expose-Headers are configured, the headers are kept with case-sensitive string. Then a CORS request comes, the headers are merged into Access-Control-Expose-Headers as case-sensitive string even if there is a same header which is not case-sensitive string. * Access-Control-Expose-Headers is handled by a list. If X-Container/Object-Meta-XXX is configured in container/object and X-Container-Meta-Access-Control-Expose-Headers, same header is listed in Access-Control-Expose-Headers. This patch provides a fix for the problems. Change-Id: Ifc1c14eb3833ec6a851631cfc23008648463bd81 commit 0bcd7fd50ec0763dcb366dbf43a9696ca3806f15 Author: Bill Huber Date: Fri Nov 20 12:09:26 2015 -0600 Update Erasure Coding Overview doc to remove Beta version The major functionality of EC has been released for Liberty and the beta version of the code has been removed since it is now in production. Change-Id: If60712045fb1af803093d6753fcd60434e637772 commit 84ba24a75640be4212e0f984c284faf4c894e7c6 Author: Alistair Coles Date: Fri Dec 18 11:24:34 2015 +0000 Fix rst errors so that html docs are complete rst table format errors don't break the gate job but do cause sections of the documents to go missing from the html output. Change-Id: Ic8c9953c93d03dcdafd8f47b271d276c7b356dc3 commit 87f7e907ee412f5847f1f9ffca7a566fb148c6b1 Author: Matthew Oliver Date: Wed Dec 16 17:19:24 2015 +1100 Pass HTTP_REFERER down to subrequests Currently a HTTP_REFERER (Referer) header isn't passed down to subrequests. This means *LO subrequests to segment containers return a 403 on a *LO GET when accessed by requests using referer ACLs. Currently the only way around referer access to *LO's is to make the segments container world readable. This change makes sure the referer header is passed into subrequests allowing a segments container to only need to be locked down with the same referer as the *LO container. This is a 1 line change to code, but also adds a unit and 2 functional functional tests (one for DLO and one for SLO). Change-Id: I1fa5328979302d9c8133aa739787c8dae6084f54 Closes-Bug: #1526575 commit e15960a5d86e00a7d420edc4af034b27da0af8fd Author: Alistair Coles Date: Thu Dec 17 12:08:45 2015 +0000 Fix incorrect kwarg in auth middleware example When calling memcache_client.set(), timeout was deprecated and is now removed as a keyword arg, use time instead. Change-Id: Iedbd5b064853ef2b386963246f639fbcd3931cd3 commit 169a7c7f9e12ebc9933bd9ca4592e13b0de8b47b Author: Alistair Coles Date: Wed Dec 16 15:28:25 2015 +0000 Fix func test --until-failure and --no-discover options This patch changes functional test classes to subclass unittest2.TestCase rather than unittest.TestCase. This fixes errors when attempting to use tox -e func -- -n and tox -e func -- --until-failure Also migrate from using nose.SkipTest to unittest2.SkipTest Change-Id: I903033f5e01833550b2f2b945894edca4233c4a2 Closes-Bug: 1526725 Co-Authored-By: Ganesh Maharaj Mahalingam commit b68311db95860ac1cab585a5ab66bd3b3abb765e Author: Kota Tsuyuzaki Date: Tue Dec 15 18:55:41 2015 -0800 Fix reconciler test to calc lastmodified as UTC Swift reconciler calculates the last-modified date as UTC but current test calculates it as local time zone. It triggers unit test failure in non-UTC environment. This patch fixes the test to calculate the last-modified as UTC as well. Change-Id: Ia0053f350daf2cb8c61ac01a933924b6e4b0cb37 Closes-Bug: #1526588 commit 1bb665331af92422290fb585de7cb6a2497236e6 Author: Venkateswarlu Pallamala Date: Mon Nov 9 19:22:38 2015 -0800 remove unused parameters in the method make the helper methods as private by using convention Change-Id: I73b9604f8d5a0e85d012aac42b7963b618f5ad97 commit 9d7f71d5754c8b45f8e7c6ab80202de09933afb8 Author: Richard Hawkins Date: Fri Aug 7 18:14:13 2015 -0500 Modify functional tests to use ostestr/testr Defcore uses Tempest, which uses Test Repository. This change makes it easier for Defcore to pull functional tests from Swift and run them. Additionally, using testr allows tests to be run in parallel. Concurrency set to 1 for now, >1 causes failures for reasons that are still TBD. With switch to ostestr all the server logs are being sent to stdout which makes it completely unreadable. Suppressing the logs by default now with a flag to enable it if desired. Co-Authored-By: John Dickinson Co-Authored-By: Robert Collins Co-Authored-By: Matthew Oliver Co-Authored-By: Ganesh Maharaj Mahalingam Change-Id: I53ef4a116996a772cf1f3abc2eb0ad60047322d5 Related-Bug: 1177924 commit 2f4b79233e30d42140bbc07059417443bf7a0757 Author: Alistair Coles Date: Tue Dec 15 15:49:42 2015 +0000 Minor cleanup of repeated identical test assertions assertDictContainsSubset is being called multiple times with same arguments in a loop. Since assertDictContainsSubset is deprecated form python 3.2, replace it with checks on individual key, value pairs. Change-Id: I7089487710147021f26bd77c36accf5751855d68 commit 60b2e02905d57f55169e506f4874b2334a1a68a5 Author: Alistair Coles Date: Mon Oct 5 16:15:29 2015 +0100 Make ECDiskFile report all fragments found on disk Refactor the disk file get_ondisk_files logic to enable ECDiskfile to gather *all* fragments found on disk (not just those with a matching .durable file) and make the fragments available via the DiskFile interface as a dict mapping: Timestamp --> list of fragment indexes Also, if a durable fragment has been found then the timestamp of the durable file is exposed via the diskfile interface. Co-Authored-By: Clay Gerrard Change-Id: I55e20a999685b94023d47b231d51007045ac920e commit 450737f886050e486f518cdce0c97596ccad848d Author: Hisashi Osanai Date: Tue Dec 15 11:33:56 2015 +0900 Fix a typo in development_auth.rst This patch uses correct name for "CORS". Change-Id: I5fee5c581a2b3adb7596a273baf05708bfa97f79 commit 40476ea0797690d3a90a9ed91906d26103dfa058 Author: John Dickinson Date: Mon Dec 14 10:52:22 2015 -0800 Document pretend_min_part_hours_passed Added a docstring for the swift-ring-builder CLI command "pretend_min_part_hours_passed". This is a dangerous operation, and that's why it hasn't been documented, but it can be useful at times. It should be made known to those who need it. Change-Id: I45bdbaacbbdda64c7510453e6d93e6d8563e3ecd commit 6ade2908cca696ce1b48a7a19f4d460081fa5b0a Author: Ondřej Nový Date: Sun Dec 13 21:13:42 2015 +0100 Deprecated param timeout removed from memcached Change-Id: Idf042a79f0db148bf9f28a9e360cb2a3c18d385a commit 52b534c9d13efde15841402dc86c99685334250f Author: hgangwx Date: Fri Dec 11 17:31:36 2015 +0800 Fix some inconsistency in docstrings Added colon after ":returns" according to: http://docs.openstack.org/developer/hacking blank lines are added before the param lines Change-Id: I008bea270c6a4b7c80d1a84a2cb9fcc9b5a7264a commit 88c9aed7c846402355a3c7831f34f3e833bbdf11 Author: Victor Stinner Date: Mon Oct 19 16:19:28 2015 +0200 Port swift.common.utils.StatsdClient to Python 3 * StatsdClient._send(): on Python 3, encode parts to UTF-8 and replace '|' with b'|' to join parts. * timing_stats(): replace func.func_name with func.__name__. The func_name attribute of functions was removed on Python 3, whereas the __name__ attribute is available on Python 2 and Python 3. * Fix unit tests to use bytes Change-Id: Ic279c9b54e91aabcc52587eed7758e268ffb155e commit ca2dcc371921aa1aded6161287cc03c0940bf198 Author: Ondřej Nový Date: Fri Dec 11 18:21:28 2015 +0100 Deprecated tox -downloadcache option removed Caching is enabled by default from pip version 6.0 More info: https://testrun.org/tox/latest/config.html#confval-downloadcache=path https://pip.pypa.io/en/stable/reference/pip_install/#caching Change-Id: I9451a0f0dee5c5a3c0ca0a52f58bd353602661a2 commit 19c7dbc0ba49ff7e1f59190a7ead9ae9d3d80238 Author: Mahati Chamarthy Date: Fri Dec 11 12:27:18 2015 +0530 Update versioned_writes doc Change-Id: Ibe53c79cf49330332112001c02a2a6b078764130 commit 211758f8cb02298fe16e59bf2954a146c6b24b83 Author: Catherine Northcott Date: Thu Nov 5 23:04:14 2015 +1300 Add support for storage policies to have more than one name This patch alters storage_policy.py to allow storage policies to have multiple names. Now users are able to add a number of human-readable aliases for storage policies. Policies now have a .name (the default name), .aliases (a string of comma seperated aliases), and .aliases_list (a list of all human readable names). Policies will always have an .aliases value if no aliases are set it will contain the default name. The policy docs and tests have been updated to reflect changes and policy.get_policy_info has been altered to display the name and aliases Change-Id: I02967ca8d7c790595e5ee551581196aa64552eea commit 70bc3d1a3a240c35eba934010fdef99a775b8715 Author: John Dickinson Date: Tue Dec 8 16:08:04 2015 -0800 added a few ruby clients to associated projects Change-Id: I4764ba505646949ff694f8939947f83c6940128a commit 73d0f1620a269f990dbd3d2796abf27e9a05e227 Author: Béla Vancsics Date: Tue Dec 8 10:17:08 2015 +0100 Not used parameter The account variable was not used in the method. Change-Id: I8e91d7616529f33b615bc52af76bfda01141d364 commit 7035639dfd239b52d4ed46aae50f78d16ec8cbfe Author: Clay Gerrard Date: Thu Oct 15 16:20:58 2015 -0700 Put part-replicas where they go It's harder than it sounds. There was really three challenges. Challenge #1 Initial Assignment =============================== Before starting to assign parts on this new shiny ring you've constructed, maybe we'll pause for a moment up front and consider the lay of the land. This process is called the replica_plan. The replica_plan approach is separating part assignment failures into two modes: 1) we considered the cluster topology and it's weights and came up with the wrong plan 2) we failed to execute on the plan I failed at both parts plenty of times before I got it this close. I'm sure a counter example still exists, but when we find it the new helper methods will let us reason about where things went wrong. Challenge #2 Fixing Placement ============================= With a sound plan in hand, it's much easier to fail to execute on it the less material you have to execute with - so we gather up as many parts as we can - as long as we think we can find them a better home. Picking the right parts for gather is a black art - when you notice a balance is slow it's because it's spending so much time iterating over replica2part2dev trying to decide just the right parts to gather. The replica plan can help at least in the gross dispersion collection to gather up the worst offenders first before considering balance. I think trying to avoid picking up parts that are stuck to the tier before falling into a forced grab on anything over parts_wanted helps with stability generally - but depending on where the parts_wanted are in relation to the full devices it's pretty easy pick up something that'll end up really close to where it started. I tried to break the gather methods into smaller pieces so it looked like I knew what I was doing. Going with a MAXIMUM gather iteration instead of balance (which doesn't reflect the replica_plan) doesn't seem to be costing me anything - most of the time the exit condition is either solved or all the parts overly aggressively locked up on min_part_hours. So far, it mostly seemds if the thing is going to balance this round it'll get it in the first couple of shakes. Challenge #3 Crazy replica2part2dev tables ========================================== I think there's lots of ways "scars" can build up a ring which can result in very particular replica2part2dev tables that are physically difficult to dig out of. It's repairing these scars that will take multiple rebalances to resolve. ... but at this point ... ... lacking a counter example ... I've been able to close up all the edge cases I was able to find. It may not be quick, but progress will be made. Basically my strategy just required a better understanding of how previous algorithms were able to *mostly* keep things moving by brute forcing the whole mess with a bunch of randomness. Then when we detect our "elegant" careful part selection isn't making progress - we can fall back to same old tricks. Validation ========== We validate against duplicate part replica assignment after rebalance and raise an ERROR if we detect more than one replica of a part assigned to the same device. In order to meet that requirement we have to have as many devices as replicas, so attempting to rebalance with too few devices w/o changing your replica_count is also an ERROR not a warning. Random Thoughts =============== As usual with rings, the test diff can be hard to reason about - hopefully I've added enough comments to assure future me that these assertions make sense. Despite being a large rewrite of a lot of important code, the existing code is known to have failed us. This change fixes a critical bug that's trivial to reproduce in a critical component of the system. There's probably a bunch of error messages and exit status stuff that's not as helpful as it could be considering the new behaviors. Change-Id: I1bbe7be38806fc1c8b9181a722933c18a6c76e05 Closes-Bug: #1452431 commit 62b9a0eeeaeb3a5a0daff981cf6fa1cb50e78a50 Author: Clay Gerrard Date: Mon Dec 7 10:19:51 2015 -0800 fix mock and assert in test_direct_client Change-Id: Ia6c14377e5dd768a5306a2448f0bf244ebc3634e commit b63caf4521b0a4a32b9afdb6d247537848d0a8d6 Author: Tushar Gohad Date: Wed Nov 11 12:53:29 2015 -0700 Bump PyECLib requirement to >=1.0.7 ... to match global-requirements. This is an intermediate step in updating requirements to PyECLib>=1.1.1. Change-Id: I79f47fff6ec1adff214ca435f0d95aaf1ffd68f9 commit 73ded056aff0f25605fee2a633f3d13e0369606a Author: Christian Schwede Date: Tue Dec 1 14:59:35 2015 +0000 Add functional test for repeated SLO segments Currently fails on master, passes with change If13af444ed301ebd8fd34a0d96a330ded648f0c4 applied. Change-Id: I05b231cc232d5b4117bccee40eebc7093114c61c commit 679e6d5251c804cab8fd41d7a13e6bdd43967842 Author: John Dickinson Date: Mon Nov 30 11:58:40 2015 -0800 Add config sample reference to KeystoneAuth docs Change-Id: I38a7bd7b85de9d674386d75220b31cc467746da8 commit 5eaa5543c766536e48127d60e03cebe9c2824abb Author: John Dickinson Date: Mon Nov 30 10:33:23 2015 -0800 add sample proxy pipeline for keystone integration Change-Id: I4b4fd9179d0234f001940e215c97d40a2a6204cd commit 28c4b7310fead32f7ce073ee4bb503a450e521f1 Author: Peter Lisák Date: Tue Nov 24 16:33:35 2015 +0100 Unification of manpages and conf-samples (default values, etc) Change-Id: I47a3127ef698b4bd1537b1562901ee9c2b5924d4 commit 8e5b38b1ddb71d631c12368e6d917beb25d3a160 Author: Tim Burke Date: Wed Oct 7 16:47:11 2015 -0700 Expose tempurl's header restrictions via /info Also, clean up the module documentation a bit. Change-Id: Iaeb5eb264b118b78738187db9242540275e77444 commit 635bc7fa8f9a442b39af4955e98fcd2e2acf1e0a Author: Ferenc Horváth Date: Wed Oct 7 11:10:28 2015 +0200 Replace string slicing with proper string methods Updated string prefix and suffix checker slicing to startswith() and endswith() methods. Using startswith() and endswith() improves readability, error-proneness and enhances maintainability. Change-Id: I1d5fbf116a61763346c6f92fd8023dbbe9bb37cf commit 6c43bdc82bc27f1f60bd32c43fac6e8be562e136 Author: Alistair Coles Date: Thu Nov 26 14:46:01 2015 +0000 Test for content-type params in multi-range response Updates the functional test to verify the fix applied by change Iff7274aa631a92cd7332212ed8b4378c27da4a1f Change-Id: Iae63ac027e4f4acfe46a36dc1325888b1f834ea4 commit b339e529c3464278b7ddd5670acd903070d1da1a Author: Samuel Merritt Date: Wed Mar 25 14:59:43 2015 -0700 Add functional test for multi-range GET requests. Change-Id: I9d417faede707e4f3570074e410344cc8955007b commit c28709f8f264f4426364b67a2a2755311111c1e4 Author: Samuel Merritt Date: Wed Nov 25 17:09:34 2015 -0800 Fix crash when a SLO repeats a segment. If a SLO manifest specified the same segment twice with no range on the latter, we would crash while trying to coalesce the segment specifications into a single request (for efficiency). Now we handle that missing range and do the right thing. Change-Id: If13af444ed301ebd8fd34a0d96a330ded648f0c4 commit ad722b3ed3ff1b92ebee829db3c289040960b24d Author: Alistair Coles Date: Wed Nov 25 18:49:39 2015 +0000 Include params in multipart message part content-type The content-type header inserted into a multipart message part is missing any params such as charset because its value is being fetched via the swob.Response content_type property, which conforms to webob spec and strips off all params. This was noticed in work on feature/crypto branch because the crypto meta param was being stripped off content-type in multipart messages, preventing the content-type being decrypted. But in general there is no suggestion in the multipart message spec [1] that params should not be included. In fact examples in [1] show the charset param included in the content-type value. To ensure that the multipart message part content-type includes the original content-type params, fetch it directly from the response headers. [1] http://www.w3.org/Protocols/rfc1341/7_2_Multipart.html Change-Id: Iff7274aa631a92cd7332212ed8b4378c27da4a1f commit e06c8743941a27c1a3f50396385268b9e812c32e Author: Alistair Coles Date: Wed Nov 25 14:10:27 2015 +0000 Improve swift-init usage statement Clarify the behavior of strict and non-strict options. Define what and specify on the command line. Drive-by fix for missing param in search_tree docstring. Change-Id: I89fff88a088bc73001464b1fa8240e14a61a117d commit 0e5cc89da56f550468512af6f3479382a1894815 Author: Tim Burke Date: Tue Oct 13 11:45:20 2015 -0700 Have versioned_writes use reverse listings This will reduce the overhead required for versioned containers in terms of memory, back-end requests, and wall time. Change-Id: I156216a14141db547de3cae1dcd5315ae31eaac2 commit aeccbc2074224c6688be6ef5aa8ba477c9631ba1 Author: Alistair Coles Date: Tue Nov 24 14:47:30 2015 +0000 Add more functional tests for reverse listings Also fix typo in the api doc. Follow-on to change I5eb655360ac95042877da26d18707aebc11c02f6 Change-Id: Ic1ea63c74a0a3b90a505865fba8fcfac584d0825 commit 41897d96a7dc3ddceccf4eed71345439b83eb243 Author: Tim Burke Date: Mon Nov 23 13:53:27 2015 -0800 Reverse-listings follow-up * With the end_prefix changes in the original commit, we no longer need the `or not name.startswith(prefix)` check. * Improve test coverage of reverse path listings. Change-Id: Iaa7d4b83647c3c150be95f88cb3cc9e4f0e33979 commit 7c1e6cd583debe43aca266643cd65f5103159dbf Author: Matthew Oliver Date: Thu Sep 11 16:51:51 2014 +1000 Add container and account reverse listings This change adds the ability to tell the container or account server to reverse their listings. This is done by sending a reverse=TRUE_VALUE, Where TRUE_VALUE is one of the values true can be in common/utils: TRUE_VALUES = set(('true', '1', 'yes', 'on', 't', 'y')) For example: curl -i -X GET -H "X-Auth-Token: $TOKEN" $STORAGE_URL/c/?reverse=on I borrowed the swapping of the markers code from Kevin's old change, thanks Kevin. And Tim Burke added some real nuggets of awesomeness. DocImpact Co-Authored-By: Kevin McDonald Co-Authored-By: Tim Burke Implements: blueprint reverse-object-listing Change-Id: I5eb655360ac95042877da26d18707aebc11c02f6 commit 898223398189f96628db7d9928f146c5bb11ba88 Author: Ganesh Maharaj Mahalingam Date: Fri Oct 30 21:30:51 2015 +0000 Unit tests for account/backend.py Add test for database create request without account Partial test for migrate call on database with storage_policy_index Change-Id: I7cfbd6bc7e2b341f433d88f600b19e54826a0e22 Signed-off-by: Ganesh Maharaj Mahalingam commit f80c97a6e964123e766c1e53d797a3b80433ac77 Author: Alistair Coles Date: Fri Nov 20 17:51:33 2015 +0000 Update .mailmap entry Fix/add entries for authors that are known to now have @hpe.com email addresses. Change-Id: I62c83b64e2378cb3d6ad33ac9e6ab111b8a8c86f commit 16de32f1686a60fa5b3aa1802dcc42783846b05b Author: Peter Lisák Date: Fri Nov 6 10:49:09 2015 +0100 Log error if a local device not identified in replicator Example: * Different port in config and in ring file. * Running daemon on server not in ring file. In both cases replication daemon is running but nothing is replicated. Error log helps to distinguish a local device can't be identified. Closes-Bug: 1508228 Change-Id: I99351b7d9946f250b7750df91c13d09352a145ce commit 848d7299a26988ea1bab3e576ba10762624dfff6 Author: Tim Burke Date: Fri Nov 20 09:04:28 2015 -0800 Ignore Content-Type from client on multipart-manifest=delete Otherwise, if a client includes something like: Content-Type: application/x-www-form-urlencoded ...we won't delete anything, and instead send back: Number Deleted: 0 Number Not Found: 0 Response Body: Response Status: 406 Not Acceptable Errors: ...despite the fact that we're managing the iterator, so we know it's acceptable. Change-Id: I791c7bda1d9df830d8dacd3783c2393db5a9ac09 commit c4eeea7820b7bbf6f5994c9878c8838c43e61b9b Author: Sivasathurappan Radhakrishnan Date: Thu Oct 29 23:04:42 2015 +0000 Added unit test cases for container/auditor.py Added unit test cases to cover all code path in _one_audit_pass function in container/auditor.py Change-Id: I8b89e94f1492e4366af3ac4260587e988ba29408 commit bdb7bf40321bc36b45a25742b6e1aed73576fabc Author: Victor Stinner Date: Thu Aug 27 01:09:41 2015 +0200 py3: Add py34 test environment to tox Add a py34 test environment to tox.ini using a whitelist of tests which pass on Python 3. Currently, only test_exceptions.py of test/unit/common/ is run on Python 3 Change-Id: I5c8a9fe7b9635f1acab8f401908975fc6fc58c7a commit 4dc4cbfc1ca50abc100d2302e928f9fc16f26195 Author: Samuel Merritt Date: Mon Nov 16 14:27:51 2015 -0800 Fix 503 on zero-byte replicated PUT with incorrect Etag Closes-Bug: 1516579 Change-Id: Iac91ed61254d3ca232521191fec25c19acb66413 commit 1b8b08039a3647ffdb30225b0939055982942159 Author: Zack M. Davis Date: Fri Oct 30 11:02:54 2015 -0700 remove remaining simplejson uses, prefer standard library import a1c32702, 736cf54a, and 38787d0f remove uses of `simplejson` from various parts of Swift in favor of the standard libary `json` module (introduced in Python 2.6). This commit performs the remaining `simplejson` to `json` replacements, removes two comments highlighting quirks of simplejson with respect to Unicode, and removes the references to it in setup documentation and requirements.txt. There were a lot of places where we were importing json from swift.common.utils, which is less intuitive than a direct `import json`, so that replacement is made as well. (And in two more tiny drive-bys, we add some pretty-indenting to an XML fragment and use `super` rather than naming a base class explicitly.) Change-Id: I769e88dda7f76ce15cf7ce930dc1874d24f9498a commit e534af645d4ff051e97e552e98271810b5588cfe Author: Tim Burke Date: Fri Nov 13 12:38:56 2015 -0800 Remove references to now-missing havana docs Change-Id: I3c708bacd468e923e574bdd5afc7acc58b072159 commit 7b7c6c5249ba598f10770308e05f3d2ad4e60871 Author: Lisak, Peter Date: Fri Oct 2 09:20:17 2015 +0200 swift-init return codes Currently, swift-init returns zero if can't locate config on start. Because of this problem, it is not possible to distinguish if managed to start server. Due to legacy two new complementary options are added. Default is context dependent. --strict returns non-zero if some config is missing (default mode if explicitly named server) --non-strict returns zero even if some config is missing (default mode if alias is used) As a side effect: If some of demanded servers already running it does not try to start unstarted and also returns non-zero (in strict mode). That is still sufficient for the goal of patch. For future improvements LSB status codes should be considered. DocImpact Change-Id: I7750abd4a94875b46f83f4aeee8509388d543c2b commit 84891ba5a1a06013779c9be24f69b26fea0cb4f0 Author: Ondřej Nový Date: Sun Nov 15 19:25:05 2015 +0100 New swauth URL Change-Id: I019f921417d2bd5325141e8a6873ee934fdf3f49 commit 001dc2531899a61480ab7819ebe8766dbf3d7d22 Author: Ondřej Nový Date: Sat Nov 14 20:14:46 2015 +0100 Missing log_name option added Change-Id: If0224f346aa7372268115284c112e8f3a60a9be4 commit 376c20d1889ab764b4a501c59ba6539b051f04bb Author: Zack M. Davis Date: Fri Nov 13 12:08:03 2015 -0800 remove Python 2.6-specific logging workaround 0ff39c14 (December 2013) introduced this LoggingHandlerWeakRef to work around a problem in which file descriptors would be leaked under Python 2.6. But we don't support 2.6 anymore. Change-Id: I1cd8249dfe17bce9cb301d13b761c11d2efd11d3 commit 6e95fb5d164e6fdc3bf036c40dd54a19fd67199a Author: John Dickinson Date: Fri Nov 13 09:46:02 2015 -0800 remove pbr from requirements.txt. It's not a run-time dependency Change-Id: I46f498122fccb0830304acba7f99fee64badfe12 commit c03d53ab7737e208d9c436088e41f60fdab96b30 Author: Tim Burke Date: Wed Nov 4 12:25:19 2015 -0800 Close EC fragment iterators in the GreenThread that's consuming them Otherwise, we might try to close them while they're already running. This should prevent conditional SLO requests from returning 500 when they should have returned 304 or 412. Change-Id: I075a892f901c18ab5c178c9c4a2f367b76ae78e2 Related-Change: I156d873c72c19623bcfbf39bf120c98800b3cada commit fd30df6e6572791e89a5360cecbf4800f1f08bb5 Author: Kota Tsuyuzaki Date: Wed Nov 11 07:55:51 2015 -0800 Small cleanup for unit/proxy/controllers/test_obj Follow up for https://review.openstack.org/#/c/236007 This fixes following minor items: - Fix a 'raise Exception class' syntax to 'raise Exception instance' - Use original eventlet.Timeout instead of swift.exceptions.Timeout imported from eventlet.Timeout - Change Timeout to initiate w/o args (1st arguments should be timeout second and we don't have to set None if we don't want to set the sec) - Add a message argument to some Exception instances Change-Id: Iab608cd8a1f4d3f5b4963c26b94ab0501837ffe1 commit 7f4139bc2669de07f55e8689956d87d9c78cc834 Author: Paul Dardeau Date: Wed Oct 28 22:25:24 2015 +0000 Added unit tests for ringbuilder command-line utility Added new unit tests: test_add_device_old_missing_region test_create_ring_number_of_arguments test_add_duplicate_devices test_rebalance_with_seed test_set_overload_number_of_arguments test_main_no_arguments test_main_single_argument test_main_with_safe Modified existing unit tests to create sample ring at start of test. This change was needed to have unit tests run correctly and demonstrate code coverage. test_unknown test_search_device_number_of_arguments test_list_parts_number_of_arguments test_set_weight_number_of_arguments test_set_info_number_of_arguments test_remove_device_number_of_arguments test_set_min_part_hours_number_of_arguments test_set_replicas_number_of_arguments test_set_replicas_invalid_value Updates to handled nested mocks. Updates to handle no exception case when SystemExit is expected. PEP8 corrections. Moved new tests from try blocks to use of assertRaises or call to run_srb using exp_results with specified exit codes. Updated run_srb to accept a dictionary of expected results. Specifically, look for 'valid_exit_codes' to test, default to (0,1). Change-Id: I4cf3f5f055a9babba140c68a9c7ff90b9c50ea62 commit ce173e9ed2e14692899a53da7eefe31f4a7c61d0 Author: Sivasathurappan Radhakrishnan Date: Fri Nov 6 01:49:58 2015 +0000 Added unit test cases for server.py Added unit test cases to cover all code paths in REPLICATE and __call__ functions in account/server.py and container/server.py Change-Id: Ia335e9a6668821d3e34b12fc3a133a707880e87f commit 2d85a3f6997fbadac210207f841c4ec25ff50cc4 Author: Tushar Gohad Date: Fri Oct 23 13:16:33 2015 +0000 EC: Use best available ec_type in unittests To minimize external library dependencies for Swift unit tests and SAIO, PyECLib 1.1.1 introduces a native backend 'liberasurecode_rs_vand.' This patch is to migrate over the unit tests to the new ec_type when available. This change will work with current pyeclib requirements (==1.0.7) and also future requirements (>=1.0.7). When we're able to raise *our* requirements to >=1.1.1 we should remove jerasure from the list of preferred backends. Related SAIO doc and example config changes should be included with that patch. Co-Authored-By: Clay Gerrard Change-Id: Idf657f0acf0479bc8158972e568a29dbc08eaf3b commit 66dc1eebb1db51c81891b1f7bcc3e85aef9b8c1d Author: Bill Huber Date: Fri Oct 16 11:27:34 2015 -0500 ObjectControllers return application errors as 499 on bad read In the _transfer_data method, we translate all (Exception, Timeout) into a 499 whereas we should consider translating them to 500 on particular returning error scenarios. This affects both ReplicatedObjectController and ECObjectControllear. Change-Id: I571bbc5b1451243907b094a5718c8735fd824268 Closes-Bug: 1504299 commit 4d0d76eb2079885c55711238906ef4ddd252aa86 Author: Ganesh Maharaj Mahalingam Date: Tue Nov 3 16:43:57 2015 +0000 Add unit tests for direct_client Add tests for direct_delete_account, direct_get_object, retry Tests to validate possible argument calls of direct_put_object Extend FakeConn to support chunks in response Change-Id: I00cbc6c535ed8b3fb29bc07fdc51fdbb1220ff10 Signed-off-by: Ganesh Maharaj Mahalingam commit 705642db4c27c34ac8eb840d40b16470b85418c8 Author: venkatamahesh Date: Tue Nov 10 19:46:31 2015 +0530 Change stackforge repo to openstack repo The projects which are moved to openstack are corrected by replacing stackforge with openstack Change-Id: I65b794a7f10df617bc2a4caf2c4010477a82dbc2 commit d755f5b52024e255787d180bbca60c09192a8850 Author: John Dickinson Date: Mon Nov 9 22:03:34 2015 -0800 suppress warning output in a unit test test_write_builder_after_device_removal() wasn't setting a default min_part_hours so a warnign was printed. Explicitly adding a min_part_hours suppresses the warning Change-Id: I6f234b72c34e066abb91f28e6eacf50e29be8842 commit 37337d5fcb0a29344381f8de7d810a9e76e10928 Author: Pradeep Kumar Singh Date: Sun May 17 13:35:58 2015 +0530 Read the response body, if response status is greater than 300. internal_client was not reading response if response status is not 200. So proxy server treats this as client disconnect and logs 499 in log file. This patch fixes that by reading response if response status is greater than or equal to 300 and in acceptable statuses. Closes-Bug: #1364752 Change-Id: I0512a25895da583956f76031e3c5de5c970bce01 commit c81f202f714c36390a25b24fa96d1b277788ade9 Author: Kota Tsuyuzaki Date: Thu Nov 5 17:37:30 2015 -0800 Add missing docs for ring.builder.rebalance commit 71993d84e88cc1d5f7742182905cace21c7e88cb added a new 'remove_dev' column to the swift.common.ring.builder.rebalance return value. This patch adds the docs for that and clean up a bit to the variable name to be easy to read. Change-Id: Idfd46e47b9f6894cbafc8b7701a4c7414212f79f commit 331382e1922add01959b88e58bc8c1d2029f4550 Author: Paul Dardeau Date: Thu Oct 29 21:09:01 2015 +0000 Add unit tests to cover print_item_locations Add unit tests to cover all code paths in print_item_locations function in cli/info.py. Update comment to match what's tested for invalid/missing policy. Update tests to verify output of print_item_locations Corrected PEP8 compliance violations. Change-Id: I84958cb70205ee8d7ea246826dd56201fa642da9 commit e0430fc74a564a0556ef5eab1accd110566aee31 Author: Ondřej Nový Date: Sun Nov 1 18:10:06 2015 +0100 Compare Swift config checksum in swift-recon --all Change-Id: I796fe0895f4e5ddeb04c0d79a73579ce8bb9aa40 commit 609095b23f6c5734c6a7aed80c488b96b31137e8 Author: Victor Stinner Date: Thu Aug 27 00:51:58 2015 +0200 On py3, use dnspython3 dependency, not dnspython Update requirements to install dnspython3 on Python 3 and dnspython on Python 2. dnspython is not compatible with Python 3, and dnspython3 is not compatible with Python 2. Add a requirement on pbr to ensure that pbr 1.6 or newer is installed. pbr 1.0 or newer is required to support environment markers in requirements which are now used to select dnspython or dnspython3 depending on the Python major version. Note: pbr 1.0 is enough, but OpenStack global requirements asks to use "pbr>=1.6". Change-Id: Ie5d437cd396972d4143ffffb96ec3c289e221b08 commit 6858510b59c23ebef527a92c3258dc3c5c6046f5 Author: Alistair Coles Date: Thu Sep 3 16:13:17 2015 +0100 Re-organise ssync tests We have some tests that exercise both the sender and receiver, but are spread across test_ssync_sender.py and test_ssync_receiver.py. This creates a new module test_ssync.py and moves the end-to-end tests into there. Change-Id: Iea3e9932734924453f7241432afda90abbc75c06 commit 48421f902e2e293a3c9ef672dbec5971ca530fd6 Author: Venkateswarlu Pallamala Date: Wed Nov 4 00:53:05 2015 -0800 Optimize the code performance Usually, the number of characters in FORBIDDEN_CHARS are fewer than the number of characters in the req.path - so it's better to use smaller count for the outer loop. Change-Id: I78b7e0ff4cae1cc90e8d29d7bddcec7fe0ba4bd4 commit 213b1a4900a75f6a0eac4af6ee53ca9c367bbd41 Author: Paul Dardeau Date: Mon Oct 26 21:17:05 2015 +0000 Added unit tests to recon.py. Added tests for unrecognized recon_types for get_replication_info, get_updater_info, and get_auditor_info. Added tests for get_device_info and get_swift_conf_md5. PEP8 compliance fixes. Updates based on code reviews. Change-Id: I3f636955b253ea7f65d4a1f333a0ad24f1ea6a1f commit bf8689474a9b3930cceed4c5f855a73ce89b1d4e Author: Mehdi Abaakouk Date: Thu Oct 22 17:33:09 2015 +0200 monkeypatch thread for keystoneclient keystoneclient uses threading.Lock(), but swift doesn't monkeypatch threading, this result in lockup when two greenthreads try to acquire a non green lock. This change fixes that. Change-Id: I9b44284a5eb598a6978364819f253e031f4eaeef Closes-bug: #1508424 commit faef717cd3fda6aeb3a0799f25d0483bc9bdce7a Author: Bill Huber Date: Thu Oct 22 10:14:29 2015 -0500 Add unit tests for swift.proxy.controllers.base This patch adds more unit tests to diminish missing pieces of the coverage in the proxy_controllers_base unit test. Change-Id: I85ba1955c681cc9d5b2a70ac31155678d2e5b6fd commit febdd6c1b4035a3b63b5deb7690b2a41014e5321 Author: Peter Lisák Date: Fri Oct 23 12:40:15 2015 +0200 Device marked to be removed in info about the ring. Showing devices with 'DEL' mark if will be removed next rebalance. Change-Id: I171aa8658b1c4ac1689ab9532fe65d114567baa7 commit b96fd0d7d8e294731dc28352cd6e586919058f90 Author: Hisashi Osanai Date: Tue Jul 21 21:00:01 2015 +0900 Remove _keystone_identity method _keystone_identity method has been maintained for backward compatibility. But there is no place to use it now so this patch replace _keystone_identity method to _integrated_keystone_identity method as _keystone_identity. Change-Id: I9464c047401f92ae31a5d3bb7aacaeb0624150e0 commit a4a5fcad8f1ec9059cfab0c63e81106355b0dbd5 Author: Kota Tsuyuzaki Date: Wed Oct 7 02:14:49 2015 -0700 Add unit tests for utils.validate_hash_path This patch add some unit tests to prevent regression and to describe validate_hash_path behavior, I found in review with https://review.openstack.org/#/c/231864/ *bonus* - Fix test_hash_path to use "with" syntax instead of "try/finally" for assigning a testing value into a global variable. Change-Id: I948999a8fb8addb9a378dbf8bee853b205aeafad commit e31ecb24b6684da692fbca8b7fe34c1dd4047a3a Author: Samuel Merritt Date: Wed Oct 21 13:05:37 2015 -0700 Get rid of contextlib.nested() for py3 contextlib.nested() is missing completely in Python 3. Since 2.7, we can use multiple context managers in a 'with' statement, like so: with thing1() as t1, thing2() as t2: do_stuff() Now, if we had some code that needed to nest an arbitrary number of context managers, there's stuff we could do with contextlib.ExitStack and such... but we don't. We only use contextlib.nested() in tests to set up bunches of mocks without crazy-deep indentation, and all that stuff fits perfectly into multiple-context-manager 'with' statements. Change-Id: Id472958b007948f05dbd4c7fb8cf3ffab58e2681 commit 73e032049ffb9b371f30d5230472b920446d08f4 Author: Paul Dardeau Date: Fri Oct 23 18:20:25 2015 +0000 Update admin guide with region. Added region prefix to example commands for adding devices to ring. Also updates description to include region prefix. Change-Id: Ie6d6485b497cea973e37909b5b19b44946c8aa89 commit 5d8d2e98ee8004cb009f4bd419fbd8cd5757aaa2 Author: Alistair Coles Date: Fri Oct 23 16:38:24 2015 +0100 Minor improvement to updater unit tests Follow up for change I7c57bba053711a27d3802efe6f2a0bf53483a54f The current tests won't prevent this kind of regression in the updaters' __init__ methods: self.node_timeout = int(float(conf.get('node_timeout', 10))) so this makes them a little tighter. Change-Id: I60bd79ac392deb1e8a108357d48474dddd9028c1 commit 1a2b54fc0a22713ff0ca575e0a22834682670c55 Author: Alistair Coles Date: Mon Oct 19 13:55:02 2015 +0100 Fix missing *-replicator conf sections in deployment guide The doc for these sections was missing because of an rst error - the source is there in rst file but didn't make it into the html output. Add doc for per_diff and max_diffs in account and container doc sections. Also, fix a bunch of other sphinx build errors and most of the warnings. Change-Id: If9ed2619b2f92c6c65a94f41d8819db8726d3893 commit 8f78c1a2b066e816cfd643d048ebbe3b654fbe87 Author: OpenStack Proposal Bot Date: Fri Oct 23 06:34:15 2015 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: Ifebb8f1b50ac4c2c0f43df48c2414ce0f3742e42 commit 59849a9923d7e214725afc14d0d229194884fc54 Author: Zack M. Davis Date: Thu Oct 22 15:13:09 2015 -0700 SLO docs(tring): manifest params in table, code block; copyedits It's æsthetically pleasing to display the segment description parameters in a table (using the reStructuredText "simple" table markup) and to use a code block (which receives syntax hightlighting in the rendered document!) for the example JSON fragment. Also, a few very minor copy changes are made (a comma after an introductory phrase clarifies; JSON and ETag receieve their canonical capitalizations; the elements of the manifest JSON aren't "files" exactly; the inclusive nature of the range is noted for the benefit of harried readers who won't read the "Range Specification" section later in the document). Change-Id: I5296fc04c6fe5410ad14bba7de0a7f8ab060b126 commit d877a422374e50ddf89dc001fb687ce85de1c6e8 Author: Victor Stinner Date: Mon Oct 19 17:16:21 2015 +0200 Fix Python 3 issues in utils * LogAdapter.exception(): on Python 3, socket.error is an alias to OSError, so group both exceptions to support Python 3. * ThreadPool: call GreenPipe() with indexed parameter, don't pass the third parameter by its name, since the parameter name changed between Python 2 (bufsize) and Python 3 (buffering). * strip_value() in test_utils: StringIO.getvalue() now requires seek(0), otherwise the buffer is filled with null characters. * test_lock_file(): on Python 3, seek(0) is now required to go the beginning of a file opened in append mode. In append mode, Python goes to the end of the file. Change-Id: I4e56a51690f016a0a2e1354380ce11cff1891f64 commit b3a0acd5c77201161dcd169850df589ede331106 Author: Ganesh Maharaj Mahalingam Date: Tue Oct 20 15:16:39 2015 +0000 DiskFileNoSpace check during object delete On a full disk, a call to delete an object will fail when it tries to write tombstones. Handling DiskFileNoSpace exception raised by swift.common.utils. Change-Id: I8f0cfcc4159ee154fcd3e7ca90c422aa5aadf0b3 Signed-off-by: Ganesh Maharaj Mahalingam Closes-Bug: 1491675 commit b376e9dc342a2ea4427467b7cd276f335298134d Author: Kota Tsuyuzaki Date: Wed Oct 21 05:33:04 2015 -0700 Add verification socket close causes StopIteration This patch add a expositive test that closing connection (fd and socket) causes StopIteration on reader at proxy server on PUT object case. And then, if the connection closed and proxy detects a lack of data length, it will result in HTTPClientDisconnected. The only reason why I leave test_client_disconnect as current is Bill is now working for the change at https://review.openstack.org/#/c/236007/ Change-Id: If68f752a2b0dd6cfad4d007ca5d155e6ffb378e5 commit 71993d84e88cc1d5f7742182905cace21c7e88cb Author: Lisak, Peter Date: Fri Oct 9 16:13:55 2015 +0200 swift-ring-builder can't remove a device with zero weight If a device with 0 weight is tried to remove, the following rebalance does not write changes into builder file. Scenario: $ swift-ring-builder object.builder set_weight --id 1 0.00 $ swift-ring-builder object.builder rebalance Wait for moving files out of the device id=1. $ swift-ring-builder object.builder remove --id 1 $ swift-ring-builder object.builder rebalance In fact, the device id=1 is not removed after rebalance (must be --force used). Change-Id: Iad5a444023eae9882a3addd7f119ff4d18559ddd commit 2135841e6e3fa6705562a72030c31f0c69fd4470 Author: asettle Date: Wed Oct 7 13:55:45 2015 +1000 Documenting the ability to specify ranges for SLO segments Change-Id: Ied854f6385f3b28975808f4707e531581bf6c18e Closes-bug: 1499152 commit b6b7578190bc5de266d08c22984622d5566b8456 Author: Lisak, Peter Date: Tue Oct 13 17:00:41 2015 +0200 node_timeout as float in configs It is more convenient to use float node_timeout for fine tunning latency. Change-Id: I7c57bba053711a27d3802efe6f2a0bf53483a54f commit 71e573f4ba8a45dd5962f9d180c6236ce85627ca Author: Victor Stinner Date: Mon Oct 19 16:59:14 2015 +0200 Port swift.common.utils.backward() to Python 3 backward() is written to handle binary files: * Replace literal native strings to literal byte strings: add b'...' prefix * Update unit tests: use byte strings * TemporaryFile(): use the default mode 'w+b', instead of using 'r+w' ('r+w' mode creates a Unicode file on Python 3). Change-Id: Ic91f7e6c605db0b888763080d49f0f501029837f commit 4a13dcc4a8d7c890ff322190ca80fbb460b55de4 Author: Matthew Oliver Date: Fri Oct 9 17:25:08 2015 +1100 Make db_replicator usync smaller containers The current rule inside the db_replicator is to rsync+merge containers during replication if the difference between rowids differ by more than 50%: # if the difference in rowids between the two differs by # more than 50%, rsync then do a remote merge. if rinfo['max_row'] / float(info['max_row']) < 0.5: This mean on smaller containers, that only have few rows, and differ by a small number still rsync+merge rather then copying rows. This change adds a new condition, the difference in the rowids must be greater than the defined per_diff otherwise usync will be used: # if the difference in rowids between the two differs by # more than 50% and the difference is greater than per_diff, # rsync then do a remote merge. # NOTE: difference > per_diff stops us from dropping to rsync # on smaller containers, who have only a few rows to sync. if rinfo['max_row'] / float(info['max_row']) < 0.5 and \ info['max_row'] - rinfo['max_row'] > self.per_diff: Change-Id: I9e779f71bf37714919a525404565dd075762b0d4 Closes-bug: #1019712 commit 43cdc29669741568a898dc87eee3e09280887bba Author: Kota Tsuyuzaki Date: Tue Oct 13 00:22:31 2015 -0700 Add backend subrequest tests for staticweb This is an followup for https://review.openstack.org/#/c/227204 Above patch changes staticweb to use just make_env instead of pre authed env. The tests included in this patch assert "backend requests generated by staticweb *NEVER* include pre authed information (i.e. authorize_override=True)". Change-Id: Ifbbc43d3b5ac114bc3d5e4fcab9cacdf9a3f1a5c commit 9993c1f124bcb6d645254196fe43d8c4f9feae6e Author: Alistair Coles Date: Wed Oct 14 15:49:35 2015 +0100 Add test to verify staticweb uses auth'd subrequests Follow on for change Icf159d7e567ac5481e710c5910db686bdcba6336 Related-Bug: 1489749 Change-Id: I96920645d50322c5232bbcd6bb38b0f26795b92d commit 4eb796b29ddee52f5010655e0d5477f2d7923854 Author: Carlos Cavanna Date: Thu Oct 8 13:40:20 2015 -0400 bug/1270114. Missing documentation for Bulk Middleware. Closes-Bug: #1270114 This patch addresses the missing bulk middleware documentation for the following bug: https://bugs.launchpad.net/swift/+bug/1270114 And feature: https://review.openstack.org/#/c/176161/ The documentation will show under: http://docs.openstack.org/developer/swift/middleware.html Change-Id: I73c101e09830d7120b5e5c1333ae8d570a9e2303 commit a7af802497f3b86d33cbee852500df456c316685 Author: Christian Schwede Date: Sat Aug 29 16:03:07 2015 +0000 Do not use pre-authenticated requests in staticweb staticweb middleware uses make_pre_authed_env, this makes it possible to anonymously list containers without any read acl set if the metadata "web-listings: true" is set on a container. Using make_env enforces correct read_acl validation; however it is now also required to add ".rlistings" to the read acl. Also, if the staticweb middleware is put in the proxy pipeline before an authentication middleware, it broke authenticated GET and HEAD requests. This has some side effects in clients, because a html response is sent that might be parsed wrongly by the client. In case of python-swiftclient this was shown as an empty container without any ACL or web-listings:true meta set. This might lead to information leaks, because a user trusts the output from python-swiftclient and assumes an empty, private container even if the container contains public readable data. staticweb now checks if "swift.authorize" is included in the environ and skips itself if not. Closes-Bug: 1489749 Change-Id: Icf159d7e567ac5481e710c5910db686bdcba6336 Depends-On: Ie24bb995023c377e49796910ad80a256b00daa03 commit 752ceb266b3e72361242250e50396671df8b9595 Author: Clay Gerrard