Reviewed: https://review.openstack.org/400985 Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0c3f8f87104af8717115c5badffd243dbaa1c430 Submitter: Jenkins Branch: feature/hummingbird commit 2d25fe6ad3573b2a06b6b3e5e66493d7b0c55693 Author: Tim Burke Date: Mon Jul 25 15:06:23 2016 -0700 Reduce backend requests for SLO If-Match / HEAD requests ... by storing SLO Etag and size in sysmeta. Previously, we had to GET the manifest for every HEAD or conditional request to an SLO. Worse, since SLO PUTs require that we HEAD every segment, we'd GET all included sub-SLO manifests. This was necessary so we could recompute the large object's Etag and content-length. Since we already know both of those during PUT, we'll now store it in object sysmeta. This allows us to: * satisfy HEAD requests based purely off of the manifest's HEAD response, and * perform the If-(None-)Match comparison on the object server, without any additional subrequests. Note that the large object content-length can't just be parsed from content-type -- with fast-POST enabled, the content-type coming out of the object-server won't necessarily include swift_bytes. Also note that we must still fall back to GETting the manifest if the sysmeta headers were not found. Otherwise, we'd break existing large objects. Change-Id: Ia6ad32354105515560b005cea750aa64a88c96f9 commit ae7dddd801e28217d7dc46bd45cd6b621f29340c Author: Ondřej Nový Date: Mon Nov 21 22:13:11 2016 +0100 Added comment for "user" option in drive-audit config Change-Id: I24362826bee85ac3304e9b63504c9465da673014 commit c3e1d847f4b9d6cc6212aae4dc1b1e6dff45fb40 Author: Thiago da Silva Date: Thu Nov 17 17:17:00 2016 -0500 breaking down tests.py into smaller pieces tests.py is currently at ~5500 lines of code, it's time to break it down into smaller files. I started with an easy middleware set of tests (i.e., versioned writes, ~600 lines of code ) so I can get some feedback. There are more complicated tests that cover multiple middlewares for example, it is not so clear where those should go. Change-Id: I2aa6c18ee5b68d0aae73cc6add8cac6fbf7f33da Signed-off-by: Thiago da Silva commit 5d7a3a4172f0f11ab870252eec784cf24b247dea Author: Ondřej Nový Date: Sat Nov 19 23:24:30 2016 +0100 Removed "in-process-" from func env tox name This shorten shebang in infra, because we are hitting 128 bytes limit. Change-Id: I02477d81b836df71780942189d37d616944c4dce commit 9ea340256996a03c8c744201297b47a0e91fe65b Author: Kota Tsuyuzaki Date: Fri Nov 18 01:50:11 2016 -0800 Don't overwrite built-in 'id' This is a follow up for https://review.openstack.org/#/c/399237 'id' is assigned as a builtin function so that we should not use 'id' for the local variable name. Change-Id: Ic27460d49e68f6cd50bda1d5b3810e01ccb07a37 commit bf473ddc9d3fcb3a3fe516b5221a588c5b196a1d Author: John Dickinson Date: Wed Nov 9 12:37:43 2016 -0800 2.11.0 release notes I have reordered the Contributors section of the AUTHORS file to simply be ordered by the first character in the author name. Change-Id: I824022e582cadc361484b20c411840f46c4c5b50 commit 2e7a7347fc58676fbaabce3d87a15866796d32e4 Author: Tim Burke Date: Thu Nov 17 13:02:06 2016 -0800 Avoid infinite loop while placing parts Previously, we could over-assign how many parts should be in a tier. This would cause the local `parts` variable to go negative, which meant that our `while parts` loop would never terminate. Change-Id: Id7e7889742ca37cf1a9c0d55fba78d967e90e8d0 Closes-Bug: 1642538 commit b94d0db9aaf117080bdcebda4b44bd02eb25749e Author: Bryan Keller Date: Wed Nov 16 22:16:53 2016 +0000 Correctly send 412 Precondition Failed in copy middleware Previously in copy middleware, if a user entered an invalid destination path with an invalid `container/object` path the server would return a 500 Internal Server Error. However, the correct response should be a 412 Precondition Failed. This patch updates copy so that it catches the 412 Precondition Failed exception and returns it to the client. Closes-Bug: #1641980 Change-Id: Ic4677ae033d05b8730c6ad1041bd9c07268e11a9 commit ba1f285fe6fc8729d7961a34023f2b29c00bac39 Author: zhangyanxian Date: Thu Nov 17 02:06:25 2016 +0000 Fix typos in test_container.py TrivialFix Change-Id: I6d3422a88b6fd6d2a495d17ff75d292ac96cc251 commit e8a5448b0767026925b5ad396f6de1be79ac2325 Author: Tim Burke Date: Tue Nov 15 14:35:42 2016 -0800 Add X-Openstack-Request-Id to Access-Control-Expose-Headers Change-Id: Ib95a693042f0b3cf204033eb5957660cb3573dcf Related-Change: I56cd4738808b99c0a08463f83c100be51a62db05 commit f850ff065e70fb56f9b9e224194553770787f516 Author: Tim Burke Date: Sat Oct 29 22:00:01 2016 +0200 SLO: Concurrently HEAD segments Before creating a static large object, we must verify that all of the referenced segments exist. Previously, this was done sequentially; due to latency between proxy and object nodes, clients must be careful to either keep their segment count low or use very long (minute+) timeouts. We mitigate this somewhat by enforcing a hard limit on segment count, but even then, HEADing a thousand segments (the default limit) with an average latency of (say) 100ms will require more than a minute and a half. Further, the nested-SLO approach requires multiple requests from the client -- as a result, Swift3 is in the position of enforcing a lower limit than S3's 10,000 (which will break some clients) or requiring that clients have timeouts on the order of 15-20 minutes (!). Now, we'll perform the segment HEADs in parallel, with a concurrency factor set by the operator. This is very similar to (and builds upon) the parallel-bulk-delete work. By default, two HEAD requests will be allowed at a time. As a side-effect, we'll also only ever HEAD a path once per manifest. Previously, if a manifest alternated between two paths repeatedly (for instance, because the user wanted to splice together various ranges from two sub-SLOs), then each entry in the manifest would trigger a fresh HEAD request. Upgrade Consideration ===================== If operators would like to preserve the prior (single-threaded) SLO creation behavior, they must add the following line to their [filter:slo] proxy config section: concurrency = 1 This may be done prior to upgrading Swift. UpgradeImpact Closes-Bug: #1637133 Related-Change: I128374d74a4cef7a479b221fd15eec785cc4694a Change-Id: I567949567ecdbd94fa06d1dd5d3cdab0d97207b6 commit 40152f4242dfe46a4c63ae69b0a0e1c83172f556 Author: Tim Burke Date: Tue Nov 15 14:12:02 2016 -0800 Fix up X-Trans-Id-Extra docs Previously, they would render as "X -Trans-Id-Extra" Change-Id: Ibaf9b365467a2c91a0f03baed4d7bc36b0fa9967 commit 8e122c6fe8aa0f3e3b96d124023728b98820147e Author: Janie Richling Date: Tue Nov 15 13:15:59 2016 -0600 Document X-Openstack-Request-Id in all responses Swift returns `X-Openstack-Request-Id` for container and object requests as well as account. A couple api-ref docs are missing this value in the response examples. Change-Id: Ifcd67a620e04be5e92b43c7749ee4acb50bb171d commit d25216bddabc77fb54c514269ee55800c87f558c Author: Thiago da Silva Date: Mon May 16 15:57:47 2016 -0400 remove double checks on account/container info Continuing the clean up in account and container info, removed duplicated validation from account_info and container_info methods, since the same validations were recently added to get_account_info and get_container_info. Change-Id: I1ad745fe809367d22649c83f38c4de7a74cac44e Signed-off-by: Thiago da Silva commit 721cd1634a82b735d0d65062c15c30fffb204a0e Author: Alistair Coles Date: Fri Nov 11 16:34:15 2016 +0000 Show file parameters as mandatory in swift-*-info man pages Changes the file parameters on command line to use the convention of for a mandatory option rather than [file] which suggests an optional flag. Also drop to lower case as is convention in other man pages. Change-Id: Icfb4b4786343eb493a73b092cad1fdadc935d066 commit 243dd97c5848c5ef634d630fcc945bd12aff102f Author: Mohit Motiani Date: Tue Oct 25 21:13:55 2016 +0000 Add OPTIONS in manpages swift-account-info, swift-container-info, swift-object-info and swift-get-node do not contain any information about the available options. This patch adds OPTIONS in these manpages. Change-Id: I832c621460ee6bf19adac8e0aadeab32be4b8da0 commit b69caeae7d717cd0b5c1d643d7f7fc12e09530a5 Author: Christian Hugo Date: Sun Oct 30 12:07:29 2016 +0100 Include debug message for rsync tempfiles The warning message for rsync tempfiles was removed in the related change. However, because our regex match might result in a false positive maybe it's still useful to log a debug message. Instead of silently ignoring rsync tempfiles, when running in debug we note the file and how we classified it - but still no warning will occur. I also consolidate our use of the regex for rsync tempfiles into the diskfile module and move the negative test for the warning logger message a little next to the positive test. Change-Id: Idb2a1a76aa275c9c2e9bad8aceea913b8f5b1c71 Related-Change: I5a5d6e24710e4880776b32edcbc07021acf77676 commit e6dea4d69995272724c772f33385530f5a35bd89 Author: John Dickinson Date: Mon Oct 3 17:18:22 2016 -0700 adding reno sphinx tree This should give the necessary tooling to set up the right links in the right places so Swift has release notes listed alongside other projects. Change-Id: I4e5f1ce1fcfbb2943036c821a24a0b4a3a2d9fc8 commit 5de9747e1629565653a3f15d28bc4b4cb03c4abd Author: Tim Burke Date: Tue Nov 8 12:32:18 2016 -0800 Stop shadowing built-in iter Change-Id: I0e3fcddfef6968353e4c0aa6871a2e7a22598459 commit 2f31c0469dd48c3a7052faeb8b5d786d5a32c21a Author: Lokesh S Date: Thu Jul 28 13:31:47 2016 +0000 Py3: Fixes header key dict Decode header to latin1 on python3 encode header to utf-8 on python2. Co-Authored-By: Alistair Coles Change-Id: I10f205a05bb3a566e52a597d9315b3a8b8c14664 commit f4adb2f28f5517772174aac18e3fe9ab1d08c913 Author: Clay Gerrard Date: Mon Nov 7 18:06:44 2016 -0800 Fix ZeroDivisionError in reconstructor.stats_line Despite a check to prevent zero values in the denominator python integer division could result in ZeroDivisionError in the compute_eta helper function. Make sure we always have a non-zero value even if it is small. NotImplemented: * stats calculation is still not great, see lp bug #1488608 Closes-Bug: #1549110 Change-Id: I54f2081c92c2a0b8f02c31e82f44f4250043d837 commit c2ce92acd6fa25706e271dc2b1504a8db54bd2ae Author: Clay Gerrard Date: Tue Oct 11 13:23:11 2016 -0700 Fix signal handling for daemons with InternalClient The intentional use of "bare except" handling in catch_errors and some daemons to prevent propagation on unexpected errors that do not inherit from Exception (like eventlet.Timeout) or even BaseException (like old-style classes) has the side effect of spuriously "handling" *expected* errors like when a signal handler raises SystemExit. The signal handler installed in our Daemon is intended to ensure first that the entire process group and any forked processes (like rsync's) receive the SIGTERM signal and also that the process itself terminates. The use of sys.exit was not a concious grandiose plans for graceful shutdown (like the running[0] = False trick that wsgi server parent process do) - the desired behavior for SIGTERM is to stop - hard. This change ensures the original goals and intentions of our signal handler are fulfilled without the undesirable side effect that can cause our daemons to confusingly log an expected message to stop as an unexpected error, and start ignoring additional SIGTERM messages; forcing our kind operators to resort to brutal process murder. Closes-Bug: #1489209 Change-Id: I9d2886611f6db2498cd6a8f81a58f2a611f40905 commit 116462d8a5a45ec876403234cf83b96dfad58ddd Author: Clay Gerrard Date: Fri Sep 16 11:58:46 2016 -0700 Add probetest for response with duplicate frags When an SSYNC connection fails after shipping a fragment, but before cleaning itself up - it can lead to multiple replicas of the same fragment index serviceable in the node_iter at the same time. Or perhaps more likely if a partner nodes win a race to rebuild waiting on a handoff. In this case, the proxy need not fail to service a GET just because of extra information. A similar guard was added to the reconstructor in a related change [1]. This underlying bug is acctually fixed by the related change [2], this probetest is just encoding the failure to help with future maintenance. 1. Related-Change: I91f3d4af52cbc66c9f7ce00726f247b5462e66f9 2. Related-Change: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3 Closes-Bug: 1624176 Change-Id: I06fc381a25613585dd18916d50e7ca2c68d406b6 commit 6574ce31eeda1bdffa2534f21eed7ab0d62bce97 Author: Alistair Coles Date: Mon Sep 26 17:25:09 2016 +0100 EC: reconstruct using non-durable fragments Previously the reconstructor would only reconstruct a missing fragment when a set of ec_ndata other fragments was available, *all* of which were durable. Since change [1] it has been possible to retrieve non-durable fragments from object servers. This patch changes the reconstructor to take advantage of [1] and use non-durable fragments. A new probe test is added to test scenarios with a mix of failed and non-durable nodes. The existing probe tests in test_reconstructor_rebuild.py and test_reconstructor_durable.py were broken. These were intended to simulate cases where combinations of nodes were either failed or had non-durable fragments, but the test scenarios defined were not actually created - every test scenario broke only one node instead of the intent of breaking multiple nodes. The existing tests have been refactored to re-use most of their setup and assertion code, and merged with the new test into a single class in test_reconstructor_rebuild.py. test_reconstructor_durable.py is removed. [1] Related-Change: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3 Change-Id: Ic0cdbc7cee657cea0330c2eb1edabe8eb52c0567 Co-Authored-By: Clay Gerrard Closes-Bug: #1624088 commit a4d77d918d99dfa8a1e7f1b0561769a17ba64932 Author: Alistair Coles Date: Thu Nov 3 15:19:25 2016 +0000 Fix KeyError when auditor finds an expired object Since the related change [1] the auditor checks deleted objects in case their tombstone is ready to be reclaimed. However, the auditor mistakes an expired object for a deleted object because DiskFileExpired is a subclass of DiskFileDeleted. This causes a KeyError to be raised and logged because the expired object has no tombstone info in the ondisk_info data structure. This patch makes the auditor catch and ignore DiskFileExpired exceptions before handling DiskFileDeleted exceptions. [1] Related-Change: I3e99dc702d55a7424c6482969e03cb4afac854a4 Change-Id: I9872b6997d09dcfd8a868c4b91b9379407283b8e Closes-Bug: #1638016 commit 3b889331b30f9a02efc38bc451b2ab625caffdd1 Author: Clay Gerrard Date: Tue Sep 20 23:44:32 2016 -0700 Make NullLogger more logger like eventlet.wsgi.server is adaptive to dealing with the named log argument being either a file like or a log like [1], but message handling is a more direct pass through if it's a log like. Since all we want to do is make eventlet's logging it a noop - it's easy to provide either interface! 1. https://github.com/eventlet/eventlet/blob/4d2cdc/eventlet/wsgi.py#L246 Change-Id: I2d792176c96931eafb3f140e6653ba8b31eda429 commit 300d388825c228ae1f51cdb8abcdf4c76ec24637 Author: Gábor Antal Date: Fri Jul 15 14:11:09 2016 +0200 Use more specific asserts in test/probe tests I changed asserts with more specific assert methods. e.g.: from assertTrue(sth == None) to assertIsNone(*) or assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type) or assertTrue(not sth) to assertFalse(sth). The code gets more readable, and a better description will be shown on fail. Change-Id: I3768faa568e3964e726ecc48ac8cb133cb088284 commit 2a75091c58948fb664016c0e91e72acd313e4610 Author: Alistair Coles Date: Mon Oct 17 20:38:52 2016 +0100 Make ECDiskFileReader check fragment metadata This patch makes the ECDiskFileReader check the validity of EC fragment metadata as it reads chunks from disk and quarantine a diskfile with bad metadata. This in turn means that both the object auditor and a proxy GET request will cause bad EC fragments to be quarantined. This change is motivated by bug 1631144 which may result in corrupt EC fragments being written to disk but appear valid to the object auditor md5 hash and content-length checks. NotImplemented: * perform metadata check when a read starts on any frag_size boundary, not just at zero Related-Bug: #1631144 Closes-Bug: #1633647 Change-Id: Ifa6a7f8aaca94c7d39f4aeb9d4fa3f59c4f6ee13 Co-Authored-By: Clay Gerrard Co-Authored-By: Kota Tsuyuzaki commit e166e19b235e1fba8e5eba5720d1c07fe32c40c1 Author: Kazuhiro MIYAHARA Date: Thu Oct 20 14:01:14 2016 +0900 Remove duplicated 'User-Agent' header from object-updater's requests When Object-Server saves async-update in pickle files, 'User-Agent: object-server %(pid)' (the key is title style) is included in the pickles. However, Object-Updater will add 'user-agent: object-updater %(pid)' (the key is lower case style) to the pickled headers and use it to make http connection. According to RFC 7230, each header field consists of a case-insensitive field name, therefore either of 'User-Agent' and 'user-agent' should not be included in the headers. This patch removes old 'User-Agent' header from object-updater's requests. Change-Id: Ia624558395584457c718b311fe80e1a8406e22ad Closes-Bug:#1635114 commit 33c18c579e12474015bdd4995120e9f47e351e72 Author: Ondřej Nový Date: Sat Oct 29 15:00:13 2016 +0200 Remove executable flag from some test modules Change-Id: I36560c2b54c43d1674b007b8105200869b5f7987 commit 2bd8d050fb5dec140ad23d5db1d281049954255b Author: Christian Hugo Date: Sun Oct 30 12:07:29 2016 +0100 Suppress unexpected file warnings for rsync temp files Do not log unexpected file warning for rsync temp files when the parsing of the timespamp fails. If the file passes a regex test suppress the logger waring, but still return it as an unexpected file from get_ondisk_files. Closes-Bug: 1616504 Change-Id: I5a5d6e24710e4880776b32edcbc07021acf77676 commit e3e457da662e92885620981a445530fbd8370604 Author: Ondřej Nový Date: Sun Nov 1 19:32:19 2015 +0100 Respect server type for --md5 check in swift-recon MD5 sum compare of server ring ignores server_type parameter now. This could be confusing and this should only test specified server_type, same as --replication, --validate-servers, etc. Change-Id: I9095309c24f42aec1330757c88e7b242b2a6a865 commit d0fe922904c8016267e89c97bef16052a2c3bb7e Author: karen chan Date: Mon Oct 17 03:37:35 2016 -0700 Mirror X-Trans-Id to X-Openstack-Request-Id Many other OpenStack services use a `[X-]OpenStack-Request-Id` header to return a unique identifier for the request. Swift will now return `X-Trans-Id` as well as `X-Openstack-Request-Id`. Change-Id: I56cd4738808b99c0a08463f83c100be51a62db05 Closes-Bug: #1572786 commit c91ca5335fb86b3b260d88285fe0248d502276f3 Author: Tim Burke Date: Sat Oct 29 18:29:48 2016 +0200 Ignore Range values like "bytes=--0" This is consistent with what we already do for other semantically-invalid values like "bytes=--1". Previously, we would return a 416 Requested Range Not Satisfiable response like we do for the semantically-valid-but-not-really-meaningful "bytes=-0" Change-Id: I932b42406c9a5ee7eaa6978a655e61022a957415 commit 30f672e7201e214fa625a43dcad867e5936e1cef Author: Tim Burke Date: Fri Oct 28 13:07:27 2016 +0200 Refactor SLO If-Match / HEAD tests Previously, we didn't make any assertions about the backend requests but rather just verified the user-visible behavior. Change-Id: Iddd4b705ee9b724a4a8a88c6fbaff36cca9612cf commit c3bc9f31ca6c99c4d54478ca7846bbfad2f26427 Author: Hanxi Liu Date: Tue Oct 4 14:55:27 2016 +0800 Add links for more detailed overview in overview_architecture Change-Id: Ie1f171d0945e3f1f4daa61bc66786c01e68e9a0a commit 99a13d93861240bfdc48730913bd6e445b645932 Author: Ondřej Nový Date: Wed Oct 19 20:17:00 2016 +0200 Fixed rysnc -> rsync typo Change-Id: I671b4206072c6e22f4ae38033502336ec32e86ad commit 9847796f01571dc7be332707dc935efac66ee5d4 Author: Ondřej Nový Date: Mon Oct 17 19:46:57 2016 +0200 Set owner of drive-audit recon cache to swift user Fixies this problem: * swift-drive-audit needs to be run by root, because only root have "umount" permission * swift-object servers typically runs as user swift * if swift-drive-audit is run by root, /var/cache/swift/drive.recon is owned by root, with 0o600 * recon middleware (inside swift-object-server) can't read this cache file: swift-object: Error reading recon cache file This patch adds "user" option to drive-audit config file. Recon cache is chowned to this user. Change-Id: Ibf20543ee690b7c5a37fabd1540fd5c0c7b638c9 commit a8b80f0727bcf760026bfa76539baec6fe706ce8 Author: Thiago da Silva Date: Tue Oct 18 15:33:02 2016 -0400 update urls to newton Change-Id: I9eb8c004e100b16de906d340b54e154cf21ddaab Signed-off-by: Thiago da Silva commit 5064ebb7441fdf02e2ca895a2ae81102e751d978 Author: Charles Hsu Date: Wed Oct 12 18:02:16 2016 +0800 Make Content-Disposition support `inline; filename=...` format. A client can process a object contents immediately when it has knowledge, if it doesn't, it stores the content to local with the filename that assigned in `Content-Disposition`. According to https://tools.ietf.org/html/rfc2183, `filename` can be use whatever `inline` or `attachment` (disposition-type) in the disposition string. Change-Id: Iba94aedfad06c1dc8bd7be5eb3c73b33fb8d5198 commit c3bc83013f94671c48df18858c566d6cf58a89dc Author: Nandini Tata Date: Thu Oct 13 18:05:05 2016 +0000 SAIO install bash scripts for a novice install Added SAIO bash scripts as an option under developer tools. They are well commented and simple to understand for a novice developer. Change-Id: Id1b2babe420f4d10e564256fac5a54b60659c5e5 Co-Authored-By: Paul Dardeau commit 3218f8b064e462d901466b04a4813e15ec96da85 Author: Alistair Coles Date: Wed Oct 12 20:00:13 2016 +0100 Prevent ssync writing bad fragment data to diskfile Previously, if a reconstructor sync type job failed to provide sufficient bytes from a reconstructed fragment body iterator to match the content-length that the ssync sender had already sent to the ssync receiver, the sender would still proceed to send the next subrequest. The ssync receiver might then write the start of the next subrequest to the partially complete diskfile for the previous subrequest (including writing subrequest headers to that diskfile) until it has received content-length bytes. Since a reconstructor ssync job does not send an ETag header (it cannot because it does not know the ETag of a reconstructed fragment until it has been sent) then the receiving object server does not detect the "bad" data written to the fragment diskfile, and worse, will label it with an ETag that matches the md5 sum of the bad data. The bad fragment file will therefore appear good to the auditor. There is no easy way for the ssync sender to communicate a lack of source data to the receiver other than by disconnecting the session. So this patch adds a check in the ssync sender that the sent byte count is equal to the sent Content-Length header value for each subrequest, and disconnect if a mismatch is detected. The disconnect prevents the receiver finalizing the bad diskfile, but also prevents subsequent fragments in the ssync job being sync'd until the next cycle. Closes-Bug: #1631144 Co-Authored-By: Kota Tsuyuzaki Change-Id: I54068906efdb9cd58fcdc6eae7c2163ea92afb9d commit 77f5b201248c0684d87289540d3f94873f5c5e38 Author: Christian Schwede Date: Fri Oct 7 12:17:08 2016 +0000 Throttle update_auditor_status calls If there are quite a few nearly empty partitions per disk you might see some write load even if your cluster is unused. The auditor will update the status file after every partition, and this might happen multiple times within a second if there is not much data stored yet. This patch throttles updates, and will only write out an updated status if the file was last updated more than a minute ago. Closes-Bug: 1631352 Change-Id: Ib61ec9cd945e6b2d28756f6ca47801674a7e6060 commit f3be45c0598fc6f2afd51e8e993fa97a61cbb454 Author: zheng yin Date: Tue Aug 16 19:17:34 2016 +0800 Simplify container_updater test code This patch simplifies the container_updater unit test code by refactoring the creation of the continer_updater into a helper method. This method by default will use a default configuration. However this configuration can be changed by passing in a dictionary of configuration elements to update, for example: self._get_container_updater({'mount_check': 'true'}) Change-Id: Ide211d559bf9381c520a04cd90d278d9b2beae50 commit 19102c6e7f501241f36e5e40d27ab3ba5dcc6655 Author: Ondřej Nový Date: Thu Aug 18 09:48:45 2016 +0200 Apply bash error handling consistently in all bash scripts Change-Id: I36dcff9cc2593bcfb6aebbe84b59d95c711eb3e4 commit b13b49a27caac17ae55b19f315d5ce31801c9522 Author: Alistair Coles Date: Tue Aug 9 16:09:38 2016 +0100 EC - eliminate .durable files Instead of using a separate .durable file to indicate the durable status of a .data file, rename the .data to include a durable marker in the filename. This saves one inode for every EC fragment archive. An EC policy PUT will, as before, first rename a temp file to: #.data but now, when the object is committed, that file will be renamed: ##d.data with the '#d' suffix marking the data file as durable. Diskfile suffix hashing returns the same result when the new durable-data filename or the legacy durable file is found in an object directory. A fragment archive that has been created on an upgraded object server will therefore appear to be in the same state, as far as the consistency engine is concerned, as the same fragment archive created on an older object server. Since legacy .durable files will still exist in deployed clusters, many of the unit tests scenarios have been duplicated for both new durable-data filenames and legacy durable files. Change-Id: I6f1f62d47be0b0ac7919888c77480a636f11f607 commit f0a8af293e1d30cbed4be89a0c0bbcb7b79db6b2 Author: Nicolas Helgeson Date: Thu Oct 6 11:06:49 2016 -0700 Update pyeclib dependency to 1.3.1 Updated swift requirements.txt to change PyECLib version so that it fixes memory leak issue. Change-Id: Ifc069989b96188fd7abe0d8edc6b30266783ad79 Partial-Bug: #1629410 commit 19426182af0064b8143ee57a29998523ad52273b Author: Nguyen Phuong An Date: Wed Sep 7 14:02:10 2016 +0700 [api-ref] Remove temporary block in conf.py The os-api-ref 1.0.0 is out. We can remove the support to older version now. Reference: https://github.com/openstack/senlin/commit/6d41feb58dd836d426c9b6f025846dbbf7295097 Change-Id: I0b28cd3f6b5c393d63b9ab2b4dcd3b7b3c20f9b4 commit f62df7b80c083862c11cbef678f75cc09bb89e8f Author: Pete Zaitcev Date: Mon Oct 3 21:08:15 2016 -0600 Add a configurable URL base to staticweb This came to light because someone ran Tempest against a standard installation of RDO, which helpfuly terminates SSL for Swift in a pre-configured load-balancer. In such a case, staticweb has no way to know what scheme to use and guesses wrong, causing Tempest to fail. Related upstream bug: https://bugs.launchpad.net/mos/+bug/1537071 Change-Id: Ie15cf2aff4f7e6bcf68b67ae733c77bb9353587a Closes-Bug: 1572011 commit 53aebba90324c0fdcb3ea61275b24db4d5a82110 Author: Luong Anh Tuan Date: Tue Oct 4 10:06:23 2016 +0700 Fix a typo in documentation remove redundant 'this' Change-Id: I8860190d882b255a3d416de685f930d2b2c0ad17 commit 2ec4189e374d1f94009fc61c1b17d00d12f7e99b Author: Tim Burke Date: Mon Sep 19 17:37:47 2016 -0700 Fix when we set state in Spliterator Also clean up a comment and some exception text Change-Id: I1e7755cc0468f9a3ba96a0dd24868f09a10c3df0 Related-Change: I24716e3271cf3370642e3755447e717fd7d9957c commit a3dbc139c398be7f2c7af0f843b85d4d53eb81ae Author: zheng yin Date: Thu Sep 29 20:24:23 2016 +0800 add assert error information There are the same error resp.status_int in different operators of one function, then I can't point out which one returns this status_int. Therefore, I assert error information. Change-Id: I58bcba97a9f92e43e0f8ef26b801b69a6906af41 commit ddd40a0645b00b0300d134bbd22e636529fecb43 Author: zheng yin Date: Wed Aug 10 20:21:34 2016 +0800 Add unit test content It lacks of some checks in unit test case. Therefore, I add them and make it completion. Change-Id: Id0d20158ae753d245a541fca5ac4fafeecc9790a commit 45e4fd5f714326324e13d781673ff01666c1ab97 Author: Luong Anh Tuan Date: Mon Oct 3 13:28:48 2016 +0700 Fix typo: remove redundant 'that' Change-Id: I9ed35d866ffb9faa41556c5c00f3b490d35412ab commit 396baf4139f64c6127a01c4f3f228901a55b1c7d Author: John Dickinson Date: Sat Sep 3 10:18:32 2016 -0700 Update test-requirements.txt The testing requirements should not cause extra packaging work for deployers (since these aren't required to deploy). This patch brings it in line with the latest global requirements, except for python-swiftclient itself (which I kept with no version specifier). By being in line with global requirements, this is a more accurate list with respect to how things are actually tested in the gate. Change-Id: Iddc92d9bb135144d38c3e5aec2e95df5787e9b5b commit 7dcfe5a7ecb3f95f4556324dea0b8519b68d68dc Author: Clay Gerrard Date: Fri Sep 30 12:16:58 2016 -0700 Remove unneeded line from ring.utils.dispersion_report This line is render inert by the condition right above it. The behavior/correctness is already well vetted by ring.test_utils:TestUtils.test_dispersion_report Change-Id: I7d7a193c52d1005cecd9c70953d55e2b9755a64d commit 9b98c89983821f709fbea3c14c264150d8613171 Author: Yushiro FURUKAWA Date: Thu Sep 29 20:48:21 2016 +0900 Revises 'url' to 'URL' and 'json' to 'JSON' Change-Id: I44743fbb9bcbce3a50ed6770264ba0f4b17803d7 commit 1004509cab443a5cbb223ffc8258c45cb97ee5fb Author: Alistair Coles Date: Thu Sep 29 19:22:36 2016 +0100 Fix failing cross-policy-versioning functional test Since [1] the functional tests for cross-policy object versioning fails with an error. These tests are skipped in all CI test jobs because none of them have more than one policy enabled. However, when running tests against a system with multiple policies the tests will not skip and the error is raised. [1] Related-Change: Ifebc1c3ce558b1df9e576a58a4100f2219dfc7e7 Related-Bug: #1629234 Change-Id: I48028928bc996252a31cbf3abfd2b5e67a74dc95 commit 05642d2958477354ce2807a93417524a11234639 Author: zheng yin Date: Fri Sep 30 11:11:20 2016 +0800 fix word spelling mistake Change-Id: Ia7b03e52b8d6a334fc2b67c94912effe0e659941 commit 03b5e76d0942c075a0286db47bc9e0f029b1e231 Author: Kota Tsuyuzaki Date: Mon Sep 26 05:37:08 2016 -0700 Improve functional tests and test client This patch includes a couple of small functional test improvement. A. Change swift_test_client.File.sync_metadata to follow Swift object metadata semantics: swift_test_client.File.sync_metadata is designed to post object user metadata to an object. However, prior to this patch, the swift_test_client.File instance keeps the existing object metadata as its member attribute and if sync_metadata is called, it sends both existing metadata and incomming metadata from caller. It looks to result in the odd state as if Swift keeps the existing metadata when POST object requested. To tell the correct Swift object metadata semantics, when POST object requested, the existing metadata in the stored object should be gone even if no metadata is overwritten. i.e. if POST object with 'X-Object-Meta-Key: Val' to a stored object with 'X-Object-Meta-foo: bar', it will result in an object with 'X-Object-Meta-Key' (note that X-Object-Meta-Foo will be deleted) The prior behavior sometimes make us confused in the reviw [1] so that, this patch fixes it to send only incomming metadata if it's set. B. Check the response status code more strictly for ObjectVersioning case This patch fixes test_versioning_check_acl on both TestObjectVersioning and TestObjectVersioningHistoryMode to assert the response status code explisitly instead of asserting just "ResponseError". (e.g. 403 when trying to delete object from other account) 1: https://review.openstack.org/#/c/360933/1/test/functional/tests.py@4142 Change-Id: Ia3e5b40f17dc0f881b695aa4be39c98b91e2bb06 commit 3ae0b80bbd91e5ec3a64657913edc25b8728efcb Author: Anh Tran Date: Thu Sep 29 10:09:14 2016 +0700 Remove redundant 'the' Change-Id: I9f6a11d714e5e9f3c0c6609465225923495aa99a commit 52473ea39277f6b3d8e7d77564189da6b1aec7c3 Author: Hanxi Liu Date: Wed Sep 28 10:58:30 2016 +0800 Use six.moves.urllib.parse instead of urllib Six urllib parse contains py3's urllib.parse and py2's urllib. Replace urllib with six.moves.urllib.parse to keep compatibility. Change-Id: Ie67987e4ffb981c2ee70360f7fa9b3fe873c2a96 Closes-bug: 1280105 commit ccff3668116bd6107003e4f774ab136bb933eb77 Author: Mahati Chamarthy Date: Mon Jul 25 20:10:44 2016 +0530 Update diskfile docstring for get_ondisk_files The return value for diskfile.BaseDiskFileManager.get_ondisk_files is a bit more rich than the current doc string describes; try to fill in some gaps. Co-Author: Alistair Coles Change-Id: Ief10644825ead124e58f3da5693bbdc1c6dc31ed commit 2f880f04486ea1b1e15e8794707a7b61ae57d965 Author: Nam Nguyen Hoai Date: Wed Sep 28 15:53:08 2016 +0700 Fix typo in docstring This patch replaces type "tuble" to "tuple". Change-Id: I0a54f10df0f48fab9553e6438735e42ae1bb5b3b commit a09e42732af4fec0a046e1ee402559e1a9e55487 Author: gecong1973 Date: Tue Sep 27 17:14:13 2016 +0800 Fix a typo in proxy-server.conf-sample TrivialFix Change-Id: If650e25979a9488c93fe93621c905003946c27e5 commit 7d2f8e59ef801092878c92d752316bc74ab8eba8 Author: Ha Van Tu Date: Tue Sep 27 16:13:05 2016 +0700 Fix typo in docstring This patch replace typo "contrain" by "constrain". Change-Id: If91550c4d55f4d072be59470b7f893d56ebd30ca commit 31c92ac6dbaf843ec7aa3ed8dfa973356096ff5b Author: Ha Van Tu Date: Tue Sep 27 11:49:35 2016 +0700 Update link reference in Swift document The reference link for REST API [1] is out of date. We should change it to the new one [2]. [1]:http://developer.openstack.org/api-ref-objectstorage-v1.html [2]:http://developer.openstack.org/api-ref/object-storage/ Change-Id: Ib3e5a78ec2df0591f862671272f51ace5ac65ca7 commit 0c8c764547c78df0b00354bb06c5fb27d381647f Author: Kota Tsuyuzaki Date: Fri Aug 26 00:15:45 2016 -0700 Add functional tests for new versioned_write mode This patch is follow up for [1] and [2] to add new functional tests for versioned_writes middlware 'history' mode. (i.e. using X-History-Location header to a container). The new test class, TestObjectHistoryModeVersioning, will use obvious setting the mode via new X-History-Location header, since the change [2], the setting X-Versions-Mode header added since [1] for incomming request has been deprecated. Hence, since [2], the syntax for stack mode is back to be same with older Swift than [1] so that the only thing we need now is just adding a test suite for the new X-History-location. It means the API has been changing like: --------------- For stack mode: --------------- Older than [1]: X-Versions-Location [1]~[2]: X-Vesions-Location (and X-Versions-Mode: 'stack' for obvious) Newer than [2]: X-Vesions-Location ----------------- For history mode: ----------------- Older than [1]: (Not supported) [1]~[2]: X-Vesions-Location and X-Versions-Mode: 'history' Newer than [2]: X-History-Location Note that this functional tests work on newer swift than [2]. And then, this patch also sets allow_versioned_writes=True for in-process testing (the container server allow_versions option was already set, so this is just enabling in the middleware too). That means that in-process functional tests (such as run by the tox envs func-in-process-*) because history mode requires the middleware allow_versioned_writes option to be explicity set to True. 1: https://review.openstack.org/#/c/214922/ 2: https://review.openstack.org/#/c/373537/ Co-Authored-By: Alistair Coles Related-Change: I555dc17fefd0aa9ade681aa156da24e018ebe74b Related-Change: Icfd0f481d4e40dd5375c737190aea7ee8dbc3bf9 Change-Id: Ifebc1c3ce558b1df9e576a58a4100f2219dfc7e7 commit 6bf63d85a7c7c5e444a20176a6c51a7f3941c93f Author: John Dickinson Date: Thu Sep 22 17:09:08 2016 -0400 authors/changelog updates for 2.10.0 Added reno releasenotes file. Change-Id: Ia14875b992aabaf452242b74e9358cfd884adbcc commit f07dfa202d98c0e9500b71a0040db32183a8106e Author: zheng yin Date: Fri Sep 23 18:07:28 2016 +0800 Add test_PUT/DELETE_bad_timestamp in obj/test_server It has valid_timestamp() to check timestamps in /POST/PUT/DELETE methods. It has test_POST_bad_timestamps in Post, but it has no test_PUT/DELETE_bad_timestamps int PUT/DELETE. Therefore,I add them Change-Id: Ib531f3b7819c1c8dc69a7b51d990581bf1e24dab commit 3da65be4b3b47728833eb6976a33cd420c887aa8 Author: Alistair Coles Date: Thu Sep 22 16:56:36 2016 +0100 Cleanup tests for auditor invalidating hashes Cleans up some auditor tests added in the Related-Change. Make auditor NOT call invalidate_hash for a reclaimable tombstone in a zero bytes file process, so that invalidate_hash is only called once per reclaimable tombstone per auditor cycle. Previously each execution of 'swift-init object-auditor once' would result in two identical entries being appended to hashes.invalid for each reclaimable tombstone. With this change that unnecessary duplication is removed. Related-Change: I3e99dc702d55a7424c6482969e03cb4afac854a4 Change-Id: I5dfaa8d74c07ca8a494c29159c1a2bed39499613 commit bfaa8e05834d4d3c78069af22387427c12e0e496 Author: Clay Gerrard Date: Wed Sep 21 17:33:30 2016 -0700 Fix ChunkWriteError when running unittests I don't think this is a real bug - just that the mocked iter wasn't closing it subiters like the real iter does. Change-Id: I44c8159f9eea8737bc86b6c7eb59a512e57e86c1 commit 19a684ddedf9259bd389267071f19c1c65b2e836 Author: Luong Anh Tuan Date: Mon Sep 12 15:49:38 2016 +0700 Fix using filter() to meet python2,3 As mentioned in link[1], if we need filter on python3, Raplace filter(lambda obj: test(obj), data) with: [obj for obj in data if test(obj)]. [1] https://wiki.openstack.org/wiki/Python3 Change-Id: Ia1ea2ec89e4beb957a4cb358b0d0cef970f23e0a commit 60a2fe0ba8e59f4abbb5e9c5a9cd093818d5b26c Author: Tim Burke Date: Tue Sep 20 13:54:55 2016 -0700 Use separate headers for versioned_writes' stack and history modes Now, instead of saying X-Versions-Location: X-Versions-Mode: history clients should just say X-History-Location: Since we've never had a release featuring a user-settable X-Versions-Mode header, support may be dropped and that is now ignored. Change-Id: Icfd0f481d4e40dd5375c737190aea7ee8dbc3bf9 commit 81d4673674febdbe0bba6e27a6d24739456eb3c4 Author: Mahati Chamarthy Date: Mon Jul 25 20:10:44 2016 +0530 Delete old tombstones - Call invalidate_hash in auditor for reclaimable tombstones - assert changed auditor behavior with a unit test - driveby test: assert get_hashes behavior with a unit test Co-Authored-By: Pete Zaitcev Co-Authored-By: Kota Tsuyuzaki Closes-Bug: #1301728 Change-Id: I3e99dc702d55a7424c6482969e03cb4afac854a4 commit 18bb99971f1a793dc75b6b3cb393d5503be43575 Author: Alistair Coles Date: Mon Sep 19 16:06:18 2016 +0100 Add more comment to authtoken sample options Prior to the Mitaka release the install guides showed services (including Swift) being in a default Keystone domain which existed by default and has id=default. This domain id is reflected in the proxy-server.conf-sample authtoken options and also shown in man page and auth docs. The Mitaka install guide shows a domain with *name* default being created, and having a random UUID assigned, in which services are created. This has caused confusion (see discussion on linked bug report). This patch does not change the sample options but does add to the comments in order to emphasize that a user may need to alter the options to match their Keystone configuration. Change-Id: I17bfcdbd983402eeb561bb704b8b1f1e27547c7d Partial-Bug: #1604674 commit 4f033a61b58d94bb63c14aa4a92949bf2c2793f0 Author: Clay Gerrard Date: Wed Sep 21 00:28:39 2016 -0700 Fix unstable json.dumps ordering in obj-server test test_GET_HEAD_with_fragment_preferences seemed to fail on ocassion on my machine; the test seemed to be assuming the order of dictionary keys in json output sent in headers; the fix was convert back to a dictionary when making the assertion on the context of the json. Change-Id: I1ea1b734c2a9fb12b8f59262bb1229421803e48e Closes-Bug: #1625956 commit cbfa79a159aa741b2982e7c009241217dde45f92 Author: Alistair Coles Date: Tue Sep 6 16:36:36 2016 +0100 api-ref: fix some header definitions Clarify Content-Type header definition for listings. Distinguish between request and response definitions for X-Account-Meta-Temp-URL-Key* headers. Insert X-Container-Meta-Quota-* headers missing from some request/response definitions. Improve X-Container-Meta-Access-Control-Expose-Headers parameter formatting. Distinguish between request and response definitions for X-Container-Meta-Temp-URL-Key* headers. Co-Authored-By: Christian Schwede Change-Id: I8fc7610395973b520aa9ddd72c94e1eb75f602cd Related-Change: I315b4e550b7d10880fbc00fce9311127ba609c2d commit d40ff7098b3dd13b5ea846b86b9e8461ac1875b1 Author: Tim Burke Date: Fri Sep 16 17:21:00 2016 -0700 Clean up api-ref examples If we're going to talk about replacing an object, we should use the same object name as the previous example. Including a non-zero content-length on PUT but not providing a body will timeout. Not including the '-' in '-H' will make curl complain: Rebuilt URL to: H/ * Could not resolve host: H * Closing connection 1 201 Created responses have content-length of zero, not 116. Change-Id: Ifd878559ee4036e4893221c7968f53021f38e236 commit 9a81b9537f9b1e2d8aff7e237095b15f8cedabf3 Author: Ondřej Nový Date: Sat Sep 17 15:42:52 2016 +0200 Fixed cut and pasted paragraph from commit message in manpage Change-Id: Iab839a917e407ce8119f0c6162751f11136cbf5b commit 5ec9a62f4db57eb6ae86704bb987a83a8b0c6faa Author: Alistair Coles Date: Tue Sep 6 16:24:13 2016 +0100 api-ref: clean up account listing examples Move the account listing sample responses to follow the sample requests, and to precede the request/response parameter definitions to be consistent with other parts of the doc. Related-Change: I315b4e550b7d10880fbc00fce9311127ba609c2d Change-Id: Ia20acacd238db4a423b8cd89af1658222b4c5828 commit 2355771d4bc87d003a7702405c0406363e0aa18b Author: Alistair Coles Date: Thu Sep 8 16:41:26 2016 +0100 api-ref: Move repeated paragraph to an include file Move repeated test re metadata header syntax to an include file and make it be rendered as a note. Also make already included text about metadata header value encoding be a note. Change-Id: I4795836587492954ad24dd5baaa5d668746d6040 commit bb24695d04088c9032188285b890db27893e2e9b Author: npraveen35 Date: Sat Sep 17 20:48:00 2016 +0530 updated output for swift stat Change-Id: Iff30acb88124e6116803bb79d32d1fc8cbb2d0ce Closes-Bug: #1622138 commit d0144962f64bbe301c2e50425631efa839e1ebd4 Author: Tim Burke Date: Fri Aug 5 16:19:30 2016 -0700 Include object sysmeta in POST responses This is *only* sysmeta, to ensure we neither change the semantics of the response (which could happen if we included allowed_headers) nor reveal previously-written metadata to write-only clients (which could happen if we ever get updateable object metadata and sent back user meta). We could conceivably send back transient sysmeta, though it seems better to wait for a use-case that demands it. Change-Id: I04fe0b36b85e546b5379bed12089ffd1bce01fcf Co-Authored-By: Thiago da Silva commit a741998bff925360e2d6a72b5ab53c38779f170a Author: Tim Burke Date: Fri Sep 16 13:31:35 2016 -0700 Turn on F812 check F812 list comprehension redefines from line ... While the current violations were benign, this sort of code can easily lead to subtle bugs. Seems worth checking, especially given how cheap it is to bring existing code in line with it. Change-Id: Ibdcf9f93b85a1f1411198001df6bdbfa8f92d114 commit 70e3cf15bf76b23c5661da8bf301ec7f054a492c Author: John Dickinson Date: Fri Sep 16 13:12:51 2016 -0700 remove comment saying we ignore H233. we actually check it Change-Id: Ia78fbb32aa6f45aa6578734dc43f3262e717bab8 commit b587d1606c5bd524efc473182f6048580fae33c8 Author: Travis McPeak Date: Fri Sep 2 09:52:17 2016 -0700 Updating Bandit config file This commit updates the Swift Bandit file to the new style introduced in Bandit 1.0. In response to the struggle with getting a Bandit config file working and kept up to date we introduced a simplified version in Bandit 1.0. This commit updates Swift's bandit.yaml to use the new version. Change-Id: Ida5dd08f4ea72a377346f2159caeb2f3741d4980 commit 44a861787a60346aded8be2a54ef4e16accccbb6 Author: Alistair Coles Date: Mon Oct 5 17:40:06 2015 +0100 Enable object server to return non-durable data This patch improves EC GET response handling: - The proxy no longer requires all object servers to have a durable file for the fragment archive that they return in response to a GET. The proxy will now be satisfied if just one object server has a durable file at the same timestamp as fragments from other object servers. This means that the proxy can now successfully GET an object that had missing durable files when it was PUT. - The proxy will now ensure that it has a quorum of *unique* fragment indexes from object servers before considering a GET to be successful. - The proxy is now able to fetch multiple fragment archives having different indexes from the same node. This enables the proxy to successfully GET an object that has some fragments that have landed on the same node, for example after a rebalance. This new behavior is facilitated by an exchange of new headers on a GET request and response between the proxy and object servers. An object server now includes with a GET (or HEAD) response: - X-Backend-Fragments: the value of this describes all fragment archive indexes that the server has for the object by encoding a map of the form: timestamp -> - X-Backend-Durable-Timestamp: the value of this is the internal form of the timestamp of the newest durable file that was found, if any. - X-Backend-Data-Timestamp: the value of this is the internal form of the timestamp of the data file that was used to construct the diskfile. A proxy server now includes with a GET request: - X-Backend-Fragment-Preferences: the value of this describes the proxy's current preference with respect to those fragments that it would have object servers return. It encodes a list of timestamp, and for each timestamp a list of fragment indexes that the proxy does NOT require (because it already has them). The presence of a X-Backend-Fragment-Preferences header (even one with an empty list as its value) will cause the object server to search for the most appropriate fragment to return, disregarding the existence or not of any durable file. The object server assumes that the proxy knows best. Closes-Bug: 1469094 Closes-Bug: 1484598 Change-Id: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3 Co-Authored-By: Paul Luse Co-Authored-By: Clay Gerrard commit acb8971c765fbc64e55f37923e8b16c0126598ac Author: Ha Van Tu Date: Thu Sep 15 14:31:55 2016 +0700 Update link reference This patch update the outdated link reference inside the doc. Change-Id: I9c5aabe869226489f361ff637e9dec731bbc3d7d commit d062af836ce8a302aec1c43837289236289cefa6 Author: Clay Gerrard Date: Wed Sep 14 10:12:38 2016 -0700 DRY out probe.common Specifically to facilitate the reuse of the retry check server function to fill in the creds for the test2 account which is required for probetests after the related change. Change-Id: I9729faa4c8c8d6d65a481bc2ea3f0566d511034c Related-Change: I8d503419b7996721a671ed6b2795224775a7d8c6 commit f679ed0cc8b21e8b71d3aacc34c2a22352c4c850 Author: Alistair Coles Date: Thu Jul 28 18:41:08 2016 +0100 Make container sync copy SLO manifests Currently the container sync daemon fails to copy an SLO manifest, and the error will stall progress of the sync process on that container. There are several reasons why the sync of an SLO manifest may fail: 1. The GET of the manifest from the source container returns an X-Static-Large-Object header that is not allowed to be included with a PUT to the destination container. 2. The format of the manifest object that is read from the source is not in the syntax required for a SLO manifest PUT. 3. Assuming 2 were fixed, the PUT of the manifest includes an ETag header which will not match the md5 of the manifest generated by the receiving proxy's SLO middleware. 4. If the manifest is being synced to a different account and/or cluster, then the SLO segments may not have been synced and so the validation of the PUT manifest will fail. This patch addresses all of these obstacles by enabling the destination container-sync middleware to cause the SLO middleware to be bypassed by setting a swift.slo_override flag in the request environ. This flag is only set for request that have been validated as originating from a container sync peer. This is justifed by noting that a SLO manifest PUT from a container sync peer can be assumed to have valid syntax because it was already been validated when written to the source container. Furthermore, we must allow SLO manifests to be synced without requiring the semantic of their content to be re-validated because we have no way to enforce or check that segments have been synced prior to the manifest, nor to check that the semantic of the manifest is still valid at the source. This does mean that GETs to synced SLO manifests may fail if segments have not been synced. This is however consistent with the expectation for synced DLO manifests and indeed for the source SLO manifest if segments have been deleted since it was written. Co-Authored-By: Oshrit Feder Change-Id: I8d503419b7996721a671ed6b2795224775a7d8c6 Closes-Bug: #1605597 commit 9ce596a5a623348dee7ee5efb6bb2af16d8add97 Author: Donagh McCabe Date: Fri Aug 12 15:52:55 2016 +0100 Corrections for the API specifications in api-ref Fixes a number of technical issues with the api-ref section including: - Added missing headers - The header descriptions were made specific to whether they are used in requests or responses and the verb in question (example: Content-Length in object HEAD is the object size, not the response body length). - Added references to API features such as bulk delete. - Many typographical fixes (e.g., spaces in the middle of header names) - Restored xml and json account/container listing examples. The following areas were not updated and it is proposed to defer them to a subsequent update. This is because I don't have time or their merit is debatable: - ACLs (as used in a Keystone context) are not described. - Account create/delete is not described. - I left List Endpoints as-is. Change-Id: I315b4e550b7d10880fbc00fce9311127ba609c2d commit 2aa5811c7fdf86df302f5ce8b0413bf51e86e7ab Author: Clay Gerrard Date: Tue Sep 13 15:10:04 2016 -0700 fix string formatting in manager Change-Id: I0d5c351735bdb6249dd9ec3e0c8b7ebe709feea6 commit f35829c291277a997c45185cdfe0b701ec725374 Author: Tim Burke Date: Mon Sep 12 15:02:38 2016 -0700 Bring "egregious range request" comment in line with reality With MAX_RANGES pulled out as a constant, developers are likely to just trust the comment rather than go verify the value. Change-Id: I47fbd4790492b272d53ee07fe5bc87170ce2c72b Releated-Change: I4dcedcaae6c3deada06a0223479e611094d57234 commit 1f690df60c0ce7b627c4ebceaecaa5370ff10042 Author: Luong Anh Tuan Date: Mon Sep 12 15:01:29 2016 +0700 Use ConfigParser instead of SafeConfigParser SafeConfigParser supports interpolation on top of ConfigParser in Python 2, and SafeConfigParser is deprecated in Python 3.2 and log warning like " DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead." So we can use ConfigParser if we don't need interpolation. Change-Id: I7e399b3cb90ded909e0d777a4d10c44f1e9299da Closes-Bug: #1618666 commit 3f2a4b2d89d66daa64cd3cf9bf31efd25702a0a9 Author: zheng yin Date: Fri Sep 9 11:13:15 2016 +0800 Add test_pop in the test_header_key_dict There is a pop() function in the header_key_dict, but it has no test_pop in the test_header_key_dict. Therefore, I add it. Change-Id: I97bf9729fae3f044bcc69014549137f45ecbe91b commit f285cafaa7ffa9bdf3b0d93d5dd00038ed38d70b Author: zheng yin Date: Fri Sep 9 16:41:22 2016 +0800 delete TODO in test_manager After test_shutdown,every method has its test case. Therefore, it is not necessary to add more tests, then I remove TODO. Change-Id: I23c1344bb18a83b154f890be0baa12799db045f8 commit 51925e5e3496c725487c30132976c9e96732e17b Author: Nandini Tata Date: Tue Aug 9 20:34:55 2016 +0000 Add man page for swift.conf Added man page for swift.conf Change-Id: I88235ed414dfc34cb46f2fea014f1e721962885a Closes-Bug: 1607027 commit 0900a2c2447eb5a6cc8b105b4331a638959dba5d Author: Timur Alperovich Date: Tue Sep 6 17:26:40 2016 -0700 Fix error handling in proxy/controllers/base.py. There are a few issues going on in the controllers/base.py:_get_response_parts_iter(). One is that the "raise" statement that attempts to re-raise the GeneratorExit, may re-raise ValueError if that was the last exception caught. Secondly, the range may not actually be set in the backend_headers (need to investigate further, as that could actually be faulty tests, since learn_size_from_content_range should always set it). The patch changes the Range construction to throw a ValueError if None is passed in that case. Lastly, the range may be only half-defined, e.g. bytes=0-. In that case, the check of how many bytes are expected vs how many bytes have been sent does not make sense. Change-Id: Ida5adf3d33c736240b2c4bae5510b5289f03dee2 commit ba610e0d1158c6a5e1a116a85c6c3026ee41fc8e Author: Thiago da Silva Date: Wed Sep 7 13:32:06 2016 -0400 add missing partition directory to example Change-Id: I2e806432d7d49e8fa940598cc6be680a362dec8b commit ee11955d3e0f46b47ba1f3aef3452f865ba851be Author: Alistair Coles Date: Wed Sep 7 17:54:36 2016 +0100 Add troubleshooting tips to keystoneauth docs Documents two common mistakes that are made when deploying Swift with Keystone auth. Change-Id: Id81af09d1d21b5c8a3871bf0e20fe6c1448d0ad9 commit e28ff0629d72d7d2abb06f99d6a2a93a0e37d732 Author: Mohit Motiani Date: Tue Aug 2 23:16:43 2016 +0000 Add regions in ring-builder.py swift-ring-builder CLI usage does not specify about regions. This patch adds a description about it. It also adds a brief description of the flags (report parameter). Change-Id: If51a15c0904857abf62636b90ac0c7ed63300871 Closes-Bug: #1583551 commit eea574a78adc54242a97ac3b5ed0b8deeea754bf Author: Lokesh S Date: Thu Jul 28 12:10:12 2016 +0000 Py3: Fixes eventlet ImportError: No module named urllib2 Python3: from eventlet.green.urllib import request Python2: from eventlet.green import urllib2 Change-Id: Ib38865e0b6a8f076b8a54de4fae221d49f315c91 commit dd5afdb6863d8e8e1a1cdfcd2ce5fe06bc45bc95 Author: zheng yin Date: Fri Jul 29 12:12:55 2016 +0800 simple log information When it has the same log information, we can use a function to simple it, then it also improve code readability. Change-Id: I5bb901099dcc832fc0651b34707901849abb0c34 commit 1cfa335a354040bdba22f20aa6ca783428f1b7b1 Author: Mohit Motiani Date: Tue Sep 6 20:02:33 2016 +0000 Fixed Typo in proxy/controller/obj.py Change-Id: I2d7217c618371dce2dfa8ed273dd5e10cbeabe3f commit 1751db201b4204ead8f1b91a8d3a388d54822610 Author: Alistair Coles Date: Thu Jun 2 16:09:18 2016 +0100 Add test for copy using a Range header Since COPY allows a copy of a ranged GET, add some more test coverage for that in both unit and functional tests. Drive-by fix to use better test assertion methods. Co-Authored-By: Thiago da Silva Change-Id: I5cb202386df0862f953f7388107c4d3466e2e46c commit ba912a10b0342c5ed25e67cbd5eedef5bda287ba Author: zheng yin Date: Tue Sep 6 11:55:41 2016 +0800 Add test_valid_api_version in the test_constraints There is a valid_api_version function in the constraints,but it has no tests in the test_constraints. Therefore, I add test_valid_api_version. Change-Id: Ic8dbe9c00f762513ec9a4ffc0b77e4e5d3e5d47b commit 8b3c04a4aa7ccd554fb6460ec2222b0437514f4e Author: Victor Stinner Date: Mon Aug 22 10:44:43 2016 +0200 py3: Port test_splice to Python 3 * add NamedTemporaryFile() wrapper * use byte strings for binary data in tests * tox.ini: run test_splice on Python 3.4 and Python 3.5 Change-Id: I042d739fbf29433733bf0c2154749bc90b210416 commit 0f761852f8aeca0081405323d1dc23ae08f35fc4 Author: Ondřej Nový Date: Sat Aug 13 17:54:24 2016 +0200 Added missing manpages for all cli tools Change-Id: Ief3cf510d426a4eeefc8bb8d12f6fa9c8343f55b Closes-Bug: #1607005 Closes-Bug: #1607007 Closes-Bug: #1607010 Closes-Bug: #1607012 Closes-Bug: #1607014 Closes-Bug: #1607016 Closes-Bug: #1607019 Closes-Bug: #1607021 Closes-Bug: #1607023 Closes-Bug: #1599923 commit 350f10bf3be9ce6c9eec4b3046afc9a05cf08454 Author: Christian Schwede Date: Thu Jul 28 08:32:17 2016 +0000 Deprecate swift-temp-url python-swiftclient includes an improved and tested method to generate tempurls. The command syntax is essentially the same, therefore we can deprecate this one by importing that method. python-swiftclient is not added as a requirement; if the import fails due to a missing swiftclient module it will just raise a deprecation warning. Closes-Bug: #1607523 Closes-Bug: #1607519 Change-Id: Ifa8bf636f20f82db4845b02d1b58699edaa39356 commit 84b264baa8d81ceafa1ed6beb840c6fb80b91cc3 Author: Kazuhiro MIYAHARA Date: Thu Sep 1 14:59:56 2016 +0900 Fix api reference of object GET request with Range parameter In RFC 7233, response body size of range requests with parameter 'bytes=N-M' is (M - N + 1). And response of object GET request with range parameter in current Swift implementation is according to the RFC. However, in current api reference explains that response body size of object GET request with 'Range: bytes=10-15' is five ( != 15 - 10 + 1). This patch fixes the api reference explanation. Change-Id: I8371864f8e5adb42c1e56b7ea26c556ea1252728 commit 1883c1ec23b6dfa7a266ed36f905fbd0501cc87c Author: Tim Burke Date: Tue Apr 26 09:15:29 2016 -0500 Make get_info requests useful with recheck_*_existence == 0 Before, when recheck_account_existence or recheck_container_existence was set to zero, get_info requests for accounts or containers wouldn't populate the env cache for the current request, so it wouldn't return the data *it just got*. Now, we'll still populate the env cache and memcache, as a cache time of zero means "keep it indefinitely". See the memcache docs at https://github.com/memcached/memcached/blob/1.4.25/doc/protocol.txt#L163 Change-Id: Ia648263073bc88e35216cafb76821b7ce02c1d03 Closes-Bug: 1224734 commit a4279112b3f3320f9b305051965a676df9dbce3e Author: Kazuhiro MIYAHARA Date: Thu Sep 1 21:24:52 2016 +0900 Remove meaningless lines in reconciler unit test There are two lines in which lists are sorted in test_reconciler.py However, the values in the lists will be same, then the two lines are meaningless. This patch removes the two lines from test_reconciler.py Change-Id: I7d72a7a4e9a27d87cd78c96f79d1b340b2ad23de commit c9488027cb6db6aa3116fbcb9a9a8f9d294d52dc Author: Christian Schwede Date: Thu Sep 1 11:42:09 2016 +0000 Make versioned_writes docstring more precise Add a note to the docstring that it is required to add a config section to the proxy-server.conf and an entry to the pipeline to support history mode. Closes-Bug: 1619261 Change-Id: I888485ab4ece6f47db081a4d58c1aab24ce72a8a commit 2a58b1efa4b70505bfa01b3fe410a25ea2fa799b Author: Christian Schwede Date: Wed Aug 31 12:49:07 2016 +0000 More Test[Account|Container]Controller test updates Added another test that tries to replicate using an unknown sync method, with and without an existing DB file. Change-Id: I09a86615f012e4341268ea30cbc06099528f896b commit ff52bb1b1c04cdad59f784f8222a4853e7a8304e Author: Kota Tsuyuzaki Date: Wed Aug 31 23:01:05 2016 -0700 Avoid warnings for versioned_writes docs This includes a couple of fixes as follows: - Add one more "-" for mode example titles to adjust the title length (it seems needed to use verbatim expression) - Remove unnecessary back slash Change-Id: If41bba040ea31f2354642b2ec32bc92ece4f0971 commit 2825909d2553527283ac57de7bd9a842988e6624 Author: Timur Alperovich Date: Tue Aug 30 17:02:21 2016 -0700 Silence "Client disconnected" warnings on reads. When a client fully reads the content and closes the iterator, the Client disconnected warning is still generated, as there is no logic to check whether the GeneratorExit exception was raised after the client received all of the data. This can be observed when doing large object reads or using an InternalClient and reading exactly Content-Length bytes from the returned app_iter body. The patch amends the behavior to hoist how many bytes client read from a given part and only raise an exception if there are more parts left or a part was not fully read. Lastly, the GeneratorExit exception is no longer swallowed and is re-raised in the handling code. Change-Id: I879149897fdb25aae977b7f17e580610b188ce04 commit 671254224a4a4710e7556535ee68bd999536ab8d Author: gengchc2 Date: Fri Aug 26 14:08:08 2016 +0800 Correct swift reraising of exception When an exception was caught and rethrown, it should call 'raise' without any arguments because it shows the place where an exception occured initially instead of place where the exception re-raised. Change-Id: I783a2a65f3e38a23ba49b6ae5a2f56a5fdaac2fc commit 0c22bc22573626dae0f81b976b56e82b265f77a2 Author: Tim Burke Date: Wed Aug 31 14:10:27 2016 -0700 Include correct version in install-guide Change-Id: I8f9de7a824aa327d566c821ec05f97d0bc8931e9 commit 3f00148c6b350efc684250f3339fa5166704d5a3 Author: Nguyen Phuong An Date: Wed Aug 31 14:37:11 2016 +0700 Config logABug feature for Swift api-ref Currently, Swift api-ref is not configured with logABug feature. When users click "Report bug" button, it leads to "openstack-manuals" which is default. Change-Id: I3b6fb410000637ff03d99110d440de88a5adc30c commit 66c905e29458666cbd3ea6638a6a4c137e6a35a8 Author: Timur Alperovich Date: Mon Aug 29 17:27:50 2016 -0700 Close the iterators in string_along. Make sure to close the underlying iterator in string_along. What is currently happening when using the InternalClient is that "Client disconnected" warnings are generated and resources are tied up until GC runs. Change-Id: If1f6c0c756aee95f53f99371439533a97d347eab commit cd8eb6db9188359039c1f13e34397cb9b24b8112 Author: zheng yin Date: Fri Aug 26 14:04:46 2016 +0800 Add test to validate the error strings in test_expirer Improve test_get_process_values_* methods in obj/test_expirer in the form assertRaises(ValueError, x.get_process_values,{}/vals), to use the assertRaises context form. This improves understandability by validateing the error strings in addition to the ValueError. Related-Change: I3d12b79470d122b2114f9ee486b15d381f290f95 Change-Id: I1c66b8894cba8328d19cf99491a8ad18ded71078 commit 4d4885acdc8107c4f8f8cd10bc6d19ba3901eb25 Author: Tim Burke Date: Fri Aug 5 18:05:45 2016 +0000 Tighten header checks for object PUT/POST paths Change-Id: If2cd059719fe5af1e73ecde5306e9f68d590831f commit 773edb4a5dd22b22f5fcab57820bcbdd4176dfcd Author: Prashanth Pai Date: Thu Mar 5 18:18:25 2015 +0530 Make object creation more atomic in Linux Linux 3.11 introduced O_TMPFILE as a flag to open() sys call. This would enable users to get a fd to an unnamed temporary file. As it's unnamed, it does not require the caller to devise unique names. It is also not accessible through any path. Hence, file creation is race-free. This file is initially unreachable. It is then populated with data(write), metadata(fsetxattr) and fsync'd before being atomically linked into the filesystem in a fully formed state using linkat() sys call. Only after a successful linkat() will the object file will be available for reference. Caveats * Unlike os.rename(), linkat() cannot overwrite destination path if it already exists. If path exists, we unlink and try again. * XFS support for O_TMPFILE was only added in Linux 3.15. * If client disconnects during object upload, although there is no incomplete/stale file on disk, the object directory would persist and is not cleaned up immediately. Change-Id: I8402439fab3aba5d7af449b5e465f89332f606ec Signed-off-by: Prashanth Pai commit 06ff865d19b522c48e23637d1234fc8f6b9401ab Author: Janie Richling Date: Wed Aug 17 00:54:09 2016 -0500 Allow DLO manifest to be updated when using post-as-copy Currently when using fast-post, the manifest is updated with the given 'x-object-manifest' header on a POST. If no such header is supplied, then the manifest will change to a regular object. This is not currently true when using post-as-copy. This patch changes the DLO POST using post-as-copy behavior to match that of using fast-post. It was also documented that 'x-object-manifest' must be provided on a POST to a manifest file. Change-Id: Ie1143ab1a2c8f8c21e258a36badbff5d947769d4 Closes-bug: 1612991 commit 4bcd3d7f6de53ccd04d42005848060b92e7733ce Author: Samuel Merritt Date: Mon Aug 8 17:57:49 2016 -0700 Support multi-range GETs for static large objects. Bonus consistency: 416 responses now always have a body. Before, if you had "swob.HTTPRequestedRangeNotSatisfiable()", you'd get a body, but if you had "swob.Response(..., conditional_response=True)", then you'd get a length-0 response body. Now you always get a response body. It's just the default

..., but at it's always there. Bonus efficiency: do a little caching of sub-SLO manifests to avoid needless re-fetches. This kicks in when there are multiple references to the same sub-SLO in a given manifest. The caching only holds 20 sub-SLOs so that a malicious user can't build a giant SLO tree and use it to run the proxy out of memory (we're already holding up to 10 manifests in memory at a time since a SLO can include another SLO to a depth of 10; this doesn't make the situation too much worse). Change-Id: I24716e3271cf3370642e3755447e717fd7d9957c commit 1fc46a0e13eff5603ba0c26d98407e3012d9de42 Author: Lokesh S