Reviewed: https://review.openstack.org/205579 Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=8ab46b64365b8eab80680f2562f81e8adb3032a3 Submitter: Jenkins Branch: feature/crypto commit 89f705e8aab144092d40a13fc4ef19ffef5f3eba Author: OpenStack Proposal Bot Date: Thu Jul 23 06:11:27 2015 +0000 Imported Translations from Transifex For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I94cf347564cb33977f33b1e64259bcb39a8cf809 commit a65e9db8752793ec37b594dc9eca5066171825db Author: Christian Schwede Date: Wed Jul 22 10:43:17 2015 +0000 Removing commented out code in test/unit/account/test_backend.py Noticed this while reviewing another change. Looks like the test itself already ensures correct functionality of the reclaim() method in AccountBroker without the commented code, thus removing this stale code. Change-Id: I6a26a7591adef9fd794ca68a4e9c493d1127f93c commit 99d052772a9585e0befdfd292fd03aefde77180a Author: Kota Tsuyuzaki Date: Mon Jul 13 01:12:43 2015 -0700 Fix 499 client disconnected on COPY EC object Currently, a COPY request for an EC object might go to fail as 499 Client disconnected because of the difference between destination request content length and actual transferred bytes. That is because the conditional response status and content length for an EC object range GET is handled at calling the response instance on proxy server. Therefore the calling response instance (resp()) will change the conditional status from 200 (HTTP_OK) to 206 (PartialContent) and will change the content length for the range GET. In EC case, sometimes Swift needs whole stored contents to decode a segment. It will make 200 HTTP OK response from object-server and proxy-server will unfortunately set whole content length to the destination content length and it makes the bug 1467677. This patch introduces a new method "fix_conditional_response" for swift.common.swob.Response that calling _response_iter() and cached the iter in the Response instance. By calling it, Swift can set correct condtional response any time after setting whole content_length to the response instance like EC case. Change-Id: If85826243f955d2f03c6ad395215c73daab509b1 Closes-Bug: #1467677 commit 62ed4f81ef80440550633eaaaa962a4f9383c2d3 Author: Timur Alperovich Date: Tue Jul 14 16:56:44 2015 -0700 Add two functional tests for delimiter. The first test verifies that a delimiter will trim entries beyond the first matching instance of delimiter (after the given matching prefix, if any) and squash duplicates. So, when setting the delimiter to "-", given blobs "test", "test-foo" and "test-bar-baz", we expect only "test" (no matching delim) and "test-" (trim all characters after the first "-", and squash duplicates). The second test verifies that when a prefix is provided, the delimiter will trim entries beyond the first matching instance of the delimiter *after the given prefix*. So "bar, "bazar" which both match the prefix "ba" will be returned as "bar" (no delimiter after the matching prefix) and "baza" (after matching the prefix the remainder after the the *next* matching delimiter "a" is trimmed). Change-Id: I49a2aa8722f83e87b7d211e5c26827e93963d92a commit 2d6c5fe2afde382edb8f04c91fca1911601b61d9 Author: Clay Gerrard Date: Tue Jul 21 09:56:13 2015 -0700 Fix Associated Projects link on Getting Started Change-Id: I50be656bc5868242489f97c96a1feaef171933e0 commit cd7b2db550e41412281da0256b468c90ebcf3e8b Author: janonymous Date: Tue Jul 21 19:23:00 2015 +0530 unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71 commit c3cc98b2c918476f1b54924e0c54e4c746eba4ac Author: Samuel Merritt Date: Mon Jul 20 17:03:20 2015 -0700 Add comment about ResumingGetter.used_source_etag This confused a couple developers and took about ten minutes to unravel in IRC; let's leave a clue for the next person. Change-Id: I356c8c7a44de23f02eaf68d23a39c9eb4c203ff1 commit 06326d74657a8796439988115002298e4a5f4630 Author: janonymous Date: Mon Jul 20 21:41:58 2015 +0530 Metaclass Python 3.x Compatibility use "six.add_metaclass" instead of "__metaclass__" Change-Id: I57390fec40afe3a965be3215db8f79b9cf862a70 commit 317fa1af49d72ecaf7a9a499d8cdf00a086b4c6f Author: Azhagu Selvan SP Date: Sat Jul 18 14:37:16 2015 +0530 Minor change in development saio setup doc Make the test-requirements install command uniform with the rest of the setup commands in the page by using the $HOME, instead of hardcoding the folder path. Change-Id: I8da7a191a301601e27f894f8268238d77b150ede commit faff98efd47fd59e28b1adb3e7ad4bc3dc6dd781 Author: janonymous Date: Thu Jul 9 07:35:47 2015 +0530 Replace python print operator with print function (pep H233, py33) 'print' function is compatible with 2.x and 3.x python versions Link : https://www.python.org/dev/peps/pep-3105/ Python 2.6 has a __future__ import that removes print as language syntax, letting you use the functional form instead Change-Id: I9d9ca5766a2773a7b4071e280f00361d16b85383 commit 8753e452b04b0af6d6b444184b628724e14fbbb1 Author: Victor Stinner Date: Wed May 27 18:01:37 2015 +0200 Replace StringIO with BytesIO for WSGI input wsgi.input is a binary stream (bytes), not a text stream (unicode). * Replace StringIO with BytesIO for WSGI input * Replace StringIO('') with StringIO() and replace WsgiStringIO('') with WsgiStringIO(): an empty string is already the default value Change-Id: I09c9527be2265a6847189aeeb74a17261ddc781a commit 6e70f3fa322da70c2a65fb919d200e52fc5add8a Author: Victor Stinner Date: Wed May 27 17:27:47 2015 +0200 Get StringIO and cStringIO from six.moves * replace "from cStringIO import StringIO" with "from six.moves import cStringIO as StringIO" * replace "from StringIO import StringIO" with "from six import StringIO" * replace "import cStringIO" and "cStringIO.StringIO()" with "from six import moves" and "moves.cStringIO()" * replace "import StringIO" and "StringIO.StringIO()" with "import six" and "six.StringIO()" This patch was generated by the stringio operation of the sixer tool: https://pypi.python.org/pypi/sixer Change-Id: Iacba77fec3045f96773d1090c0bd48613729a561 commit 51f806d3e3d3a1fcbc80d2f7d7ddbe5cc4c024c9 Author: John Dickinson Date: Tue Jul 14 20:49:08 2015 -0700 remove Python 2.6 from the classifier Change-Id: I67233e9c7b69826242546bd6bd98c24b81070579 commit 278adf5c20101a191979ce1e4d6277e5f209149e Author: Hisashi Osanai Date: Tue Jul 14 15:33:45 2015 +0900 Make logic of unit tests responsive to the method names The two methods, test_authorize_succeeds_for_tenant_name_in_roles and test_authorize_succeeds_for_tenant_id_in_roles, have names that don't match what they are testing. tenant_name and tenant_id need to be switched. Change-Id: I7cb0a7d2b2111127fd5d6b55f2da6a3eadf2235d commit 1cc3eff958fdd4fb07c2b74c52df7829d3125466 Author: Victor Stinner Date: Fri Jul 10 13:04:44 2015 +0200 Fixes for mock 1.1 The new release of mock 1.1 is more strict. It helped to find bugs in tests. Closes-Bug: #1473369 Change-Id: Id179513c6010d827cbcbdda7692a920e29213bcb commit ff192cfe5705324497a389aa2f22227d75dc0f8e Author: janonymous Date: Wed Jul 8 18:38:22 2015 +0530 Replace reduce and unichr , these are no longer available in py3 * Replace reduce() with six.moves.reduce() * Replace unichr with six.unichr Change-Id: I2038e47e0a6522dd992fd2a4aeff981cf7750fe0 commit 4beceab4f4be99f14025815cf7ed4510ea77f460 Author: OpenStack Proposal Bot Date: Thu Jul 9 06:14:56 2015 +0000 Imported Translations from Transifex For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I9ff1dde06be45fc7d6c441a1e1c07221f839a9a1 commit 56ee39a7e13417203c5e1816d7a3184a07f85826 Author: Matthew Oliver Date: Thu Jul 9 15:19:32 2015 +1000 Ring builder code clean up follow up patch This is a simple change that cleans up a NIT from Sam's 'stop moving partitions unnecessarily when overload is on' patch. Change-Id: I9d9f1cc23e2bb625d8e158f4d3f64e10973176a1 commit 6cafd0a4c0bb8f311fc59df580b42e801214effd Author: Oshrit Feder Date: Wed Jul 8 15:18:22 2015 +0300 Fix Container Sync example Container-sync realm uses cluster_ as a prefix to specify clusters' names. At use, the prefix should not be included. Fixing the examples and sample conf to make it clearer that only the name of the cluster should be passed. Change-Id: I2e521d86faffb59e1b45d3f039987ee023c5e939 commit 125238612f58481316db68d7087252bb7729f447 Author: Janie Richling Date: Sat Jul 4 17:08:32 2015 -0500 Add CORS unit tests to base In earlier versions of swift when a request was made with an existing origin, but without any CORS settings in the container, it was possible to get an unhandled exception due to a method call on the "None" return of cors.get('allow_origin', ''). Unit tests have been added to assert that this problem cannot go undetected again. Change-Id: Ia74896dabe1cf5a307c551b15a43ab1fd789c213 Fixes: bug 1468782 commit e87a74695e2f55dbb0e01721de9d46e230e36008 Author: John Dickinson Date: Sat Jun 6 13:03:15 2015 -0700 add domain_remap to /info Also added the value of default_reseller_prefix to /info. I did not add the reseller_prefixes value to /info because deployers may not want to expose all of the resellers that are available. Change-Id: I9ca2f002dc395913bb646390d6908dfb8f554df1 commit 12473104f005b76ebe4b6dbe118a5ac37518eb52 Author: Clay Gerrard Date: Tue Jul 7 18:02:53 2015 -0700 Cleanup error messages in ECDiskFileWriter.commit I think the error messages make better sense and it's more obvious what the error handling code was trying to bring to the surface for operators. If you run this test with nosetests --verbose --nocapture test_diskfile.py:TestECDiskFile.test_commit_fsync_dir_raises_DiskFileErrors ... you can review most of the potential log output Change-Id: I40c9d77f44e087ee61d9642e924b0a4039d6ca9a commit bd3717df65f57da9e7f06292ba349442d17fc939 Author: John Dickinson Date: Thu Jul 2 09:54:30 2015 -0700 update AUTHORS file Change-Id: I509ceeb522bd83c407beb40c22c2924a1d92b8ff commit cd20961abd20f01a14554c14d4f61a7f0d8deb88 Author: janonymous Date: Sun Jul 5 10:59:50 2015 +0530 Replace dict.iteritems() with dict.items() in bin directory of swift. The iteritems() of Python 2 dictionaries has been renamed to items() on Python 3. Change-Id: I4bdc064c90bab56cd60f2dca2a5a78426ffbb31c commit 8eca02d3a1db1e0ed68b15ed41bc62e6c89870c7 Author: janonymous Date: Fri Jul 3 00:45:58 2015 +0530 Replace xrange() with six.moves.range() xrange is no longer available in py3. replacing xrange with range from six.moves Change-Id: Ib863c316a0724bd9c4f53c2e5a8d1bcd42c4dc5a commit 10f367224d5f30c3398169ffe3087f225f3343fc Author: janonymous Date: Fri Jul 3 00:05:36 2015 +0530 Fix Python 3 issues * Replace long with six.integer_type six.integer_type :In Python 2, this is long and int, and in Python 3, just int. 'long' is an undefined variable in py3. Change-Id: Ibe4c5d0d81fe883f4fe33be93a68d814a228ad28 commit 2328983b7535d24d08942e62095b4346f2626566 Author: Samuel Merritt Date: Wed Jun 24 17:07:32 2015 -0700 Stop moving partitions unnecessarily when overload is on. When overload was on and in use, the ring builder was unnecessarily moving partitions. It would converge on a good solution and settle down eventually, but it moved more partitions than necessary along the way. There are three partition gatherers used in the ring builder: dev-removed, dispersion, and weight, in that order. The dev-removed gatherer will pick up all partitions on removed devices. The dispersion gatherer picks up replicas of partitions that are suboptimally dispersed. The weight gatherer picks up partitions on devices which are overweight. The weight gatherer was not overload-aware, so it would pick up partitions that did not need to move. Consider a device that would normally have 100 partitions assigned, assume we set overload to 0.1 so that this device will hold up to 110 (10 extra) for the sake of dispersion, and assume the device actually has 104 partitions assigned to it. The correct behavior is to gather nothing from this device because it has fewer than the maximum. Prior to this commit, the weight gatherer would remove 4 partitions from this device; they would subsequently be reassigned by the overload-aware partition placer (_reassign_parts()). In a ring with multiple overloaded devices, the builder would pick up some partitions from each, shuffle them, and then assign them back to those same devices. Obviously, this just created extra replication work for no benefit. Now, the weight gatherer takes overload into account, and will no longer needlessly gather partitions. That's not all, though; this change worsened the behavior of a ring with more overload than necessary. Before, the ring would balance as best it could, using the minimal amount of overload. With the weight-gatherer change, the ring builder will stop gathering partitions once a device reaches its maximum-permissible assignment including overload. For example, imagine a 3-replica, 4-zone ring with overload=0.2 and weights: z1: 100 z2: 60 z3: 60 z4: 60 Since z1 has more than 1/3 of the weight, z2, z3, and z4 must take more than their fair share for the sake of dispersion. Now, turn up the weights some: z1: 100 z2: 80 z3: 80 z4: 80 Overload is no longer needed; this ring can balance. However, z2, z3, and z4 would end up keeping more than their fair share since (a) they already had extra due to earlier conditions, and (b) the weight gatherer won't pick up partitions from them since they're not overburdened once it takes overload into account. To fix this, we compute the minimum overload factor required for optimal dispersion and then use min(user-input-overload, minimum-overload) during rebalance. This way, we don't overload devices more than the user says, but if the user sets overload higher than necessary, we'll still give the best balance possible. Change-Id: If5666ba654ee25da54f9144f3b78840273a49627 commit ccf0758ef1f362312a6197f86455f680ebd990d3 Author: Samuel Merritt Date: Thu Jun 11 15:40:28 2015 -0700 Add ring-builder analyzer. This is a tool to help developers quantify changes to the ring builder. It takes a scenario (JSON file) describing the builder's basic parameters (part_power, replicas, etc.) and a number of "rounds", where each round is a set of operations to perform on the builder. For each round, the operations are applied, and then the builder is rebalanced until it reaches a steady state. The idea is that a developer observes the ring builder behaving suboptimally, writes a scenario to reproduce the behavior, modifies the ring builder to fix it, and references the scenario with the commit so that others can see that things have improved. I decided to write this after writing my fourth or fifth hacky one-off script to reproduce some bad behavior in the ring builder. Change-Id: I114242748368f142304aab90a6d99c1337bced4c commit e860ec138913dc4c450725766a77e76a7101d06e Author: Christian Schwede Date: Fri May 29 10:11:44 2015 +0000 Ignore invalid PIDs in swift-init While reviewing ab9f6340 I recognized that invalid pids in the pid files raise an exceptions (in case it is not a number), and also accept a 0 that is impossible to use as a pid for any Swift process. In the latter case the process was recognized as runnning, even if it did not - and thus restarting didn't worked. This patch ensures a better handling of any non-numeric value and also removes a pid file if it uses 0 as the pid. Change-Id: I09c415c7a75e4b9a0574a08f46f48fff5bb11d02 commit 0a6f2ab870a868e3036dbb212c0e7dd8375a39eb Author: asettle Date: Fri Jun 26 10:57:35 2015 +1000 Correcting minor grammatical errors 1. Removing unnecessary semi colon 2. Removing unnecessary content Change-Id: Ie95403a4d96db2b8465e75495061fc059098c922 commit 4d4db054d35c7e9de9ed28ac24965e8408dd9f82 Author: Minwoo Bae Date: Mon May 18 14:08:25 2015 -0500 After the .durable has been written, fsync the directory. Added try-except statements in _finalize_durable() to fsync the directory after a successful fsync of the .durable file. Added test_commit_fsync_dir_raises_DiskFileErrors() for testing whether certain assertions hold for the change to include fsync_dir(). Some more error details have been included in the logger. Closes-Bug: #1470651 Change-Id: I4791d75ade8542678369ba0811ef39af6e955cc6 commit d4d7c0e1b11af8f20f965f410901a56e1f8ee69e Author: Clay Gerrard Date: Tue Jun 30 17:24:13 2015 -0700 Fix race in test_server.TestObjectController.test_PUT_ssync_multi_frag If the test ran across a one second boundary it would fail because while the timestamp normalization was doing some rounding it was making no attempt to reuse the same timestamp on subsequent requests. Change-Id: Ic560032bcfacd6f0d10cfc0f4f10e5d6c2bc8dd5 commit a88bd6e6b9253b6488a2bf1a354ce29b51f37fe1 Author: ZhiQiang Fan Date: Sun Jun 28 22:04:46 2015 -0700 fix api overview document The content about Object Storage HTTP requests constraints seems to be a table but broken, this patch fixes it into correct rst table format. Change-Id: I1b4c62da3e6d91add3ee0218707c1628c8f04b33 commit c95a0efe7973a660f4abcd13f4ea6211d72c4baa Author: Clay Gerrard Date: Thu Jun 25 01:44:10 2015 -0700 Make ssync_sender a better HTTP client When a server responses with an error - if that error includes a body - the client should read the body. This cleans up some ugly eventlet/wsgi.server log output related to chunked transfer disconnect (invalid literal for int() with base 16). Change-Id: Ibd06ddee9f216fce07fa33c3a7d8306b59eb6d77 Closes-Bug: #1466138 commit 44b76a1b1b83ac9563010f1ddfca5fca76e567bf Author: Minwoo Bae Date: Thu Jun 18 14:21:06 2015 -0500 EC Reconstructor: Do not reconstruct existing fragments. The EC reconstructor needs to verify that the fragment needing to be reconstructed does not reside in the collection of node responses. Otherwise, resources will be spent unnecessarily reconstructing the fragment. Moreover, this could cause a segfault on some backends. This change adds the necessary verification steps to make sure that a fragment will only be rebuilt in the case it is missing from the other fragment archives. Added some tests to provide coverage for these scenarios. Change-Id: I91f3d4af52cbc66c9f7ce00726f247b5462e66f9 Closes-Bug: #1452553 commit d124ce5792d93e1efcdb083211ecef381f7b7173 Author: Clay Gerrard Date: Thu Jun 25 01:35:07 2015 -0700 Fix ValueError in ssync_receiver httplib's putheader method will cast whatever you give it to a string. where we allow the default dict.get default of None to be passed to putheader unmodified ssync_receiver is surpised that the non-empty string isn't able to be converted to an integer. We can avoid surprising the ssync_receiver in this way by sending the empty string as a better default. Change-Id: Ie9df9927ff4d3dd3f334647f883b2937d0d81030 commit edfca861b6fa39972df276fb1f37aa81583a605d Author: Christian Schwede Date: Fri Jun 26 08:54:22 2015 +0000 Increase httplib._MAXHEADERS Python 2.7.9+ and 3.2.6+ limits the number of maximum headers in httplib to 100 [1,2,3]. This setting is too low for Swift. By default the maximum number of allowed headers depends on the number of max allowed metadata settings plus a default value of 32 for regular http headers. If for some reason this is not enough (custom middleware for example) it can be increased with the extra_header_count constraint. [1] https://bugs.python.org/issue16037 [2] https://hg.python.org/cpython/raw-file/15c95b7d81dc/Misc/NEWS [3] https://hg.python.org/cpython/raw-file/v3.2.6/Misc/NEWS Co-Authored-By: Clay Gerrard Co-Authored-By: Matthew Oliver Co-Authored-By: Thomas Herve Change-Id: I388fd697ec88476024b0e9f1ae75ba35ff765282 commit 81ceee056de604d81a921280f453731eadf7be09 Author: Kota Tsuyuzaki Date: Mon Jun 22 23:24:19 2015 -0700 Add one more test for ssync_receiver To prevent 409 conflict on a primary node during ssyncing, ssync-receiver should add x-ssync-backend-frag-index generated from x-ssync-backend-node-index of the SSYNC replication request header. The change is done by previous work[1], but we need more test for that. This patch addes one more assertion if the x-ssync-backend-frag-index is in the ssync subrequest correctly. *BONUS* Fix some weird mock and add some sanities. 1: https://review.openstack.org/#/c/191521/ Change-Id: I27aff713c69017c0bc4c60b4833184e1285595d7 commit c2c1366b0794bcdbbbaef7fedaef2cab19cd8db8 Author: janonymous Date: Wed Jun 24 21:07:16 2015 +0530 Fixed Errors in Swift-Form Post middleware section rst. Inline markup cannot be nested in RST. Fixing markups by using a single option for the whole line and making some minor error corrections. Change-Id: I42bfd7dfe5c93a69436ecc5d154f2e61ca83fa82 commit 514daea6317edc02587f7abfaae8770e1ef9cc4b Author: janonymous Date: Fri Jun 12 23:40:51 2015 +0530 Fixed Formatting Error in Swift -Form Post middleware section. Fixed Formatting error in ``action=”SWIFT_URL``” => action=”``SWIFT_URL``” Change-Id: Id461129ef7f5b6412f94d36920c942a4181c0eb7 Closes-Bug: #1464740 commit e70b66586e13718e7ac0078c2f2cd1135c61a1f1 Author: Victor Stinner Date: Wed Jun 24 09:36:37 2015 +0200 Replace dict.iteritems() with dict.items() The iteritems() of Python 2 dictionaries has been renamed to items() on Python 3. According to a discussion on the openstack-dev mailing list, the overhead of creating a temporary list using dict.items() on Python 2 is very low because most dictionaries are small: http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html Patch generated by the following command: sed -i 's,iteritems,items,g' \ $(find swift -name "*.py") \ $(find test -name "*.py") Change-Id: I6070bb6c684be76e8e77222a7d280ec6edd43496 commit e6165a7879d796efd3992260ef23a7f95ceeab32 Author: paul luse Date: Mon May 25 14:41:42 2015 -0700 Add policy support to dispersion tools Doesn't work for anything other than policy 0. updated to allow user to specify policy name on cmd line (as with object-info) which then makes populate/report work with 3x, 2x, or EC style policies Change-Id: Ib7c298f0f6d666b1ecca25315b88539f45cf9f95 Closes-Bug: 1458688 commit e5c962a28ce99f01c6cd0b68b7434ddcd0e55f62 Author: Victor Stinner Date: Mon May 25 18:28:02 2015 +0200 Replace xrange() with six.moves.range() Patch generated by the xrange operation of the sixer tool: https://pypi.python.org/pypi/sixer Manual changes: * Fix indentation for pep8 checks * Fix TestGreenthreadSafeIterator.test_access_is_serialized of test.unit.common.test_utils: replace range(1, 11) with list(range(1, 11)) * Fix UnsafeXrange docstring, revert change Change-Id: Icb7e26135c5e57b5302b8bfe066b33cafe69fe4d commit 4a5b851207c98a380a32e27a183dd98022be9d16 Author: Christian Schwede Date: Mon Jun 22 14:48:41 2015 +0200 Add note about updatedb to the docs Change-Id: Ia1aa0bb1f93ee487e2f7ddf76a7a08efa8f3ba41 commit ac8a76958510646c369df9ffc2bf22005e43d153 Author: paul luse Date: Sat Jun 13 11:03:56 2015 -0700 EC Ssync: Update parms to include node and frag indices Previously we sent the ssync backend frag index based on the node index. We need to be more specific for ssync to handle both sync and revert cases so now we send the frag index based on the job contents (as determined by the ec recon)) and the node index as a new header based on, well, the node index. The rcvr can now validate the incoming pair to reject (400) when a primary node is being asked to accept fragments that don't belong to it. Additionally, by having the frag index the rcvr can reject (409) an attempt to accept a fragment when its a handoff and already has one that needs to be reverted. Fixes-bug: #1452619 Change-Id: I8287b274bbbd00903c1975fe49375590af697be4 commit 12d8a53fffea6e4bed8ba3d502ce625f5c6710b9 Author: Samuel Merritt Date: Thu Jun 18 12:58:03 2015 -0700 Get better at closing WSGI iterables. PEP 333 (WSGI) says: "If the iterable returned by the application has a close() method, the server or gateway must call that method upon completion of the current request[.]" There's a bunch of places where we weren't doing that; some of them matter more than others. Calling .close() can prevent a connection leak in some cases. In others, it just provides a certain pedantic smugness. Either way, we should do what WSGI requires. Noteworthy goofs include: * If a client is downloading a large object and disconnects halfway through, a proxy -> obj connection may be leaked. In this case, the WSGI iterable is a SegmentedIterable, which lacked a close() method. Thus, when the WSGI server noticed the client disconnect, it had no way of telling the SegmentedIterable about it, and so the underlying iterable for the segment's data didn't get closed. Here, it seems likely (though unproven) that the object server would time out and kill the connection, or that a ChunkWriteTimeout would fire down in the proxy server, so the leaked connection would eventually go away. However, a flurry of client disconnects could leave a big pile of useless connections. * If a conditional request receives a 304 or 412, the underlying app_iter is not closed. This mostly affects conditional requests for large objects. The leaked connections were noticed by this patch's co-author, who made the changes to SegmentedIterable. Those changes helped, but did not completely fix, the issue. The rest of the patch is an attempt to plug the rest of the holes. Co-Authored-By: Romain LE DISEZ Change-Id: I168e147aae7c1728e7e3fdabb7fba6f2d747d937 Closes-Bug: #1466549 commit df134df901a13c2261a8205826ea1aa8d75dc283 Author: Darrell Bishop Date: Thu May 14 22:14:15 2015 -0700 Allow 1+ object-servers-per-disk deployment Enabled by a new > 0 integer config value, "servers_per_port" in the [DEFAULT] config section for object-server and/or replication server configs. The setting's integer value determines how many different object-server workers handle requests for any single unique local port in the ring. In this mode, the parent swift-object-server process continues to run as the original user (i.e. root if low-port binding is required), binds to all ports as defined in the ring, and forks off the specified number of workers per listen socket. The child, per-port servers drop privileges and behave pretty much how object-server workers always have, except that because the ring has unique ports per disk, the object-servers will only be handling requests for a single disk. The parent process detects dead servers and restarts them (with the correct listen socket), starts missing servers when an updated ring file is found with a device on the server with a new port, and kills extraneous servers when their port is found to no longer be in the ring. The ring files are stat'ed at most every "ring_check_interval" seconds, as configured in the object-server config (same default of 15s). Immediately stopping all swift-object-worker processes still works by sending the parent a SIGTERM. Likewise, a SIGHUP to the parent process still causes the parent process to close all listen sockets and exit, allowing existing children to finish serving their existing requests. The drop_privileges helper function now has an optional param to suppress the setsid() call, which otherwise screws up the child workers' process management. The class method RingData.load() can be told to only load the ring metadata (i.e. everything except replica2part2dev_id) with the optional kwarg, header_only=True. This is used to keep the parent and all forked off workers from unnecessarily having full copies of all storage policy rings in memory. A new helper class, swift.common.storage_policy.BindPortsCache, provides a method to return a set of all device ports in all rings for the server on which it is instantiated (identified by its set of IP addresses). The BindPortsCache instance will track mtimes of ring files, so they are not opened more frequently than necessary. This patch includes enhancements to the probe tests and object-replicator/object-reconstructor config plumbing to allow the probe tests to work correctly both in the "normal" config (same IP but unique ports for each SAIO "server") and a server-per-port setup where each SAIO "server" must have a unique IP address and unique port per disk within each "server". The main probe tests only work with 4 servers and 4 disks, but you can see the difference in the rings for the EC probe tests where there are 2 disks per server for a total of 8 disks. Specifically, swift.common.ring.utils.is_local_device() will ignore the ports when the "my_port" argument is None. Then, object-replicator and object-reconstructor both set self.bind_port to None if server_per_port is enabled. Bonus improvement for IPv6 addresses in is_local_device(). This PR for vagrant-swift-all-in-one will aid in testing this patch: https://github.com/swiftstack/vagrant-swift-all-in-one/pull/16/ Also allow SAIO to answer is_local_device() better; common SAIO setups have multiple "servers" all on the same host with different ports for the different "servers" (which happen to match the IPs specified in the rings for the devices on each of those "servers"). However, you can configure the SAIO to have different localhost IP addresses (e.g. 127.0.0.1, 127.0.0.2, etc.) in the ring and in the servers' config files' bind_ip setting. This new whataremyips() implementation combined with a little plumbing allows is_local_device() to accurately answer, even on an SAIO. In the default case (an unspecified bind_ip defaults to '0.0.0.0') as well as an explict "bind to everything" like '0.0.0.0' or '::', whataremyips() behaves as it always has, returning all IP addresses for the server. Also updated probe tests to handle each "server" in the SAIO having a unique IP address. For some (noisy) benchmarks that show servers_per_port=X is at least as good as the same number of "normal" workers: https://gist.github.com/dbishop/c214f89ca708a6b1624a#file-summary-md Benchmarks showing the benefits of I/O isolation with a small number of slow disks: https://gist.github.com/dbishop/fd0ab067babdecfb07ca#file-results-md If you were wondering what the overhead of threads_per_disk looks like: https://gist.github.com/dbishop/1d14755fedc86a161718#file-tabular_results-md DocImpact Change-Id: I2239a4000b41a7e7cc53465ce794af49d44796c6 commit af72881d1dcc46486d8d652b92a0ec9aa9ca2bfa Author: Samuel Merritt Date: Mon Jun 15 13:36:36 2015 -0700 Use just IP, not port, when determining partition placement In the ring builder, we place partitions with maximum possible dispersion across tiers, where a "tier" is region, then zone, then IP/port,then device. Now, instead of IP/port, just use IP. The port wasn't really getting us anything; two different object servers on two different ports on one machine aren't separate failure domains. However, if someone has only a few machines and is using one object server on its own port per disk, then the ring builder would end up with every disk in its own IP/port tier, resulting in bad (with respect to durability) partition placement. For example: assume 1 region, 1 zone, 4 machines, 48 total disks (12 per machine), and one object server (and hence one port) per disk. With the old behavior, partition replicas will all go in the one region, then the one zone, then pick one of 48 IP/port pairs, then pick the one disk therein. This gives the same result as randomly picking 3 disks (without replacement) to store data on; it completely ignores machine boundaries. With the new behavior, the replica placer will pick the one region, then the one zone, then one of 4 IPs, then one of 12 disks therein. This gives the optimal placement with respect to durability. The same applies to Ring.get_more_nodes(). Co-Authored-By: Kota Tsuyuzaki Change-Id: Ibbd740c51296b7e360845b5309d276d7383a3742 commit 09e7477a391e7425af62a6516f43c69bc98b7080 Author: janonymous Date: Mon Jun 15 22:10:45 2015 +0530 Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d commit 5ad369c1cb0f480b164a5b5e7eb194081835c266 Author: Victor Stinner Date: Tue Jun 9 00:22:12 2015 +0200 Add six requirement The six module is needed to add Python 3 support to Swift. Change-Id: Ie637ed3458c7ff56c26834bca73203ed55604d74 commit 037a0c5dbe01a57b741d34988e21f4f763b6204b Author: janonymous Date: Sat Jun 6 17:42:41 2015 +0530 Object are sorted and concatenated by swift in cardinal order. Modified values in example in decimal to be more precise, added a small description on Object are sorted and concatenated by swift in cardinal order as a small phrase would be sufficient. Change-Id: I914ded4e5726e50bb93b05759c3bfb76edda53ab backport: none Closes-Bug: #1383893 commit 11e5c4adf0e02c8fdab0d8584fc943d96e1165cd Author: Koert van der Veer Date: Tue Dec 16 11:15:19 2014 +0100 Allow default reseller prefix in domain_remap middleware Previously, the reseller prefix needed to be provided in the host name even when the domain was unique to that reseller. With the default_reseller_prefix, any domain which matches in this middleware, will will be passed on with a reseller prefix, whether or not it was provided. Change-Id: I5aa5ce78ad1ee2e3660cce4c3e07306f8999f02a Implements: blueprint domainremap-reseller-domains commit c953e8cc9049e94cd01239d5e4a30a56f26a97d3 Author: Minwoo Bae Date: Tue Apr 28 13:18:32 2015 -0500 The hash_cleanup_listdir function should only be called when necessary. Introduced a conditional that will specify to _finalize_put() whether the put() method is being called within the context of ECDiskFileWriter. Included some unit tests that will test the number of calls to hash_cleanup_listdir() during put(), commit(), and delete() operations. Change-Id: I287b2648591fdbf686898dc9047beda80a64da0c commit 93de6c73e587a672663fa0997f439b73b336769c Author: Hisashi Osanai Date: Tue Jun 2 17:00:42 2015 +0900 Add metadata size tests on the border This patch adds a test for a 204 when a single metadata item in a POST causes the backend aggregate constraints check to be on the border. Background: Overall metadata size constraint is enforced in the container and account backends as well as in the proxy controllers. Whereas the proxy controller can check that constraints are not exceeded by a single PUT or POST request, the backend checks that constraints are not exceeded by the aggregate of all PUTs and POSTs. The change [1] added a test for a 400 when a single metadata item in a POST causes the backend aggregate constraints check to go over limit. [1] I1489e29686013cbd3d70283d8756b548aea3c2e1 Change-Id: Iac86ea71240ddde177e625c279c21aef67659d10 commit af734b3fb6fdbc11b8a0b775256e7642eb485b8d Author: Mark Seger Date: Wed Jun 3 08:16:06 2015 -0400 Change usage help and Attention messages to warnings Change-Id: I1396aaffe36e739606f15f7fef37b11bd83f1fc1 commit 4f2ed8bcd0468f3b69d5fded274d8d6b02ac3d10 Author: Samuel Merritt Date: Fri Mar 20 09:56:30 2015 -0700 EC: support multiple ranges for GET requests This commit lets clients receive multipart/byteranges responses (see RFC 7233, Appendix A) for erasure-coded objects. Clients can already do this for replicated objects, so this brings EC closer to feature parity (ha!). GetOrHeadHandler got a base class extracted from it that treats an HTTP response as a sequence of byte-range responses. This way, it can continue to yield whole fragments, not just N-byte pieces of the raw HTTP response, since an N-byte piece of a multipart/byteranges response is pretty much useless. There are a couple of bonus fixes in here, too. For starters, download resuming now works on multipart/byteranges responses. Before, it only worked on 200 responses or 206 responses for a single byte range. Also, BufferedHTTPResponse grew a readline() method. Also, the MIME response for replicated objects got tightened up a little. Before, it had some leading and trailing CRLFs which, while allowed by RFC 7233, provide no benefit. Now, both replicated and EC multipart/byteranges avoid extraneous bytes. This let me re-use the Content-Length calculation in swob instead of having to either hack around it or add extraneous whitespace to match. Change-Id: I16fc65e0ec4e356706d327bdb02a3741e36330a0 commit f1f4bb30cd8ad930ddb7a232b2744b48e35a0480 Author: Christian Schwede Date: Tue Jun 2 19:51:39 2015 +0000 Fix testing issues When functional tests are run in tox and an exception is raised when connecting to Swift (for example: Swift not running, missing python-keystoneclient package used by python-swiftclient) 0 tests are executed, but tox returns a success. An exception is raised during tests, caused by a missing python-keystoneclient in python-swiftclient. Instead of adding python-keystoneclient as a dependency in python-swiftclient the package is added to the test-requirements.txt in Swift itself. Note that adding python-keystoneclient to the test-requirements in python-swiftclient is not sufficient (it's already in there). The exception in setup_package() is catched by the openstack.nose_plugin, thus disabling this plugin for now as well. Also fixing two test errors seen on the gate regarding the tempurl middleware. There was also an update to tox, environment variables were no longer passed with versions >= 2.0 (http://tox.readthedocs.org/en/latest/changelog.html). Swift test environment variables have been added to the passenv to re-enable the former behavior, as well as environment variables required to pass proxy settings. This also led to skipped tempauth tests, and together with the missing python-keystoneclient no tests were executed. Related-Bug: 1461440 Related-Bug: 1455102 Co-Authored-By: Alistair Coles Change-Id: Ideea071017d04912c60ed0bc76532adbb446c31d commit 2759a6f159d4a90e4ad4c8e9e479085d195f8d14 Author: Pradeep Kumar Singh Date: Wed May 13 21:18:02 2015 +0530 Handle Disk IO error Exception in swift account auditor Swift account auditor fails to quarantine corrupt db due to Disk IO error. This patch fixes that by handling Disk IO Error Exception. Closes-Bug:1169189 Change-Id: I031ee2a5775e4a88d4fb00d972d553936147c42e commit 1bef06eec8f5f780914ac701d63f9c498b29119b Author: Michael Barton Date: Sun May 31 23:10:15 2015 +0000 Don't quarantine on read_metadata ENOENT An operation that removes an existing .ts or .meta out from under another concurrent operation at the right point can cause the whole object to be needlessly quarantined. Closes-Bug: #1451520 Change-Id: I37d660199e54411d0610889f9ee230b13747244b commit 8b4af92dac813d707d124b3be37df586260fe74a Author: Alistair Coles Date: Mon Jun 1 17:57:25 2015 +0100 Tighten up overall metadata size tests Overall metadata size constraint is enforced in the container and account backends as well as in the proxy controllers. Whereas the proxy controller can check that constraints are not exceeded by a single PUT or POST request, the backend checks that constraints are not exceeded by the aggregate of all PUTs and POSTs. The current functional tests only exercise the proxy controller checks, since they test for a 400 when sending excessive metadata in a single POST. This patch adds a test for a 400 when a single metadata item in a POST causes the backend aggregate constraints check to go over limit. The extra coverage of the new assertions can be seen by modifying swift/common/db.DatabasBroker.validate_metadata() to always return None immediately - only the new assertions fail when functests are run. Change-Id: I1489e29686013cbd3d70283d8756b548aea3c2e1 commit b4c1d73ad56e0705d1c5dd1fc4f1d89b09fceff1 Author: Christian Schwede Date: Mon Jun 1 06:50:33 2015 +0000 Make swift-recon compatible for servers without storage policies Swift recon introduced a new key for storage policies, and the CLI expected this key in the server response. However, if one updates the CLI but not yet the server an exception will be raised, because there is no default value and no check if the key is included in the response. This change checks if the policies key is included in the response and updates one test to ensure backward compability. Closes-Bug: 1453599 Change-Id: I7c7a90f9933bec2ab45595df9dc600a6cba65666 commit 38787d0fb5102e41b153e4629d2ef374a02965e9 Author: Samuel Merritt Date: Tue May 26 16:19:54 2015 -0700 Remove simplejson from staticweb Since we're dropping Python 2.6 support, we can rely on stdlib's json and get rid of our dependency on simplejson. This lets us get rid of some redundant Unicode encoding. Before, we would take the container-listing response off the wire, JSON-deserialize it (str -> unicode), then pass each of several fields from each entry to get_valid_utf8_str(), which would encode it, (unicode -> str), decode it (str -> unicode), and then encode it again (unicode -> str) for good measure. The net effect was that each object's name would, in the proxy server, go str -> unicode -> str -> unicode -> str. By replacing simplejson with stdlib json, we get a guarantee that each container-listing entry's name, hash, content_type, and last_modified are unicodes, so we can stop worrying about them being valid UTF-8 or not. This takes an encode and decode out of the path, so we just have str -> unicode -> str. While it'd be ideal to avoid this, the first transform (str -> unicode) happens when we decode the container-listing response body (json.loads()), so there's no way out. Change-Id: I00aedf952d691a809c23025b89131ea0f02b6431 commit 736cf54adf3ee85d2f473e5e5374f9833422967c Author: Samuel Merritt Date: Thu May 28 15:30:47 2015 -0700 Remove simplejson from tests Since we're dropping Python 2.6 support, we can rely on stdlib's json and get rid of our dependency on simplejson. This commit just takes simplejson out of the unit and functional tests. They still pass. Change-Id: I96f17df81fa5d265395a938b19213d2638682106 commit 68c30b80b47c281c549d101c43d79e718e91d21d Author: OpenStack Proposal Bot Date: Thu May 28 06:08:12 2015 +0000 Imported Translations from Transifex For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I0c7b2bdb0edcc3bb3fa06903ec8457ca5e2dd688 commit 5374ba3a80a5b895542196502eac4d9300ba53d2 Author: John Dickinson Date: Wed May 27 12:28:04 2015 -0700 drop Python 2.6 testing support Change-Id: I78f21e5794e8ba7a095f03d279247516a241f555 commit 191f2a00bd9121fddc03d8f07f15e5e34790541e Author: Alistair Coles Date: Mon Apr 27 16:39:23 2015 +0100 Remove _ensure_flush() from SSYNC receiver The Receiver._ensure_flush() method in ssync_receiver.py has the following comment: Sends a blank line sufficient to flush buffers. This is to ensure Eventlet versions that don't support eventlet.minimum_write_chunk_size will send any previous data buffered. If https://bitbucket.org/eventlet/eventlet/pull-request/37 ever gets released in an Eventlet version, we should make this yield only for versions older than that. The reference pull request was included with eventlet 0.14 [1] and swift now requires >=0.16.1 so it is safe to remove _ensure_flush() and save > 8k bytes per SSYNC response. [1] https://bitbucket.org/eventlet/eventlet/commits/4bd654205a4217970a57a7c4802fed7ff2c8b770 Change-Id: I367e9a6e92b7ea75fe7e5795cded212657de57ed commit 3aa06f185ac4256a8883c565bdc90b1ffbd519ca Author: Alistair Coles Date: Mon Apr 27 09:17:46 2015 +0100 Make SSYNC receiver return a reponse when initial checks fail The ssync Receiver performs some checks on request parameters in initialize_request() before starting the exchange of missing hashes and updates e.g. the destination device must be available; the policy must be valid. Currently if any of these checks fails then the receiver just closes the connection, so the Sender gets no useful response code and noise is generated in logs by httplib and wsgi Exceptions. This change moves the request parameter checks to the Receiver constructor so that the HTTPExceptions raised are actually sent as responses. (The 'connection close' exception handling still applies once the 'missing_check' and 'updates' handshakes are in progress.) Moving initialize_request() revealed the following lurking bug: * initialize_request() sets req.environ['eventlet.minimum_write_chunk_size'] = 0 * this was previously ineffective because the Response environ had already been copied from Request environ before this value was set, so the Response never used the value :/ * Now that it is effective (a good thing) it causes the empty string yielded by the receiver when there are no missing hashes in missing_checks() to be sent to the sender immediately. This makes the Sender.readline() think there has been an early disconnect and raise an Exception (a bad thing), as revealed by test/unit/obj/test_ssync_sender.py:TestSsync.test_nothing_to_sync The fix for this is to simply make the receiver skip sending the empty string if there are no missing object_hashes. Change-Id: I036a6919fead6e970505dccbb0da7bfbdf8cecc3 commit bb716573ab5c8455348ec013feb894421e0e1f1c Author: Tim Burke Date: Wed May 20 00:39:41 2015 -0700 Allow SLO PUTs to forgo per-segment integrity checks While manifests still require 'etag' and 'size_bytes' fields for each segment (to catch user errors like 'etaf' or 'size_btyes'), an explicit null for either will skip that particular integrity check and instead use whatever value is retrieved when HEADing the segment. So, if a user uploads a manifest like: [{"path": "/con/obj_seg_1", "etag": null, "size_bytes": 1048576}, {"path": "/con/obj_seg_2", "etag": "etag2", "size_bytes": null}, {"path": "/con/obj_seg_3", "etag": null, "size_bytes": null}] then the etag will only be verified for the /con/obj_seg_2 segment, and the segment size will only be verified for the /con/obj_seg_1 segment. However, the manifest that's ultimately stored (and can be retrieved with a ?multipart-manifest=get query-string) will still look like: [{"name": "/con/obj_seg_1", "hash": "etag1", "bytes": 1048576, ...}, {"name": "/con/obj_seg_2", "hash": "etag2", "bytes": 1048576, ...}, {"name": "/con/obj_seg_3", "hash": "etag3", "bytes": 1234, ...}] This allows the middleware to continue performing integrity checks on object GET. Change-Id: I2c4e585221387dd02a8679a50398d6b614407b12 DocImpact commit a1c327022c70907ccc159d6203de26b37c3a4586 Author: Samuel Merritt Date: Tue May 26 16:43:55 2015 -0700 Remove simplejson from swift-recon Since we're dropping Python 2.6 support, we can rely on stdlib's json and get rid of our dependency on simplejson. All swift-recon was doing with json was decoding a JSON response (from the recon middleware) and printing it to the terminal. This still works just fine. Change-Id: I28cf25a7c2856f230d4642c62fb8bf9c4d37e9e5 commit a3559edc2342e2cf92a5188336ab263ffd038554 Author: Clay Gerrard Date: Fri Apr 17 16:30:30 2015 -0700 Exclude local_dev from sync partners on failure If the primary left or right hand partners are down, the next best thing is to validate the rest of the primary nodes. Where the rest should exclude not just the left and right hand partners - but ourself as well. This fixes a accidental noop when partner node is unavailable and another node is missing data. Validation: Add probetests to cover ssync failures for the primary sync_to nodes for sync jobs. Drive-by: Make additional plumbing for the check_mount and check_dir constraints into the remaining daemons. Change-Id: I4d1c047106c242bca85c94b569d98fd59bb255f4 commit 666bf06c26bc9e0d6256d054835386e50e67b8a2 Author: Samuel Merritt Date: Wed May 6 16:29:06 2015 -0700 EC: don't 503 on marginally-successful PUT On EC PUT in an M+K scheme, we require M+1 fragment archives to durably land on disk. If we get that, then we go ahead and ask the object servers to "commit" the object by writing out .durable files. We only require 2 of those. When we got exactly M+1 fragment archives on disk, and then one connection timed out while writing .durable files, we should still be okay (provided M is at least 3). However, we'd take our M > 2 remaining successful responses and pass that off to best_response() with a quorum size of M+1, thus getting a 503 even though everything worked well enough. Now we pass 2 to best_response() to avoid that false negative. There was also a spot where we were getting the quorum size wrong. If we wrote out 3 fragment archives for a 2+1 policy, we were only requiring 2 successful backend PUTs. That's wrong; the right number is 3, which is what the policy's .quorum() method says. There was a spot where the right number wasn't getting plumbed through, but it is now. Change-Id: Ic658a199e952558db329268f4d7b4009f47c6d03 Co-Authored-By: Clay Gerrard Closes-Bug: 1452468 commit e7c8c578d9e5b0aa7e56b02bd9c39baa99d2d6ae Author: Michael MATUR Date: Mon May 25 15:13:01 2015 +0200 fixup!Patch of "parse_content_disposition" method to meet RFC2183 The spec of Content-Disposition does not require a space character after comma: http://www.ietf.org/rfc/rfc2183.txt Change-Id: Iff438dc36ce78c6a79bb66ab3d889a8dae7c0e1f Closes-Bug: #1458497 commit af8d842076ba269fed7f4128d0c7503ab5d1a94a Author: Joanna H. Huang Date: Tue Oct 21 09:24:25 2014 +0000 Replaced setting run_pause with standard interval The deprecated directive `run_pause` should be replaced with the more standard one `interval`. The `run_pause` should be still supported for backward compatibility. This patch updates object replicator to use `interval` and support `run_pause`. It also updates its sample config and documentation. Co-Authored-By: Joanna H. Huang Co-Authored-By: Kamil Rykowski Change-Id: Ie2a3414a96a94efb9273ff53a80b9d90c74fff09 Closes-Bug: #1364735 commit f11d92d566757a54ff1e3800ec0bfac098347a68 Author: Kota Tsuyuzaki Date: Fri May 22 16:58:04 2015 -0700 Add swift-durability-calculator line to docs This commits add a line (link and small doc) for swift-durability-calculator which provides a browser based durability calculation tool to docs as an associated project. Change-Id: I4ea8015f512616dc25072080bef79b8734971ccf commit f864092455ebcd40b3568633c3524cc5c64d3309 Author: Clay Gerrard Date: Wed May 20 17:50:07 2015 -0700 Add Swift Inspector to assoicated projects Change-Id: I5b5448674ea455119a51509ab5e7cd11a764b5a7 commit e54781a2aa29808aff654861d8a0aafd24b6620c Author: Thiago da Silva Date: Tue May 19 20:27:06 2015 +0000 add object post and delete methods to BaseObjectController Adding post and delete methods to BaseObjectController that can be overriden by ObjectCOntroller subclasses. These methods are similar to the PUT and GET methods that were introduced as part of the EC work Change-Id: I197364bc3e2f2287c0afc8948863e3cdeab90383 Signed-off-by: Thiago da Silva commit ab9f63402de6d554528699a02955854ac28264c5 Author: Takashi Kajinami Date: Mon Nov 24 21:44:03 2014 +0900 Add process name checking into swift-init Swift-init uses pid files to detect existing swift processes by pid. However, it mistakes an unrelated process for a swift process and makes a wrong decision, when the unrelated process is running with pid written in swift pid files. This patch adds process name checking into swift-init and enable it to remove invalid pid files in such situation. Change-Id: Ibca026bdfbdacdd92c8763e1eb15d98293c70656 Closes-Bug: #1327106 commit aa4866eb6014d7fbb8531eb5d927d5611ebfd06d Author: Thiago da Silva Date: Wed May 13 20:10:59 2015 +0000 move replication code to ReplicatedObjectController Moving _connect_put_node, send_file, _transfer_data and _store_object methods to ReplicatedObjectController. Each one of these methods are specific to replication policy. EC policy implements their own version of them. Of these four methods, only _store_object and _connect_put_node are required to be implemented by the policy specific Object Controllers. Change-Id: Ifc72461b77dbfdaae9d63417f1286e8b5da3ca4e Signed-off-by: Thiago da Silva commit 025c4c4339e7ace2f5be8cb3a3cddf6c38ceff37 Author: Kota Tsuyuzaki Date: Fri Apr 24 02:15:36 2015 -0700 Remove confusable query string on post as copy Current post as copy routine (i.e. POST object with post_as_copy option turned on) on Object Controller uses "multipart-manifest" query string which is feeded to env['copy_hook'] to decide which data (the manifest or object pointed by the manifest) should be copied. However, the way using the query string will confuse operators looking at logging system (or analyzing the log) because whole POST object requests have 'multipart-manifest=get' like as: POST /v1/AUTH_test/d4c816b24d38489082f5118599a67920/manifest-abcde%3Fmultipart-manifest%3Dget We cannot know whether the query string was added by hand (from user) or not. In addition, the query isn't needed by the backend conversation between proxy-server and object-server. (Just needed by "copy_hook" on the proxy controller!) To remove the confusable query string and to keep the log to be clean, this patch introduces new environment variable "swift.post_as_copy" and changes proxy controller and the copy_hook to use the new env. This item was originally discussed at https://review.openstack.org/#/c/177132/ Co-Authored-By: Alistair Coles Change-Id: I0cd37520eea1825a10ebd27ccdc7e9162647233e commit 98b725fec639c5501c645ce4e4dc9d12c686f91d Author: Alistair Coles Date: Fri May 1 13:02:29 2015 +0100 Cleanup and extend end to end ssync tests Extends the existing end to end ssync tests with a test using replication policy. Also some cleanup and improvements to the test framework e.g. rather than faking the connection between sender and receiver, use a real connection and wrap it to capture traffic for verification. Change-Id: Id71d2eb3fb8fa15c016ef151aacf95f97196a902 commit 518262ab6ecd8faa2b915df118ffc70a30112a18 Author: paul luse Date: Tue May 12 15:21:13 2015 -0700 Remove 1 line of dead code from EC reconstructor Assuming nobody intentionally left this in here for some reason... Change-Id: I4bf43bb3828e062c0342557243076ed62d6790f4 commit 29f4393d88426fd6c34f2cfe43a8c434bfad8d47 Author: Samuel Merritt Date: Fri May 8 15:55:14 2015 -0700 Remove workaround for old eventlet version Swift now requires eventlet >= 0.16.1, so we can get rid of this workaround for a bug in eventlet 0.9.16. Change-Id: I4a1200b9bd9266896a704a840fda0d1b720bc86d commit 90b84d3a699811a99c97ebbe4f71a14d2f76a0e5 Author: Tim Burke Date: Fri May 8 11:45:12 2015 -0700 Properly re-raise exceptions in proxy_logging Previously, this could encounter TypeErrors, presumably because sys.exc_clear() was called somewhere in the block of code between catching the exception and re-raising. Related-Bug: 1181146 Change-Id: Iadeea3f61e70bf83dc0eb063fdb27edd16f3ca32 commit 664a632c01f8c5c80826b223b9ade774bfe2ed9a Author: Christian Schwede Date: Fri May 8 08:41:39 2015 +0200 Update my mailmap entry Change-Id: I5d21a55d0fa4cab6eaa6ff426819aa1dc997de2f commit 0b20a18e5216b991855c461e41e9ef32e17fadb4 Author: Pete Zaitcev Date: Thu May 7 13:16:15 2015 -0600 Spell "rebalance" right in swift-ring-builder.1 See Red Hat bug #1218269. Change-Id: I814eb4b3c0821f5a8df5feea2bda3a964aace536 commit 1faad248f833735585aa8f6135babceb46fbb6f8 Author: Emmanuel Cazenave Date: Tue May 5 12:31:22 2015 +0200 X-Auth-Token should be a bytestring. Change-Id: I2aa941d74883e17e9548b0144a4a2e2db33aba95 Closes-Bug: 1451773 commit 55dd705a863c4500330cbd2b8c2fec46d618dc71 Author: Christian Schwede Date: Wed May 6 19:53:09 2015 +0200 Add missing statsd metrics section for object-reconstructor Change-Id: Id3f98e5f637ff537a387262b40f21c05876fca91 commit c77c79b2c9b9ca7790c29577341cf36c5e9012cf Author: OpenStack Proposal Bot Date: Tue May 5 06:08:09 2015 +0000 Imported Translations from Transifex For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I9b0156c7fc315182d80604bf353586455bbc34d1 commit 94215049fd37f810ddf4e6b8122cce02aea3e6e3 Author: Samuel Merritt Date: Mon May 4 15:08:51 2015 -0700 Bump up a timeout in a test Got a slow crappy VM like I do? You might see this fail occasionally. Bump up the timeout a little to help it out. Change-Id: I8c0e5b99012830ea3525fa55b0811268db3da2a2 commit 03536dbb55c219b94389e635babec2df2e2759fa Author: Prashanth Pai Date: Wed Apr 29 12:11:59 2015 +0530 Fix incorrect passing of file object to fsync() swift.common.utils.fsync() requires a file descriptor as argument but file object handle was being passed. Change-Id: I316b58f6bc37de0945eff551e4e50565653664f5 Signed-off-by: Prashanth Pai commit bfbc94c3cb34eb9ff288fb817dee667cc870d9eb Author: Alistair Coles Date: Tue Apr 28 10:45:50 2015 +0100 Fix intermittent container replicator test failure Intermittent failure of this test could be due to insufficient time elapsing between either the local and remote db's being created or during the debug_timing calls. This patch forces greater timestamp separation and forces debug_timing to always log timings. Also add message to the failign assertion so if this does fail again we get some clue as to why. Closes-Bug: 1369663 Change-Id: I4b69b2e759d586a14abd0931a68dbdf256d57c32 commit dbb9d4b7938f24ee588d260c9b51ca65d5095749 Author: Christian Schwede Date: Mon Apr 27 08:52:18 2015 +0200 Add missing docstring in direct_client Added a missing docstring to the direct_delete_container method. Also checked other docstrings in the same file and fixed a wrong docstring element in direct_delete_object. Added raises: docstring entry to all methods that raise an Exception. Change-Id: If463a0f9ddff3fe2d13f6d97fcfa955e91d0f01f commit 8cdf0fdebe9eb782322fccfc11253dc959cf321d Author: Clay Gerrard Date: Mon Apr 27 13:29:50 2015 -0700 Fix account replication during pre-storage-policy upgrade Old account schemas don't send the storage_policy_index key for container rows during replication, and if the recieving end is already running an upgraded server it is surprised with a KeyError. Normally this would work itself out if the old schema recieved any updates from container layer, or a new container is created, or requires a row sync from another account database - but if the account databases have rows out of sync and there's no activity in the account otherwise, there's nothing to force the old schemas to be upgraded. Rather than force the old schema that already has a complete set of container rows to migrate even in the absense of activity we can just fill in default legacy value for the storage policy index and allow the accounts to get back in sync and migrate the next time a container update occurs. FWIW, I never able to get a cluster upgrade to get stuck in this state without some sort of account failure that forced them to get their rows out of sync (in my cause I just unlinked a pending and then made sure to force all my account datbases to commit pending files before upgrading - leading to an upgraded cluster that absolutly needed account-replication to solve a row mismatch for inactive accounts with old schemas) Closes-Bug #1424108 Change-Id: Iaf4ef834eb24f0e11a52cc22b93a864574fabf83 commit 03380380efb5d5c0b2a3acfaa64b486ee2cb6e64 Author: Clay Gerrard Date: Thu Apr 23 19:39:16 2015 -0700 Simplify ring.builder.RingBuilder.__deepcopy__ Only container classes (lists, sets, dicts, graphs, collections, etc) need to track objects they deepcopy in the memo dict - particularly when they may contain other containers! As they recreate a new container with the same items as themselves, they'll reference the memo for each item they contain before making a deepcopy of it, and place a reference to the copied item into memo after they do. Trying to help out some other container class in this endeavor by attempting to add ourselves to the memo dict in some useful manor on their behalf however; is not helpful. All we need to do to make sure we're being a good __deepcopy__ implementation is make sure we pass on memo to our calls of deepcopy so that other container classes can avoid making additional deepcopy's of our containers if they already have a memorized copy (which would be odd since unique instances of RingBuilders aren't expected to share state, but hey - python doesn't have private attributes so you never know!) Change-Id: Ifac444dffbf79d650b2d858f6282e05d8ea741a0 commit 43ace3c62893364b6e3c130df56438995627598d Author: Samuel Merritt Date: Mon Apr 20 12:17:56 2015 -0700 Make RingBuilders deep-copy-able We used to be able to deep-copy RingBuilder objects, but the addition of debug logging (8d3b3b2) broke that since you can't deep-copy a Python logger. This commit fixes that. Swift doesn't really deep-copy RingBuilders anywhere, but third-party code might. Change-Id: If8bdadd93d9980db3d8a093f32d76ca604de9301 commit 215cd551df8be066edafd2a1e16d0bd143ec214b Author: Samuel Merritt Date: Tue Apr 21 17:38:04 2015 -0700 Bulk upload: treat user xattrs as object metadata Currently, if you PUT a single object, then you can also associate metadata with it by putting it in the request headers, prefixed with "X-Object-Meta". However, if you're bulk-uploading objects, then you have no way to assign any metadata. The tar file format* allows for arbitrary UTF-8 key/value pairs to be associated with each file in an archive (as well as with the archive itself, but we don't care about that here). If a file has extended attributes, then tar will store those as key/value pairs. This commit makes bulk upload read those extended attributes, if present, and convert those to Swift object metadata. Attributes starting with "user.meta" are converted to object metadata, and "user.mime_type"** is converted to Content-Type. For example, if you have a file "setup.py": $ setfattr -n user.mime_type -v "application/python-setup" setup.py $ setfattr -n user.meta.lunch -v "burger and fries" setup.py $ setfattr -n user.meta.dinner -v "baked ziti" setup.py $ setfattr -n user.stuff -v "whee" setup.py This will get translated to headers: Content-Type: application/python-setup X-Object-Meta-Lunch: burger and fries X-Object-Meta-Dinner: baked ziti Swift will handle xattrs stored by both GNU and BSD tar***. Only xattrs user.mime_type and user.meta.* are processed; others are ignored. This brings bulk upload much closer to feature-parity with non-bulk upload. * The POSIX 1003.1-2001 (pax) format, at least. There are a few different, mutually-incompatible tar formats out there, because of course there are. This is the default format on GNU tar 1.27.1 or later. ** http://standards.freedesktop.org/shared-mime-info-spec/latest/ar01s02.html#idm140622087713936 *** Even with pax-format tarballs, different encoders store xattrs slightly differently; for example, GNU tar stores the xattr "user.rubberducky" as pax header "SCHILY.xattr.user.rubberducky", while BSD tar (which uses libarchive) stores it as "LIBARCHIVE.xattr.user.rubberducky". One might wonder if this is some programmer's attempt at job security. Change-Id: I5e3ce87d31054f5239e86d47c45adbde2bb93640 commit f8dee761bd36f857aa1288c27e095907032fad68 Author: Andreas Jaeger Date: Mon Apr 20 11:15:35 2015 +0200 Release Import of Translations from Transifex Manual import of Translations from Transifex. This change also removes all po files that are less than 66 per cent translated since such partially translated files will not help users. This updates also recreates all pot (translation source files) to reflect the state of the repository. This change needs to be done manually since the automatic import does not handle the proposed branches and we need to sync with latest translations. Note: This is part of importing of translations, there are no new translations for this project, thus only the pot file gets updated. Change-Id: I0cbfdae3bd1662da54c58e91a13f49419eba9b2d commit cd7c58e93690fd25f5266754d0593c656dd51e2e Author: Kota Tsuyuzaki Date: Mon Apr 20 00:18:25 2015 -0700 Use reconstruct insetad of decode/encode With bumping PyECLib up to 1.0.7 on global requirements, we can use the "reconstruct" function directly instead of the current hack doing decode/encode on reconstructor. That is because the hack was for treating PyECLib < 1.0.7 (strictly jearsure scheme) reconstruction bug so we don't have to do decode/encode anymore. Closes-Bug: 1446801 Co-Authored-By: Clay Gerrard Change-Id: I69aae495670e3d0bdebe665f73915547a4d56f99 commit 281cb1c210f0292bad190cabaae447145fa5eade Author: Tushar Gohad Date: Wed Apr 15 17:34:48 2015 -0700 Bump PyECLib version from 1.0.3 to 1.0.7 In addition to fixing several bugs, 1.0.7 eliminates the need for a few work-around code in Swift. This code was only to hide issues in the current version, but it also ends up breaking some third-party integration. In order to enable expected functionality and to avoid dealing with deprecation issues right from the beginning, we need to bump the minium PyECLib requirement to 1.0.7. Closes-Bug: 1446727 Change-Id: I03e059e7335656c22be28ffd6157b56e13bdfc1b commit 51e31c5c7147f3ba61437e132b12b491ca718ce4 Author: Clay Gerrard Date: Wed Apr 15 15:31:06 2015 -0700 Don't apply the wrong Etag validation to rebuilt fragments Because of the object-server's interaction with ssync sender's X-Backend-Replication-Headers when a object (or fragment archive) is pushed unmodified to another node it's ETag value is duped into the recieving ends metadata as Etag. This interacts poorly with the reconstructor's RebuildingECDiskFileStream which can not know ahead of time the ETag of the fragment archive being rebuilt. Don't send the Etag from the local source fragment archive being used as the basis for the rebuilt fragent archive's metadata along to ssync. Closes-Bug: 1446800 Change-Id: Ie59ad93a67a7f439c9a84cd9cff31540f97f334a commit 2080f7dbd897d6130542dbf88e37641a41625eb5 Author: Alistair Coles Date: Thu Feb 26 15:16:22 2015 +0000 Fix tempauth acl checks when simplejson has no speedups As documented in linked bug report, tempauth unit tests were seen to fail on a system where simplejson was installed but without the speedups extension. This is because the tempauth account acl validation checks that values are type str, but without the speedups extension the json parser is returning unicode objects. Fix is to have the acl validator tolerate those objects being unicode or str. Also change common/bufferedhttp.py to coerce ring device to type str when constructing a path, in order to avoid a UnicodeDecodeError when httplib sends a message that has non-ascii header values. Change-Id: I01524282cbaa25dc4b6dfa09f3f4723516cdba99 Closes-Bug: 1425776 commit 27f6fba5c3b9e0461d52c1deffe32130e7e9af51 Author: Kota Tsuyuzaki Date: Mon Apr 20 00:18:25 2015 -0700 Use reconstruct insetad of decode/encode With bumping PyECLib up to 1.0.7 on global requirements, we can use the "reconstruct" function directly instead of the current hack doing decode/encode on reconstructor. That is because the hack was for treating PyECLib < 1.0.7 (strictly jearsure scheme) reconstruction bug so we don't have to do decode/encode anymore. Co-Authored-By: Clay Gerrard Change-Id: I69aae495670e3d0bdebe665f73915547a4d56f99 commit 0c391d6daffe1943f6def803db42e08e6d6846d2 Author: Samuel Merritt Date: Mon Apr 20 16:47:10 2015 -0700 SAIO instructions: ensure ~/bin exists before copying into it Change-Id: I16cd211b00b529ccc4b46f6b10497c32b6741896 commit c5c281ba6dd97a301ba80511e3356d6ca536d701 Author: Minwoo B Date: Mon Apr 20 17:03:25 2015 -0500 Included step in development_saio.rst for installing dependencies in requirements.txt. Change-Id: I6ed1704148e5ae1e3164d10080c350d81856f7a9 commit 2203b46e3f14ef68a090aaea284f0a0442bbb86f Author: Tushar Gohad Date: Wed Apr 15 17:34:48 2015 -0700 Bump PyECLib version from 1.0.3 to 1.0.7 In addition to fixing several bugs, 1.0.7 eliminates the need for a few work-around code in Swift. This code was only to hide issues in the current version, but it also ends up breaking some third-party integration. In order to enable expected functionality and to avoid dealing with deprecation issues right from the beginning, we need to bump the minium PyECLib requirement to 1.0.7. Change-Id: I03e059e7335656c22be28ffd6157b56e13bdfc1b commit f6482bdece27144ee083a53d696469528d7940c2 Author: Thierry Carrez Date: Thu Apr 16 22:08:47 2015 +0200 Set default branch to stable/kilo Open stable/kilo branch by setting defaultbranch for git-review. Change-Id: I81bcda30f99173416eaaa3f1d42da32f3ab5b6d2 commit a2a5b6aa6664b61fa31d25f329426ff089372f17 Author: Samuel Merritt Date: Thu Apr 16 11:42:12 2015 -0700 Functional test for SLO PUT overwriting one of its own segments Change-Id: I4855816848f4fdb148d0b82735cf79bc68429617 commit e4d326b5a7dd186d762726faa45733ff2900343d Author: Kazuhiro MIYAHARA Date: Thu Feb 19 17:38:10 2015 +0900 Fix conflict SLO reponse This patch fixes Swift to respond "409 Conflict" when a segment object path of the manifest on PUT SLO is same as requested object path. It is because the request will overwrite the segment and then it will absolutely cause "409 Conflict" on GET SLO. e.g.: request: PUT "http://hostname/v1/AUTH_account/container/segment_object_00?multipart-manifest=put" manifest file: [{"path" : "container/segment_object_00", "etag" : "", "size_bytes" : }, {"path" : "container/segment_object_01", "etag" : "", "size_bytes" : }, {"path" : "container/segment_object_02", "etag" : "", "size_bytes" : }] Change-Id: I4f4f7b9dbeb6a7c355b801c7e0ae560aa19a70b4 Closes-Bug: 1417936 commit 38ae7bb89c95292c171cb89c1d62859cee0cc24f Author: Alistair Coles Date: Wed Apr 15 23:03:21 2015 +0100 Make the reaper use same timestamp for replica deletes The account reaper is using a unique timestamp when deleting replicas of the same resource. This will result in unnecessary replication traffic after reaping. This patch makes the reaper use a single timestamp per resource. Probe test is modified to check that delete times are equal across replicas before replicators run. test_direct_client.py is modified to check that it uses explicit timestamp when passed to direct_delete_[object|container] methods. Drive-by bug fixes in the probe test e.g. it was not sending X-Backend-Storage-Policy-Index when doing a direct GET to check object state, so the 404s being verified could in fact be due to diskfile not existing rather than diskfile being deleted. Closes-Bug: 1442879 Change-Id: I8bab22d66308bb9d3294e1e0def017c784228423 commit 7bdb8266d4b926526acdea22e8c9c634855eeecc Author: Thiago da Silva