Start tracking the container count per policy including reporting
it in account HEAD and supporting installations where the DB
existed before the updated schema.
Migration is triggered by the account audtior; if the database is
un-migrated it will continue to report policy_stats without the per
policy container_count keys.
commit 4d23a0fcf5faa6339a1a58fcbdab8687a6c88feb
Author: Samuel Merritt <email address hidden>
Date: Thu Aug 28 09:39:38 2014 -0800
Reject overly-taxing ranged-GET requests
RFC 7233 says that servers MAY reject egregious range-GET requests
such as requests with hundreds of ranges, requests with non-ascending
ranges, and so on.
Such requests are fairly hard for Swift to process. Consider a Range
header that asks for the first byte of every 10th MiB in a 4 GiB
object, but in some random order. That'll cause a lot of seeks on the
object server, but the corresponding response body is quite small in
comparison to the workload.
This commit makes Swift reject, with a 416 response, any ranged GET
request with more than fifty ranges, more than three overlapping
ranges, or more than eight non-increasing ranges.
This is a necessary prerequisite for supporting multi-range GETs on
large objects. Otherwise, a malicious user could construct a Range
header with hundreds of byte ranges where each individual byterange
requires the proxy to contact a different object server. If seeking
all over a disk is bad, connecting all over the cluster is way worse.
Add "--no-overlap" option to swift-dispersion populate
This change allows the user to use a "--no-overlap" parameter when
running the tool multiple times. It will increase the coverage by
whatever is specified in the dispersion_coverage field of the conf
file in a manner where existing container/objects are left in place
and no partition is populated more than once.
Test for unicode names in AccountBroker.merge_items()
ContainerBroker.merge_items() had a bug in it where non-ASCII Unicode
names would possibly result in duplicate entries in container
databases. AccountBroker.merge_items() doesn't do the same
bulk-operations tricks that ContainerBroker does, so it doesn't
currently have the bug. This commit just adds a test to ensure the bug
doesn't creep in should someone decide to make AccountBroker look more
like ContainerBroker someday.
commit 7d0e5ebe690bf3cf41ccd970281d532a23284e58
Author: Samuel Merritt <email address hidden>
Date: Tue Jun 10 14:15:27 2014 -0700
Zero-copy object-server GET responses with splice()
This commit lets the object server use splice() and tee() to move data
from disk to the network without ever copying it into user space.
Requires Linux. Sorry, FreeBSD folks. You still have the old
mechanism, as does anyone who doesn't want to use splice. This
requires a relatively recent kernel (2.6.38+) to work, which includes
the two most recent Ubuntu LTS releases (Precise and Trusty) as well
as RHEL 7. However, it excludes Lucid and RHEL 6. On those systems,
setting "splice = on" will result in warnings in the logs but no
actual use of splice.
Note that this only applies to GET responses without Range headers. It
can easily be extended to single-range GET requests, but this commit
leaves that for future work. Same goes for PUT requests, or at least
non-chunked ones.
On some real hardware I had laying around (not a VM), this produced a
37% reduction in CPU usage for GETs made directly to the object
server. Measurements were done by looking at /proc/<pid>/stat,
specifically the utime and stime fields (user and kernel CPU jiffies,
respectively).
Note: There is a Python module called "splicetee" available on PyPi,
but it's licensed under the GPL, so it cannot easily be added to
OpenStack's requirements. That's why this patch uses ctypes instead.
Also fixed a long-standing annoyance in FakeLogger:
When replicated container entries get round-tripped through json, they wind up
with unicode objects for names. This causes equality checks to fail against
container entries, and you can wind up with duplicate records. My bad.
When a ring is rebalanced the number of changed partitions is counted.
Before this patch partitions might be rebalanced, but actually no data
is moved - for example, when a partition is assigned to the same device
as before. This results in a wrong number of reassigned partitions that
is shown to the user.
This patch remembers the partition allocation before the rebalance, and
compares it to the new allocation after a rebalance. Only partitions
that are stored on a different device than before are counted.
Over on the EC branch, we need to be able to parse multipart MIME
documents in the object server. The formpost middleware has a
perfectly good MIME parser, but it seems sort of awful to import
things from formpost in swift/obj/server.py, so I pulled it out into
common.utils.
Cleanup and add clarification to the documentation
for using Keystone auth.
Update to refer to auth_token middleware being
distributed as part of the keystomemiddelware project
rather than keystone.
Include capabilities (/info) in the list of reasons
why delay_auth_decision might need to be set in
auth_token middleware config.
Add description of the project_id:user_id format
for container ACLs and emphasize that ids rather than
names should be used since this patch has now merged: https://review.openstack.org/#/c/86430
Reviewed: https:/ /review. openstack. org/124503 /git.openstack. org/cgit/ openstack/ swift/commit/ ?id=bcaa00f25f3 e8bd4123645256c 535d0efe8a6733
Committed: https:/
Submitter: Jenkins
Branch: feature/ec
commit 15fbf9fe7cf33ed 4b56569078400a2 ba070b6bce
Author: paul luse <email address hidden>
Date: Thu Sep 11 06:55:45 2014 -0700
Add container_count to policy_stat table
Start tracking the container count per policy including reporting
it in account HEAD and supporting installations where the DB
existed before the updated schema.
Migration is triggered by the account audtior; if the database is
un-migrated it will continue to report policy_stats without the per
policy container_count keys.
Closes-Bug: #1367514 b3df303609a4ac5 10765a19162
Change-Id: I07331cea177e19
commit d10462e8704e7f5 efe03c4024b541a 3780348615
Author: Darrell Bishop <email address hidden>
Date: Tue Sep 23 09:11:44 2014 -0700
Fix profile tests to clean up its tempdirs.
Change-Id: I363651046cf414 e14f15affd83404 3aabd5427c0
commit b68258a322cb004 151b84584d00b3c 76ee01bc29
Author: Mahati Chamarthy <email address hidden>
Date: Fri Sep 5 01:42:17 2014 +0530
Added instructions to create a label or UUID to the XFS volume and mount using it.
Change-Id: Idcaf16a278d6c3 4770af9b1f17d69 bdd94fb86b7
commit 4d23a0fcf5faa63 39a1a58fcbdab86 87a6c88feb
Author: Samuel Merritt <email address hidden>
Date: Thu Aug 28 09:39:38 2014 -0800
Reject overly-taxing ranged-GET requests
RFC 7233 says that servers MAY reject egregious range-GET requests
such as requests with hundreds of ranges, requests with non-ascending
ranges, and so on.
Such requests are fairly hard for Swift to process. Consider a Range
header that asks for the first byte of every 10th MiB in a 4 GiB
object, but in some random order. That'll cause a lot of seeks on the
object server, but the corresponding response body is quite small in
comparison to the workload.
This commit makes Swift reject, with a 416 response, any ranged GET
request with more than fifty ranges, more than three overlapping
ranges, or more than eight non-increasing ranges.
This is a necessary prerequisite for supporting multi-range GETs on
large objects. Otherwise, a malicious user could construct a Range
header with hundreds of byte ranges where each individual byterange
requires the proxy to contact a different object server. If seeking
all over a disk is bad, connecting all over the cluster is way worse.
DocImpact
Change-Id: I4dcedcaae6c3de ada06a0223479e6 11094d57234
commit 5efdab6055bc996 38e4e1388bef685 b19682025d
Author: OpenStack Proposal Bot <email address hidden>
Date: Mon Sep 22 06:07:56 2014 +0000
Imported Translations from Transifex
Change-Id: Ibd8882766a87c6 d77e786f7635b12 90391e43f10
commit 4faf1702706b289 521329243e5802c f86e54c7f7
Author: Lorcan <email address hidden>
Date: Tue Sep 2 18:12:18 2014 +0100
Add "--no-overlap" option to swift-dispersion populate
This change allows the user to use a "--no-overlap" parameter when
running the tool multiple times. It will increase the coverage by
whatever is specified in the dispersion_coverage field of the conf
file in a manner where existing container/objects are left in place
and no partition is populated more than once.
Related-Bug: #1233045
Change-Id: I139fed2f4c967b a18d073b7ecd1e9 46ed4da1271
commit c1f6569c00951e3 b57d58a0cd32dc2 8a638f5a81
Author: Rafael Rivero <email address hidden>
Date: Thu Sep 18 21:16:35 2014 -0700
Fixes several typos (Swift)
Corrects spelling errors found in comments.
Change-Id: I228a888e3f2565 69ea32ef1613092 dbd63e13c62
commit 1678083d0e48f24 e6a74eada874f01 ac2755cf78
Author: Samuel Merritt <email address hidden>
Date: Thu Sep 18 17:04:30 2014 -0700
Test for unicode names in AccountBroker. merge_items( )
ContainerBr oker.merge_ items() had a bug in it where non-ASCII Unicode merge_items( ) doesn't do the same
names would possibly result in duplicate entries in container
databases. AccountBroker.
bulk-operations tricks that ContainerBroker does, so it doesn't
currently have the bug. This commit just adds a test to ensure the bug
doesn't creep in should someone decide to make AccountBroker look more
like ContainerBroker someday.
Change-Id: Id2ac129828dbdf 55b609d839ce4d9 d42437ee0a3
commit 7d0e5ebe690bf3c f41ccd970281d53 2a23284e58
Author: Samuel Merritt <email address hidden>
Date: Tue Jun 10 14:15:27 2014 -0700
Zero-copy object-server GET responses with splice()
This commit lets the object server use splice() and tee() to move data
from disk to the network without ever copying it into user space.
Requires Linux. Sorry, FreeBSD folks. You still have the old
mechanism, as does anyone who doesn't want to use splice. This
requires a relatively recent kernel (2.6.38+) to work, which includes
the two most recent Ubuntu LTS releases (Precise and Trusty) as well
as RHEL 7. However, it excludes Lucid and RHEL 6. On those systems,
setting "splice = on" will result in warnings in the logs but no
actual use of splice.
Note that this only applies to GET responses without Range headers. It
can easily be extended to single-range GET requests, but this commit
leaves that for future work. Same goes for PUT requests, or at least
non-chunked ones.
On some real hardware I had laying around (not a VM), this produced a
37% reduction in CPU usage for GETs made directly to the object
server. Measurements were done by looking at /proc/<pid>/stat,
specifically the utime and stime fields (user and kernel CPU jiffies,
respectively).
Note: There is a Python module called "splicetee" available on PyPi,
but it's licensed under the GPL, so it cannot easily be added to
OpenStack's requirements. That's why this patch uses ctypes instead.
Also fixed a long-standing annoyance in FakeLogger:
>>> fake_logger. warn('stuff' ) get_lines_ for_level( 'warn')
>>> fake_logger.
[]
>>>
This, of course, is because the correct log level is 'warning'. Now
you get a KeyError if you call get_lines_for_level with a bogus log
level.
Change-Id: Ic6d6b833a5b04c a2019be94b1b90d 941929d21c8
commit eaab4d3fd6b7a33 0c1b904dc69d7ea 4f0fbe8781
Author: Michael Barton <email address hidden>
Date: Thu Sep 18 19:20:51 2014 +0000
container. merge_items bug
When replicated container entries get round-tripped through json, they wind up
with unicode objects for names. This causes equality checks to fail against
container entries, and you can wind up with duplicate records. My bad.
Change-Id: I3aee2ad8dbd3a6 17efe37e887cfb9 02a3e4a1646
commit 09bdc87cbc1e7bc 1918f9b5094bec2 66b6761d75
Author: Christian Schwede <email address hidden>
Date: Thu Sep 11 08:01:51 2014 +0000
Return correct number of changed partitions
When a ring is rebalanced the number of changed partitions is counted.
Before this patch partitions might be rebalanced, but actually no data
is moved - for example, when a partition is assigned to the same device
as before. This results in a wrong number of reassigned partitions that
is shown to the user.
This patch remembers the partition allocation before the rebalance, and
compares it to the new allocation after a rebalance. Only partitions
that are stored on a different device than before are counted.
Partial-Bug: 1367826 1791f9191cef78c ff1b3e2645f
Also-By: Florent Flament <email address hidden>
Change-Id: Iacfd514df3af35
commit a7604da065f0259 31f42bbcf9307bb abda3a37a9
Author: Samuel Merritt <email address hidden>
Date: Wed Sep 10 17:29:05 2014 -0700
Move multipart MIME parser into utils
Over on the EC branch, we need to be able to parse multipart MIME server. py, so I pulled it out into
documents in the object server. The formpost middleware has a
perfectly good MIME parser, but it seems sort of awful to import
things from formpost in swift/obj/
common.utils.
Change-Id: Ieb4c05d02d8e4e f51a3a11d26c503 786b1897f60
commit 49fa5b8fb467bb5 900dda36da47d46 d4c5882bb0
Author: Alistair Coles <email address hidden>
Date: Wed Sep 10 16:09:13 2014 +0100
Update documentation for using keystone auth
Cleanup and add clarification to the documentation
for using Keystone auth.
Update to refer to auth_token middleware being
distributed as part of the keystomemiddelware project
rather than keystone.
Include capabilities (/info) in the list of reasons
why delay_auth_decision might need to be set in
auth_token middleware config.
Add description of the project_id:user_id format /review. openstack. org/#/c/ 86430
for container ACLs and emphasize that ids rather than
names should be used since this patch has now merged:
https:/
DocImpact 74f1d2d163016ca 2d40ec2d589
blueprint keystone-v3-support
Change-Id: Idda4a3dcf82404