OpenStack Object Storage (swift)

Improve error reporting when running on an unsupported filesystem

Bug #966671 reported by Ben Hartshorne on 2012-03-27

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Object Storage (swift)	Fix Released	Low	Matthew Oliver	OpenStack Object Storage (swift) 2.2.1

Bug Description

When you start Swift (specifically the object-server) with an unsupported filesystem (eg one that doesn't support XATTR) it does not complain. You don't get any errors until you try and PUT an object, at which point it dumps a stack trace into syslog (indicating IOError):

Mar 27 22:10:02 i-000001c9 object-server ERROR __call__ error with PUT /vdb/4548/AUTH_.auth/.account_id/AUTH_4c3531a8-0735-4ec4-a88b-fd
eb357cccde : #012Traceback (most recent call last):#012 File "/usr/lib/pymodules/python2.6/swift/obj/server.py", line 715, in __call__
#012 res = getattr(self, req.method)(req)#012 File "/usr/lib/pymodules/python2.6/swift/obj/server.py", line 520, in PUT#012 file
.put(fd, tmppath, metadata)#012 File "/usr/lib/python2.6/contextlib.py", line 34, in __exit__#012 self.gen.throw(type, value, trace
back)#012 File "/usr/lib/pymodules/python2.6/swift/obj/server.py", line 251, in mkstemp#012 yield fd, tmppath#012 File "/usr/lib/p
ymodules/python2.6/swift/obj/server.py", line 520, in PUT#012 file.put(fd, tmppath, metadata)#012 File "/usr/lib/pymodules/python2.
6/swift/obj/server.py", line 275, in put#012 write_metadata(fd, metadata)#012 File "/usr/lib/pymodules/python2.6/swift/obj/server.p
y", line 89, in write_metadata#012 setxattr(fd, '%s%s' % (METADATA_KEY, key or ''), metastr[:254])#012 File "/usr/lib/pymodules/pyt
hon2.6/xattr/__init__.py", line 188, in setxattr#012 return xattr(f).set(attr, value, options=options)#012 File "/usr/lib/pymodules
/python2.6/xattr/__init__.py", line 81, in set#012 self._set(name, value, 0, options | self.options)#012 File "/usr/lib/pymodules/p
ython2.6/xattr/__init__.py", line 16, in _func#012 return func(first, *args)#012IOError: [Errno 95] Operation not supported

I suggest checking that the filesystem supports the necessary functionality (eg setting or checking an xattr) on process start so that the error appears when you start swift rather than when you try and PUT an object.

(fwiw, the user-visible error is just a 500 server error without any additional information.)

Revision history for this message

Pete Zaitcev (zaitcev) wrote on 2012-06-08:

Aww man, this is so true. However, I am against checking for a list of supported filesystems (such as XFS). If we decide to fix this, we need to devise a test that actually stores an xattr. In Fedora, people run Swift on ext4, for example. This requires some coding and testing... Honestly, I'm tempted to punt it. I knew that xattrs were necessary when I installed Swift, even back in 2010.

Revision history for this message

Pete Zaitcev (zaitcev) wrote on 2012-06-08:

(I'm marking this Confirmed, but this may be later rejected)

Changed in swift:
status:	New → Confirmed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-06-08: Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.openstack.org/32269

Changed in swift:
assignee:	nobody → Kun Huang (academicgareth)
status:	Confirmed → In Progress

Revision history for this message

Kun Huang (academicgareth) wrote on 2013-06-21:

I'm trying to fix this, but have some problems.

If we need a check, we must know which dir we will check for xattr

That dir is something like "/srv/node/sda/object/xxx"

/srv/node/ is devices_dir in object-server.conf
sda/ is in ring
object/ means object server

In object-server, we could get the 1st and 3rd easily. But sda in the ring is a little hard.
Typically, the ring.devs looks like
[
{....ip:192.168.1.1, device:sda, region:1...},
{....ip:192.168.1.2, device:sdb, region:1...},
{....ip:192.168.1.3, device:sdc, region:1...},
]
We want the specific ip of one object server, and only way we could use to distinguish device is ip. If user set ip as 192.168.1.x, we could easily get its own device. but if user set ip as 0.0.0.0, there're on other ways to get device. This issue block my whole solution.

Is there any other thought to fix this bug?

Revision history for this message

Madhuri Kumari (madhuri-rai07) wrote on 2013-11-27:

Hi Kun Huang,

This bug is also affecting me.

I tried to write data over filesystem that doesnot support xattr and it failed giving error message:

Object PUT failed: http://127.0.0.1:8080/v1/AUTH_test/new/file.txt 503 Internal Server Error [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently

Log from storage1.error:
object-server: ERROR __call__ error with PUT /sdb1/667/AUTH_test/container/cirros.img : #012Traceback (most recent call last):#012 File "/root/swift/swift/obj/server.py", line 666, in __call__#012 res = method(req)#012 File "/root/swift/swift/common/utils.py", line 1915, in wrapped#012 return func(*a, **kw)#012 File "/root/swift/swift/common/utils.py", line 687, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/root/swift/swift/obj/server.py", line 438, in PUT#012 writer.put(metadata)#012 File "/root/swift/swift/obj/diskfile.py", line 663, in put#012 self._finalize_put, metadata, target_path)#012 File "/root/swift/swift/common/utils.py", line 2212, in force_run_in_thread#012 return self._run_in_eventlet_tpool(func, *args, **kwargs)#012 File "/root/swift/swift/common/utils.py", line 2195, in _run_in_eventlet_tpool#012 raise result#012IOError: [Errno 95] Operation not supported (txn: tx88bb330458364718adb7f-0052943338)

which shows that data is written first to disk partition and after that metadata is written to disk but it failed due to xattr support. So it is better to check support of xattr when object-server starts rather before putting data.

Revision history for this message

Samuel Merritt (torgomatic) wrote on 2013-11-27:

This isn't something that can be checked at object-server start. Devices can be mounted and unmounted while the object server is running, the ring can change while the object server is running... and that's just off the top of my head.

One way to fix this might be to change the ordering of data and metadata writes in the object server. Currently, it looks something like this:

  * create tempfile
  * fallocate()
  * write data, ..., write data
  * write xattrs
  * rename tempfile

If we moved the metadata writing up, then we'd be better off:

  * create tempfile
  * fallocate()
  * write xattrs
  * write data, ..., write data
  * rename tempfile

Then we could catch IOError with the right errno and turn that into a 507 response from the object server, and we can also log a human-readable message about xattr support at the same time.

If we did that, then the proxy could see the 507 and move on to a handoff node before sending any request body to the backend; as it is now, the proxy sends all the bytes over and then gets a 500, which is wasteful.

Revision history for this message

Peter Portante (peter-a-portante) wrote on 2013-11-27:

Sam, that sounds like a good idea. Couple of questions:

Do we know all of the xattrs that need to be written before all the data is given to us? I had thought that we don't always know the content length, or something like that. In that case, we can fall back to the previous behavior.

And don't forget the fsync at the end before the close and the rename. :)

Revision history for this message

Samuel Merritt (torgomatic) wrote on 2013-11-27:

Oh yeah, if we're getting data with Transfer-Encoding: chunked, then we don't know the content length. Darn.

Still, chunked transfers can fail early for other reasons, like running out of disk, so another late failure mode isn't the end of the world. It makes the code more complex, but it might be worth it.

Revision history for this message

Samuel Merritt (torgomatic) wrote on 2013-11-27:

And by "fail early", I of course mean "fail late".

Someone send coffee, please. :)

Kun Huang (academicgareth) on 2013-11-28

Changed in swift:
assignee:	Kun Huang (academicgareth) → nobody

Tom Fifield (fifieldt) on 2013-12-06

Changed in swift:
status:	In Progress → Confirmed

Revision history for this message

Madhuri Kumari (madhuri-rai07) wrote on 2013-12-25:

#10

Hi Samuel,

I have a question how can the underlying filesystem be changed while object server is running?
According to your comment#6, the check for xattr should not be done at start of server.

But i think the filesystem cannot be changed while object server is running. So I think the check can be done at the time server starts rather than before writing the data chunks to partition.

Could you please validate my understanding?

Revision history for this message

Samuel Merritt (torgomatic) wrote on 2013-12-25:

#11

> But i think the filesystem cannot be changed while object server is running.

Think again. :)

One can unmount a disk (say, /dev/sdf mounted at /srv/node/d25), format it with a different filesystem, and then remount it at its old mount point, and one can do this while the object server is running.

clayg (clay-gerrard) on 2014-03-20

Changed in swift:
importance:	Undecided → Low

Matthew Oliver (matt-0) on 2014-06-13

Changed in swift:
assignee:	nobody → Matthew Oliver (matt-0)

Revision history for this message

Matthew Oliver (matt-0) wrote on 2014-06-13:

#12

Unless I'm reading the code wrong, the object server first attempts to read metadata (disk_file.read_metadata()) before writing an object. This check metadata uses xattr.getxattr, which will cause a 'IOError: [Errno 95] Operation not supported' like mentioned at the start of this bug.

At the moment, this isn't handled the same as the setxttr.
My thoughts are we can wrap this code with a better exception handling and should be able to use this to alert back to the user that the xattr isn't supprted by the current filesystem.

Best part is, this would allow us to send an error back before the write, just like Sam has suggested previously.

If I'm wrong please correct me. Either way, I'll do some testing and write up a patch soon.

Matt

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-13:

#13

Fix proposed to branch: master
Review: https://review.openstack.org/99883

Changed in swift:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-11: Fix merged to swift (master)

#14

Reviewed: https://review.openstack.org/99883
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=2659888c921d09bd1dd23cda6ee2f158187d80e6
Submitter: Jenkins
Branch: master

commit 2659888c921d09bd1dd23cda6ee2f158187d80e6
Author: Matthew Oliver <email address hidden>
Date: Fri Jun 13 19:12:31 2014 +1000

When a filesystem does't support xattr return a 507

    Currently when the object server tries to write an object's metadata
    to a filesystem that doesn't support xattr, it errors with a stacktrace
    and returns a 500 error back to the user with no information.

    This patch catches the resulting IOError when attempting to read or write
    the xattr metadata, logs the error nicely and then returns a 507 error
    back to the user.

Seeing as this change is sending back a 507, it also catches and logs
the out of disk space errors (ENOSPC and EDQUOT).

Change-Id: I31932b57582817a0b3b58dd315a996bd0bcbc99b
Closes-Bug: #966671

Changed in swift:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-12-01: Fix proposed to swift (feature/ec)

#15

Fix proposed to branch: feature/ec
Review: https://review.openstack.org/138165

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-12-01: Fix merged to swift (feature/ec)

#16

Download full text (15.6 KiB)

Reviewed: https://review.openstack.org/138165
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0d3ebf09b94b41782b2c2a6bbcf255bf1203eca0
Submitter: Jenkins
Branch: feature/ec

commit 977d7c14daa38ab9c9d79bbf8b92371024b93fc8
Author: John Dickinson <email address hidden>
Date: Wed Nov 26 14:19:08 2014 -0800

Fix tempfile bugs from commit 6978275

    Commit 6978275 changed xprofile middleware's usage of mktemp
    and moved to using tempfile. But it was clearly never tested,
    because the os.close() calls never worked. This patch updates
    that previous patch to use a context to open and close the file.

Change-Id: I40ee42e8539551fd8e4dfb353f50146ab40a7847

commit dec97fc3ba2c71884f1c098e7d9cd1f709f74958
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Nov 26 06:13:29 2014 +0000

Imported Translations from Transifex

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: Ibf319f7cc1b5036ad8031776cf2c6018fb8a0159

commit 01f6e860066640a2ba1406a23c93a72b34ec495e
Author: Clay Gerrard <email address hidden>
Date: Fri Nov 21 17:28:13 2014 -0800

Add Expected Failure for ssync with sys-meta

    Sysmeta included with an object PUT persists with the PUT data - if an
    internal operation such as POST-as-copy during partial failure, or ssync
    with fast-POST (not supported), causes that data to be lost then the
    associated sysmeta will also be lost.

    Since object sys-meta persistence in the face of a POST when the
    original .data is unavailable requires fast-POST with .meta files the
    probetest that validates object sys-meta persistence of a POST when the
    most up-to-date copy of the object with sys-meta is unavailable
    configures an InternalClient with object_post_as_copy = false.

    This non-default configuration option is not supported by ssync and
    results in a loss of sys-meta very similar to the object sys-meta
    failure you would see with object_post_as_copy = true when the COPY part
    of the POST is unable to retrieve the most recently written object with
    sys-meta.

    Until we can fix the default POST behavior to make metadata updates
    without stomping on newer data file timestamps we should expect object
    sys-meta to be "very very best possible but not really guaranteed
    effort".

Until we can fix ssync to replicate metadata updates without stomping on
newer data file timestamps we should expect this test to fail.

    When ssync replication of fast-POST metadata update is fixed this test
    will fail signaling that the expected failure cruft should be removed,
    but other parts of ssync replication will still work and some other bugs
    can be fixed while we wait.

Change-Id: Ifc5d49514de79b78f7715408e0fe0908357771d3

commit a8751ae557616cab1cafd98a338cad352526a262
Author: Cedric Dos Santos <email address hidden>
Date: Tue Nov 25 12:37:05 2014 +0100

Correct misspelled words

In some files I found misspelling words.

bin/swift-reconciler-enqueue#l26
prima...

Reviewed:  https://review.openstack.org/138165
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0d3ebf09b94b41782b2c2a6bbcf255bf1203eca0
Submitter: Jenkins
Branch:    feature/ec

commit 977d7c14daa38ab9c9d79bbf8b92371024b93fc8
Author: John Dickinson <me@not.mn>
Date:   Wed Nov 26 14:19:08 2014 -0800

Fix tempfile bugs from commit 6978275
    
    Commit 6978275 changed xprofile middleware's usage of mktemp
    and moved to using tempfile. But it was clearly never tested,
    because the os.close() calls never worked. This patch updates
    that previous patch to use a context to open and close the file.
    
    Change-Id: I40ee42e8539551fd8e4dfb353f50146ab40a7847

commit dec97fc3ba2c71884f1c098e7d9cd1f709f74958
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Wed Nov 26 06:13:29 2014 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: Ibf319f7cc1b5036ad8031776cf2c6018fb8a0159

commit 01f6e860066640a2ba1406a23c93a72b34ec495e
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Nov 21 17:28:13 2014 -0800

Add Expected Failure for ssync with sys-meta
    
    Sysmeta included with an object PUT persists with the PUT data - if an
    internal operation such as POST-as-copy during partial failure, or ssync
    with fast-POST (not supported), causes that data to be lost then the
    associated sysmeta will also be lost.
    
    Since object sys-meta persistence in the face of a POST when the
    original .data is unavailable requires fast-POST with .meta files the
    probetest that validates object sys-meta persistence of a POST when the
    most up-to-date copy of the object with sys-meta is unavailable
    configures an InternalClient with object_post_as_copy = false.
    
    This non-default configuration option is not supported by ssync and
    results in a loss of sys-meta very similar to the object sys-meta
    failure you would see with object_post_as_copy = true when the COPY part
    of the POST is unable to retrieve the most recently written object with
    sys-meta.
    
    Until we can fix the default POST behavior to make metadata updates
    without stomping on newer data file timestamps we should expect object
    sys-meta to be "very very best possible but not really guaranteed
    effort".
    
    Until we can fix ssync to replicate metadata updates without stomping on
    newer data file timestamps we should expect this test to fail.
    
    When ssync replication of fast-POST metadata update is fixed this test
    will fail signaling that the expected failure cruft should be removed,
    but other parts of ssync replication will still work and some other bugs
    can be fixed while we wait.
    
    Change-Id: Ifc5d49514de79b78f7715408e0fe0908357771d3

commit a8751ae557616cab1cafd98a338cad352526a262
Author: Cedric Dos Santos <cedric.dos.sant@gmail.com>
Date:   Tue Nov 25 12:37:05 2014 +0100

Correct misspelled words
    
    In some files I found misspelling words.
    
    bin/swift-reconciler-enqueue#l26
       primarly => primarily
    swift/account/backend.py#l309
       ommited => omitted
    swift/container/replicator.py#l158
       successfull => successful
    test/unit/account/test_backend.py#1450
       non_existant_policy_index => non_existent_policy_index
    test/unit/account/test_backend.py#1451
       'test-non-existant-policy'=> 'test-non-existent-policy'
    test/unit/account/test_backend.py#1453
       non_existant_policy_index => non_existent_policy_index
    
    Change-Id: I976236e3200a6fbdc20be464acff182b6cface81

commit 98de48d898f1419b0a0cfc273ec778e60331e623
Author: Shilla Saebi <shilla.saebi@gmail.com>
Date:   Sat Nov 22 15:38:48 2014 -0500

Fix typo in apache_deployment doc
    
    Change-Id: I42d76f544290dbda62633de90608d41caadac084

commit a1872b0498e1aa4182a4373c89beeaaaa219ea17
Author: Shilla Saebi <shilla.saebi@gmail.com>
Date:   Sat Nov 22 15:35:10 2014 -0500

Fix 2 typos in admin_guide file
    
    Change-Id: Ibf1e5dbf6ff4747c7f23f6638321ab41bba3021b

commit 0dc4b0a7b75237c09caffdac8c0dfd92bf8e3286
Author: Shilla Saebi <shilla.saebi@gmail.com>
Date:   Sat Nov 22 16:11:37 2014 -0500

Fix typos in overview_large_objects and versioning doc
    
    
    Change-Id: I1a919ad1b0298d5817f9eb2caf5e3bd7b3243c2c

commit 7a0c4d248257259612d3471ab42669ca9d90c573
Author: Takashi Kajinami <kajinamit@nttdata.co.jp>
Date:   Mon Nov 24 22:05:07 2014 +0900

Remove invalid connection checking in db_replicator
    
    Account/Container-replicator checks connection generation and timeout
    in HTTP REPLICATE Request in _repl_to_node, but it doesn't really checks
    connection but only construction of ReplConnection class.
    This patch removes that invalid checking.
    
    Change-Id: Ie6b4062123d998e69c15638b741e7d1ba8a08b62
    Closes-Bug: #1359018

commit 1c9bc0b522bed333b04a46ed7bd2c66a4eb89860
Author: Jay S. Bryant <jsbryant@us.ibm.com>
Date:   Thu Oct 2 14:10:04 2014 -0500

Handle os.listdir failures in object-updater
    
    While investigating bug 1375348 I discovered the problem
    reported there was not limited to the object-auditor.  The
    object-updater has similar bugs.
    
    This patch catches the unhandled exception that can be thrown
    by os.listdir if the self.devices directory is inaccessible.
    
    Change-Id: I6293b840916bb63cf9eebbc05068d9a3c871bdc3
    Related-bug: 1375348

commit 8cc075a8fb7561c736cb38d629f5b3d8ddb67497
Author: Jay S. Bryant <jsbryant@us.ibm.com>
Date:   Thu Nov 20 15:56:58 2014 -0600

mock out os.listdir to return a list
    
    os.listdir returns a list of items.  The test case had been
    written to return a single item which, though not really changing
    the result of the test, was not the best approach.
    
    This patch updates the test case to return a list instead of a single
    item.
    
    Change-Id: I793e0636440c0de0ca339c6592adec3e8b4ee1b4

commit fb353e1756df02622ea257acc987df4ccd094872
Author: John Dickinson <me@not.mn>
Date:   Thu Nov 20 10:22:27 2014 -0800

update AUTHORS
    
    Change-Id: I416e81b20a129377782f5d9298f8b8f5be079c27

commit 6c02adc33e3238f3fe0b75f2857503d1036f4737
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu Nov 20 06:11:14 2014 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Co-Authored-By: Pearl Yajing Tan <pearl.y.tan@seagate.com>
    
    Change-Id: Ifa3e292b8d5afbef8a99121b233e5ea596e672b7

commit 87d8626505c31511911facd5e1a1c3b3a65e8663
Author: Eohyung Lee <liquidnuker@gmail.com>
Date:   Thu Nov 20 11:38:49 2014 +0900

fix example typo
    
    5 * 1024 * 1024 = 5242880
    
    Change-Id: I0eeb6e2d9fbd79103cd8c658627344f73fed9498

commit ddf8b0594bb7e5ea9022982a7c5e15d9b261c22e
Author: Andreas Jaeger <aj@suse.de>
Date:   Wed Nov 19 09:11:55 2014 -0500

Fix translation setup
    
    Fix the output directory, it should be swift/locale.
    This fixes the importing of translations.
    
    Change-Id: I48311773c9d200c3b1739dc796618849416096ed

commit e0307f950bccde1337898e16087af726429e13f4
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Mon Nov 17 12:30:15 2014 -0800

Always use FakeMemcache for in-process tests
    
    Better isolation and consistency for in-process functests to always use
    the FakeMemcache.  If you want to test the real memcache you have real
    functional tests.
    
    Change-Id: Ic483f794e122130bd7694c9a5f9a2b1cd0b9a653

commit 6f9ca6122efac6c1c252a948cd5cc18c58c625ff
Author: Anne Gentle <anne@openstack.org>
Date:   Mon Nov 17 16:11:05 2014 -0600

Adds v1 API documentation to doc/source/api
    
    After discussion https://review.openstack.org/#/c/129384/ moving
    to the doc directory in swift repo.
    
    This lets us eliminate the object-api repo along with all the <service>-
    api repos and move content to audience-centric locations.
    
    Change-Id: Ia0d9973847f7409a02dcc1a0e19400a3c3ecdf32

commit 11a72a4a5084dbcb5539596c50793e45c5dac525
Author: Thiago da Silva <thiago@redhat.com>
Date:   Mon Nov 17 11:33:41 2014 -0500

move slo, dlo after tempauth in pipeline
    
    Noticed that slo and dlo middleware were placed before
    tempauth, they should be placed after
    
    DocImpact
    
    Change-Id: Ia931e2280125d846f248b23e219aebad14c66210
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit 2792fe81a93dbaa95e58f14099db5e11dd8cde68
Author: Daisuke Morita <morita.daisuke@lab.ntt.co.jp>
Date:   Tue Sep 30 11:06:08 2014 -0400

Show the sum of every policy's amount in /recon/async
    
    After the release of Swift ver. 2.0.0, some recon responses do not
    show each policy's information yet. To make things worse, some recon
    results only count on policy-0's score, therefore the total is not
    shown in the recon results.
    
    With this patch, async_pending count of recon results becomes
    policy-aware. Suppose a number of async_pending files for policy-0 is 2
    and a number for policy-1 is 3, recon sums up every policy's amount
    as follows.
    
    $ curl http://<host>:<port>/recon/async
    {"async_pending": 5} # It showed 2 before this commit
    
    Related-Bug: 1375332
    Change-Id: Ifc88b8c9e06b9f022a926a87ed807e938e1e0412

commit c9f824637845f342b6996058e0fea8338bd1305d
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Mon Aug 11 17:09:48 2014 +0100

Make in process functional tests use sample proxy-server.conf
    
    This patch was first motivated by noticing that the proxy
    server pipeline used for in process functional tests was
    out of date with respect to the pipeline in
    /etc/proxy-server.conf.sample. Rather than cut and paste
    the current pipeline into the in process setup, it seems
    like a better idea would be to have the in process tests
    always use the sample config.
    
    A further benefit is that in process functional tests will
    pick up changes to the sample config introduced by patches -
    previously test/functional/__init__.py would need to be
    manually modified to run in process functional tests
    on new middleware for example.
    
    Note: because the pipeline is now loaded using entry points,
    'python setup.py [develop|install]' will now be needed
    before running the tests.
    
    Obvious next steps would be to do the same for the backend
    servers, and to allow alternative config files and dir's
    to be specified, but this patch is the first step.
    
    Also drive-by fixes some typos in proxy-server.conf.sample
    
    Change-Id: If442bd7c2b1721ec92839c4490924ba33e1545d8

commit e429cd81be711f42441a08e34c077dcd7a97bed0
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Nov 13 16:40:05 2014 -0800

Make error limits survive a ring reload
    
    The proxy was storing the error count and last-error time in the
    ring's internal data, specifically in the device dictionaries. This
    works okay, but it means that whenever a ring changes, all the error
    stats reset.
    
    Now the error stats live in the proxy server object, so they survive a
    ring reload.
    
    Better yet, the error stats are now keyed off of the node's
    IP/port/device triple, so if you have the same device in two rings
    (like with multiple storage policies), then the error stats are
    combined. If the proxy server sees a 507 for an objec request in
    policy X, then that will now result in that particular object disk
    being error-limited for requests in policies Y and Z as well.
    
    Change-Id: Icc72b68b99f37367bb16d43688e7e45327e3e022

commit b98fe3b77b6b422e5e5978f6cf82a11fb87aedfc
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Nov 11 17:03:29 2014 -0800

Prefer X-Backend-Timestamp for X-Newest
    
    When a X-Backend-Timestamp is available it would generally preferred
    over a less specific value and sorts correctly against any X-Timestamp
    values anyway.
    
    Change-Id: I08b7eb37ab8bd6eb3afbb7dee44ed07a8c69b57e

commit 466403723c4c1fd575b1340e0f9214ee28f0aeb7
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon Nov 3 14:20:08 2014 -0800

Make resetswift customizable via environment
    
    Instead of recommending to edit resetswift to replace "/dev/sdb1" with
    "/srv/swift-disk", use an environment variable instead. This way I can
    set SAIO_BLOCK_DEVICE=/srv/swift-disk in my .bashrc, and then when I'm
    testing out changes to resetswift, I don't need to remember to edit
    the modified script, nor do I end up submitting changes with the wrong
    default in there.
    
    The variable defaults to /dev/sdb1, so if you use the script unmodified
    and don't set SAIO_BLOCK_DEVICE, nothing changes for you.
    
    Change-Id: I741a8c91c2c54a4f32bc391cd794ef4206402753

commit 331b14238effc9d1928e478bba86122f7e2525c1
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Nov 7 13:53:46 2014 -0800

Reject object names with Unicode surrogates
    
    Technically, you can't encode surrogates into UTF-8 at all, but Python
    2 lets you get away with it. Python 3 does not.
    
    We already have a check for surrogate pairs (commit 0080337), but not
    one for lone surrogates. This commit forbids object names with lone
    surrogates in them.
    
    The problem with surrogates is trivially reproducible:
    
        swift@saio:~$ python2.7
        Python 2.7.3 (default, Feb 27 2014, 19:58:35)
        [GCC 4.6.3] on linux2
        Type "help", "copyright", "credits" or "license" for more information.
        >>> b'\xed\xa0\xbc'.decode('utf-8')
        u'\ud83c'
        >>>
    
        swift@saio:~$ python3.3
        Python 3.3.5 (default, Aug  4 2014, 15:27:24)
        [GCC 4.6.3] on linux
        Type "help", "copyright", "credits" or "license" for more information.
        >>> b'\xed\xa0\xbc'.decode('utf-8')
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
        UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: invalid continuation byte
        >>>
    
    See also http://bugs.python.org/issue9133
    
    Change-Id: I7c31022e8a028c3cdf2ed1586349509d96cfded9

commit 2659888c921d09bd1dd23cda6ee2f158187d80e6
Author: Matthew Oliver <matt@oliver.net.au>
Date:   Fri Jun 13 19:12:31 2014 +1000

When a filesystem does't support xattr return a 507
    
    Currently when the object server tries to write an object's metadata
    to a filesystem that doesn't support xattr, it errors with a stacktrace
    and returns a 500 error back to the user with no information.
    
    This patch catches the resulting IOError when attempting to read or write
    the xattr metadata, logs the error nicely and then returns a 507 error
    back to the user.
    
    Seeing as this change is sending back a 507, it also catches and logs
    the out of disk space errors (ENOSPC and EDQUOT).
    
    Change-Id: I31932b57582817a0b3b58dd315a996bd0bcbc99b
    Closes-Bug: #966671

commit 0a5268c34caa25487c48380a1821e4deac178538
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Tue Sep 16 14:46:08 2014 +0000

Fix bug in swift-ring-builder list_parts
    
    The number of shown replicas in the partition list might differ from the
    actual number of replicas (as shown in the bugreport).
    
    This codes simply iterates for the builder._replica2part2dev and
    remembers the number of replicas for each partition.
    
    The code to find the partitions was moved to swift/common/ring/utils.py
    to make it easier to test, and a test to ensure the correct number of
    replicas is returned was added.
    
    Closes-Bug: 1370070
    Change-Id: Id6a3ed437bb86df2f43f8b0b79aa8ccb50bbe13e

Thierry Carrez (ttx) on 2014-12-15

Changed in swift:
milestone:	none → 2.2.1
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

python-roundup #9133
[2:10] Edit

Bug watches keep track of this bug in other bug trackers.