error when writing to handoffs

[John Dickinson]

When two of the three primary replicas are missing (eg not running), an object PUT returns an error:

Nov 29 22:13:42 openslice proxy-server ERROR with Object server 127.0.0.1:6010/sdb1 re: Expect: 100-continue on /AUTH_abc/c/x: Connection refused (txn: txc8c4c6533f884e32a60c950ef652f904) (client_ip: x.x.x.x)
Nov 29 22:13:42 openslice proxy-server ERROR with Object server 127.0.0.1:6040/sdb4 re: Expect: 100-continue on /AUTH_abc/c/x: Connection refused (txn: txc8c4c6533f884e32a60c950ef652f904) (client_ip: x.x.x.x)
Nov 29 22:13:42 openslice proxy-server STDOUT: Traceback (most recent call last):
Nov 29 22:13:42 openslice proxy-server STDOUT: File "/usr/lib/pymodules/python2.6/eventlet/hubs/poll.py", line 105, in wait
Nov 29 22:13:42 openslice proxy-server STDOUT: writers.get(fileno, noop).cb(fileno)
Nov 29 22:13:42 openslice proxy-server STDOUT: File "/usr/lib/pymodules/python2.6/eventlet/greenthread.py", line 192, in main
Nov 29 22:13:42 openslice proxy-server STDOUT: result = function(*args, **kwargs)
Nov 29 22:13:42 openslice proxy-server STDOUT: File "/home/john/swift/swift/proxy/controllers/obj.py", line 521, in _connect_put_node
Nov 29 22:13:42 openslice proxy-server STDOUT: for node in nodes:
Nov 29 22:13:42 openslice proxy-server STDOUT: ValueError: generator already executing
Nov 29 22:13:42 openslice proxy-server STDOUT: Removing descriptor: 15
Nov 29 22:13:42 openslice proxy-server Handoff requested (1) (txn: txc8c4c6533f884e32a60c950ef652f904) (client_ip: x.x.x.x)
Nov 29 22:13:42 openslice proxy-server ERROR Unhandled exception in request: #012Traceback (most recent call last):#012 File "/home/john/swift/swift/proxy/server.py", line 240, in handle_request#012 return handler(req)#012 File "/home/john/swift/swift/common/utils.py", line 1457, in wrapped#012 return func(*a, **kw)#012 File "/home/john/swift/swift/proxy/controllers/base.py", line 86, in wrapped#012 return func(*a, **kw)#012 File "/home/john/swift/swift/proxy/controllers/obj.py", line 733, in PUT#012 conns = [conn for conn in pile if conn]#012 File "/usr/lib/pymodules/python2.6/eventlet/greenpool.py", line 218, in next#012 return self.waiters.get().wait()#012 File "/usr/lib/pymodules/python2.6/eventlet/greenthread.py", line 166, in wait#012 return self._exit_event.wait()#012 File "/usr/lib/pymodules/python2.6/eventlet/event.py", line 120, in wait#012 current.throw(*self._exc)#012 File "/usr/lib/pymodules/python2.6/eventlet/greenthread.py", line 192, in main#012 result = function(*args, **kwargs)#012 File "/home/john/swift/swift/proxy/controllers/obj.py", line 521, in _connect_put_node#012 for node in nodes:#012ValueError: generator already executing (txn: txc8c4c6533f884e32a60c950ef652f904) (client_ip: x.x.x.x)
Nov 29 22:13:42 openslice object-server 127.0.0.1 - - [29/Nov/2012:22:13:42 +0000] "PUT /sdb3/123050/AUTH_abc/c/x" 499 - "-" "txc8c4c6533f884e32a60c950ef652f904" "curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5" 0.0050
Nov 29 22:13:42 openslice object-server 127.0.0.1 - - [29/Nov/2012:22:13:42 +0000] "PUT /sdb2/123050/AUTH_abc/c/x" 499 - "-" "txc8c4c6533f884e32a60c950ef652f904" "curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5" 0.0040

[AllenGuo (xiaoguo)]

I tried this(PUT an object, and then remove two of the three replica, then,an object PUT ), but didn't get this error. More detailed info of your practice?

[John Dickinson (notmyname)]

Find which three primary servers an object will talk to for a PUT. Kill 2 of them, and then do the PUT. This should fall over to handoff nodes, but instead it causes a 500.

[Chuck Thier (cthier)]

I tried to reproduce as well on my SAIO by stopping two of the object servers, and doing a PUT, but could not reproduce.

[clayg (clay-gerrard)]

notmyname posted this for more info:

https://gist.github.com/2909fa42c97e50138e3c

At least two other core (myself included) have not been able to reproduce yet.

[Daniele Valeriani (omame)]

Tried to follow the steps of comment #2, but hands off correctly: I get the three replicas by using two of the handoff nodes. I'm running swift-1.7.5 on a multi-server installation running Ubuntu Precise.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Object Storage (swift) because there has been no activity for 60 days.]