(192.168.91.43):
Aug 27 12:45:29 localhost object-server: ERROR __call__ error with PUT /d24/12/AUTH_BMPA_Intel/ssbench_64MB_000075_ec1014-12/large_324011 :
#012Traceback (most recent call last):#012 File "/usr/local/lib/python2.7/dist-packages/swift/obj/server.py", line 938, in __call__#012
res = method(req)#012 File "/usr/local/lib/python2.7/dist-packages/swift/common/utils.py", line 2668, in wrapped#012 return func(*a, **k
w)#012 File "/usr/local/lib/python2.7/dist-packages/swift/common/utils.py", line 1208, in _timing_stats#012 resp = func(ctrl, *args, **k
wargs)#012 File "/usr/local/lib/python2.7/dist-packages/swift/obj/server.py", line 617, in PUT#012 for chunk in iter(timeout_reader, '')
:#012 File "/usr/local/lib/python2.7/dist-packages/swift/obj/server.py", line 397, in timeout_reader#012 return file_like.read(self.netw
ork_chunk_size)#012 File "/usr/local/lib/python2.7/dist-packages/swift/common/utils.py", line 494, in read#012 chunk = next(self.iterato
r)#012 File "/usr/local/lib/python2.7/dist-packages/swift/obj/ssync_receiver.py", line 346, in subreq_iter#012 min(left, self.app.networ
k_chunk_size))#012 File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 188, in read#012 return self._chunked_read(self.rfile,
length)#012 File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 178, in _chunked_read#012 self.chunk_length = int(rfile.read
line().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: ''
Aug 27 12:45:29 localhost object-server: 192.168.91.41/d24/12 EXCEPTION in replication.Receiver: #012Traceback (most recent call last):#012
File "/usr/local/lib/python2.7/dist-packages/swift/obj/ssync_receiver.py", line 104, in __call__#012 for data in self.updates():#012 Fi
le "/usr/local/lib/python2.7/dist-packages/swift/obj/ssync_receiver.py", line 298, in updates#012 line = self.fp.readline(self.app.networ
k_chunk_size)#012 File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 193, in readline#012 return self._chunked_read(self.rfi
le, size, True)#012 File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 178, in _chunked_read#012 self.chunk_length = int(rfi
le.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: ''
(192.168.91.41):
Aug 27 12:45:29 localhost object-reconstructor: 192.168.91.43:6003/d24/12 EXCEPTION in replication.Sender: #012Traceback (most recent call last):#012 File "/usr/local/lib/python2.7/dist-packages/swift/obj/ssync_sender.py", line 75, in __call__#012 self.updates()#012 File "/usr/local/lib/python2.7/dist-packages/swift/obj/ssync_sender.py", line 290, in updates#012 self.send_put(url_path, df)#012 File "/usr/local/lib/python2.7/dist-packages/swift/obj/ssync_sender.py", line 344, in send_put#012 for chunk in df.reader():#012 File "/usr/local/lib/python2.7/dist-packages/swift/obj/reconstructor.py", line 97, in reader#012 for chunk in self.rebuilt_fragment_iter:#012 File "/usr/local/lib/python2.7/dist-packages/swift/obj/reconstructor.py", line 331, in fragment_payload_iter#012 policy, fragment_payload, frag_index)#012 File "/usr/local/lib/python2.7/dist-packages/swift/obj/reconstructor.py", line 290, in _reconstruct#012 [frag_index])[0]#012 File "/usr/lib/python2.7/dist-packages/pyeclib/ec_iface.py", line 247, in reconstruct#012 available_fragment_payloads, missing_fragment_indexes)#012 File "/usr/lib/python2.7/dist-packages/pyeclib/core.py", line 116, in reconstruct#012 raise ECDriverError(e)#012ECDriverError: Reconstruct ERROR: Fragment integrity check failed. Please inspect syslog for liberasurecode error report.
(192.168.91.41 syslog):
Aug 27 12:45:29 r7n01 liberasurecode[3413]: Invalid fragment header (get idx)!
Aug 27 12:45:29 r7n01 liberasurecode[3413]: Could not properly partition the fragments!
good logs paul!
I think cleaning up the way that ssync sender disconnects would be very helpful to taking some noise out of the logs.
SSYNC has a (weird?) tunneling protocol (currently) where it wraps HTTP requests inside of an ongoing chunked-transfer HTTP request. When it disconnects that old eventlet chunked transfer ValueError disconnect message pops (I'm not sure why that shows up in the logs unless eventlet_debug = True).
In order to disconnect cleanly ssync_sender can write a chunked transfer termination line to the wire.