ssync disconnect exception is not helpful and too noisy

[Admin6]

Hi there,

First sorry if this is an already known bug, I've not enough skills around swift to analyse it deeper.

I have some exceptions logs (let's say it happens not regularly on an average once every 3-4 minutes) regarding EC reconstructions. Here is an example of what I see :

root@GRAPHITE:/var/log/swift# fgrep "s02z2ecd09/225064" object-reconstructor.log
Sep 8 10:41:22 STACO3 object-reconstructor: 10.10.2.52:7000/s02z2ecd09/225064 EXCEPTION in replication.Sender: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/obj/ssync_sender.py", line 126, in __call__#012 self.updates()#012 File "/usr/lib/python2.7/dist-packages/swift/obj/ssync_sender.py", line 346, in updates#012 self.send_put(url_path, df_alt)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/ssync_sender.py", line 406, in send_put#012 self.connection.send('%x\r\n%s\r\n' % (len(msg), msg))#012 File "/usr/lib/python2.7/httplib.py", line 873, in send#012 self.sock.sendall(data)#012 File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 385, in sendall#012 tail = self.send(data, flags)#012 File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 379, in send#012 return self._send_loop(self.fd.send, data, flags)#012 File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 366, in _send_loop#012 return send_method(data, *args)#012error: [Errno 32] Broken pipe

root@GRAPHITE:/var/log/swift# fgrep "s02z2ecd09/225064" object-server.log
Sep 8 10:41:22 STACO2 object-server: 10.10.2.53/s02z2ecd09/225064 EXCEPTION in replication.Receiver: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/obj/ssync_receiver.py", line 166, in __call__#012 for data in self.updates():#012 File "/usr/lib/python2.7/dist-packages/swift/obj/ssync_receiver.py", line 422, in updates#012 method, path = line.strip().split(' ', 1)#012ValueError: need more than 1 value to unpack

I can provide more details if you tell me what too look for.

[clayg (clay-gerrard)]

2020-06-18T07:07:13.020256+00:00 ss0694 object-reconstructor: err 10.150.147.69:6003/d3601/27583 EXCEPTION in ssync.Sender:
Traceback (most recent call last):
  File "/opt/ss/lib/python2.7/site-packages/swift/obj/ssync_sender.py", line 176, in __call__
    response)
  File "/opt/ss/lib/python2.7/site-packages/swift/obj/ssync_sender.py", line 283, in missing_check
    connection.send(b'%x\r\n%s\r\n' % (len(msg), msg))
  File "/opt/ss/lib/python2.7/httplib.py", line 858, in send
    self.sock.sendall(data)
  File "/opt/ss/lib/python2.7/site-packages/eventlet/greenio/base.py", line 403, in sendall
    tail = self.send(data, flags)
  File "/opt/ss/lib/python2.7/site-packages/eventlet/greenio/base.py", line 397, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/ss/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe
​

^ more recent traceback