TCP connections are not sending their entire buffers
Bug #1552864 reported by
Rahman Syed
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Designate |
Fix Released
|
Critical
|
Graham Hayes |
Bug Description
When sending large payloads over networks, it appears that the connection is dropped before the entire TCP payload is sent from Designate.
For example, when performing an AXFR for a zone with a large number of records (several hundred or higher) in a deployment with mDNS and PowerDNS on a network that fragments TCP payloads, error messages can be observed. PDNS example: "Remote nameserver closed TCP connection"
Reporter: Erik Andersson
Changed in designate: | |
importance: | Undecided → Critical |
milestone: | none → mitaka-rc1 |
status: | New → Triaged |
To post a comment you must log in.
OK, after changing https:/ /github. com/openstack/ designate/ blob/d5d0706705 c64dba847cea5a3 0b4a6be39ecd63f /designate/ service. py#L342 to sendall() we get:
2016-03-03 19:02:42 ERROR designate.service [req-76111e6a- 17e9-4a9e- 8e03-39ac34d829 cb - - - - -] Unhandled exception while processing request from 89.101. 195.206: 60079 graham/ designate/ designate/ service. py", line 343, in _dns_handle sendall( tcp_response) lib/python2. 7/dist- packages/ eventlet/ greenio/ base.py" , line 388, in sendall data[tail: ], flags) lib/python2. 7/dist- packages/ eventlet/ greenio/ base.py" , line 379, in send loop(self. fd.send, data, flags) lib/python2. 7/dist- packages/ eventlet/ greenio/ base.py" , line 374, in _send_loop exc=socket. timeout( "timed out")) lib/python2. 7/dist- packages/ eventlet/ greenio/ base.py" , line 203, in _trampoline closed= self._mark_ as_closed) lib/python2. 7/dist- packages/ eventlet/ hubs/__ init__. py", line 162, in trampoline lib/python2. 7/dist- packages/ eventlet/ hubs/hub. py", line 294, in switch switch( )
2016-03-03 19:02:42.008 TRACE designate.service Traceback (most recent call last):
2016-03-03 19:02:42.008 TRACE designate.service File "/home/
2016-03-03 19:02:42.008 TRACE designate.service client.
2016-03-03 19:02:42.008 TRACE designate.service File "/usr/local/
2016-03-03 19:02:42.008 TRACE designate.service tail += self.send(
2016-03-03 19:02:42.008 TRACE designate.service File "/usr/local/
2016-03-03 19:02:42.008 TRACE designate.service return self._send_
2016-03-03 19:02:42.008 TRACE designate.service File "/usr/local/
2016-03-03 19:02:42.008 TRACE designate.service timeout_
2016-03-03 19:02:42.008 TRACE designate.service File "/usr/local/
2016-03-03 19:02:42.008 TRACE designate.service mark_as_
2016-03-03 19:02:42.008 TRACE designate.service File "/usr/local/
2016-03-03 19:02:42.008 TRACE designate.service return hub.switch()
2016-03-03 19:02:42.008 TRACE designate.service File "/usr/local/
2016-03-03 19:02:42.008 TRACE designate.service return self.greenlet.
2016-03-03 19:02:42.008 TRACE designate.service timeout: timed out
2016-03-03 19:02:42.008 TRACE designate.service
Adding an "eventlet.sleep(0)" before the sendall() allows this to complete properly