| 2016-04-11 02:19:10 |
drax |
description |
In Object GET operation, in case object service crashes, client connection is not closed as eventlet layer.
Below iterator is used to send data to client :
================
def _make_app_iter(self, req, node, source):
"""
Returns an iterator over the contents of the source (via its read
func). There is also quite a bit of cleanup to ensure garbage
collection works and the underlying socket of the source is closed.
:param req: incoming request object
:param source: The httplib.Response object this iterator should read
from.
:param node: The node the source is reading from, for logging purposes.
"""
#NOTE: I am not pasting the complete code, but areas where code can create issues.
#NOTE: in case chunk is empty by Object service this iterator will finish without any exception. Rather exit gracefully although complete body of the object was not read.
if not chunk:
break
with ChunkWriteTimeout(self.app.client_timeout, GenericTimeout):
yield chunk
================
Please note in case Object service crashes in between when transferring Body of the object this iterator will not any exception rather it will simply finish and above layer of eventlet.wsgi won't close the connection.
Below is the function which is called in an loop in BaseRequestHandler.py. function is overridden in eventlet.wsgi.HttpProtocol class:
================
def handle_one_request(self):
if self.server.max_http_version:
self.protocol_version = self.server.max_http_version
#NOTE: As per the client it is still expecting chunk from proxy service , but as object service has crashed and iterator finished cleanly it cannot send anymore data to client. But this statement will indefinitely hang now.
try:
self.raw_requestline = self.rfile.readline(self.server.url_length_limit)
if not self.raw_requestline:
self.close_connection = 1
return
================
In a normal Object GET operation, whenever the Client gets all the data from the proxy service it closes the connection from its end and the "readline" call returns empty string immediately. As raw_requestline is empty this will set the close_connection to true and cause the request loop to finish at BaseRequestHandler level
================
class HTTPServer(SocketServer.TCPServer):
def handle(self):
"""Handle multiple requests if necessary."""
self.close_connection = 1
self.handle_one_request()
while not self.close_connection:
self.handle_one_request()
class BaseRequestHandler:
def __init__(self, request, client_address, server):
self.request = request
self.client_address = client_address
self.server = server
try:
self.setup()
self.handle()
self.finish()
================
In the cases where the iterator finishes gracefully without checking that completion of body transfer this issue will occur.
I have below fix to resolve this issue in "_make_app_iter" funtion.
================
def _make_app_iter(self, req, node, source):
+ content_length = source.length or 0
bytes_read_from_source = 0
if not chunk:
+ if (content_length - bytes_read_from_source) > 0:
+ self.app.logger.error(_('Incomplete bytes : %s' %bytes_read_from_source))
+ raise
except GeneratorExit:
if not req.environ.get('swift.non_client_disconnect'):
self.app.logger.warn(_('Client disconnected on read'))
+ if (content_length - bytes_read_from_source) > 0:
+ self.app.logger.error(_('Incomplete bytes : %s' %bytes_read_from_source))
+ raise
================
In case whenever generator exits, it will check whether content length requested is completed or not . In case of not an exception will be generated. |
In Object GET operation, in case object service crashes, client connection is not closed as eventlet layer. For certain client if they have not introduced a timeout for chunks then they will hang indefinitely.
Below iterator is used to send data to client :
================
def _make_app_iter(self, req, node, source):
"""
Returns an iterator over the contents of the source (via its read
func). There is also quite a bit of cleanup to ensure garbage
collection works and the underlying socket of the source is closed.
:param req: incoming request object
:param source: The httplib.Response object this iterator should read
from.
:param node: The node the source is reading from, for logging purposes.
"""
#NOTE: I am not pasting the complete code, but areas where code can create issues.
#NOTE: in case chunk is empty by Object service this iterator will finish without any exception. Rather exit gracefully although complete body of the object was not read.
if not chunk:
break
with ChunkWriteTimeout(self.app.client_timeout, GenericTimeout):
yield chunk
================
Please note in case Object service crashes in between when transferring Body of the object this iterator will not any exception rather it will simply finish and above layer of eventlet.wsgi won't close the connection.
Below is the function which is called in an loop in BaseRequestHandler.py. function is overridden in eventlet.wsgi.HttpProtocol class:
================
def handle_one_request(self):
if self.server.max_http_version:
self.protocol_version = self.server.max_http_version
#NOTE: As per the client it is still expecting chunk from proxy service , but as object service has crashed and iterator finished cleanly it cannot send anymore data to client. But this statement will indefinitely hang now.
try:
self.raw_requestline = self.rfile.readline(self.server.url_length_limit)
if not self.raw_requestline:
self.close_connection = 1
return
================
In a normal Object GET operation, whenever the Client gets all the data from the proxy service it closes the connection from its end and the "readline" call returns empty string immediately. As raw_requestline is empty this will set the close_connection to true and cause the request loop to finish at BaseRequestHandler level
================
class HTTPServer(SocketServer.TCPServer):
def handle(self):
"""Handle multiple requests if necessary."""
self.close_connection = 1
self.handle_one_request()
while not self.close_connection:
self.handle_one_request()
class BaseRequestHandler:
def __init__(self, request, client_address, server):
self.request = request
self.client_address = client_address
self.server = server
try:
self.setup()
self.handle()
self.finish()
================
In the cases where the iterator finishes gracefully without checking that completion of body transfer this issue will occur.
I have below fix to resolve this issue in "_make_app_iter" funtion.
================
def _make_app_iter(self, req, node, source):
+ content_length = source.length or 0
bytes_read_from_source = 0
if not chunk:
+ if (content_length - bytes_read_from_source) > 0:
+ self.app.logger.error(_('Incomplete bytes : %s' %bytes_read_from_source))
+ raise
except GeneratorExit:
if not req.environ.get('swift.non_client_disconnect'):
self.app.logger.warn(_('Client disconnected on read'))
+ if (content_length - bytes_read_from_source) > 0:
+ self.app.logger.error(_('Incomplete bytes : %s' %bytes_read_from_source))
+ raise
================
In case whenever generator exits, it will check whether content length requested is completed or not . In case of not an exception will be generated. |
|
| 2016-04-11 02:21:15 |
drax |
description |
In Object GET operation, in case object service crashes, client connection is not closed as eventlet layer. For certain client if they have not introduced a timeout for chunks then they will hang indefinitely.
Below iterator is used to send data to client :
================
def _make_app_iter(self, req, node, source):
"""
Returns an iterator over the contents of the source (via its read
func). There is also quite a bit of cleanup to ensure garbage
collection works and the underlying socket of the source is closed.
:param req: incoming request object
:param source: The httplib.Response object this iterator should read
from.
:param node: The node the source is reading from, for logging purposes.
"""
#NOTE: I am not pasting the complete code, but areas where code can create issues.
#NOTE: in case chunk is empty by Object service this iterator will finish without any exception. Rather exit gracefully although complete body of the object was not read.
if not chunk:
break
with ChunkWriteTimeout(self.app.client_timeout, GenericTimeout):
yield chunk
================
Please note in case Object service crashes in between when transferring Body of the object this iterator will not any exception rather it will simply finish and above layer of eventlet.wsgi won't close the connection.
Below is the function which is called in an loop in BaseRequestHandler.py. function is overridden in eventlet.wsgi.HttpProtocol class:
================
def handle_one_request(self):
if self.server.max_http_version:
self.protocol_version = self.server.max_http_version
#NOTE: As per the client it is still expecting chunk from proxy service , but as object service has crashed and iterator finished cleanly it cannot send anymore data to client. But this statement will indefinitely hang now.
try:
self.raw_requestline = self.rfile.readline(self.server.url_length_limit)
if not self.raw_requestline:
self.close_connection = 1
return
================
In a normal Object GET operation, whenever the Client gets all the data from the proxy service it closes the connection from its end and the "readline" call returns empty string immediately. As raw_requestline is empty this will set the close_connection to true and cause the request loop to finish at BaseRequestHandler level
================
class HTTPServer(SocketServer.TCPServer):
def handle(self):
"""Handle multiple requests if necessary."""
self.close_connection = 1
self.handle_one_request()
while not self.close_connection:
self.handle_one_request()
class BaseRequestHandler:
def __init__(self, request, client_address, server):
self.request = request
self.client_address = client_address
self.server = server
try:
self.setup()
self.handle()
self.finish()
================
In the cases where the iterator finishes gracefully without checking that completion of body transfer this issue will occur.
I have below fix to resolve this issue in "_make_app_iter" funtion.
================
def _make_app_iter(self, req, node, source):
+ content_length = source.length or 0
bytes_read_from_source = 0
if not chunk:
+ if (content_length - bytes_read_from_source) > 0:
+ self.app.logger.error(_('Incomplete bytes : %s' %bytes_read_from_source))
+ raise
except GeneratorExit:
if not req.environ.get('swift.non_client_disconnect'):
self.app.logger.warn(_('Client disconnected on read'))
+ if (content_length - bytes_read_from_source) > 0:
+ self.app.logger.error(_('Incomplete bytes : %s' %bytes_read_from_source))
+ raise
================
In case whenever generator exits, it will check whether content length requested is completed or not . In case of not an exception will be generated. |
In Object GET operation, in case object service crashes, client connection is not closed as eventlet layer. For certain client if they have not introduced a timeout for chunks then they will hang indefinitely.
Below iterator is used to send data to client :
================
def _make_app_iter(self, req, node, source):
"""
Returns an iterator over the contents of the source (via its read
func). There is also quite a bit of cleanup to ensure garbage
collection works and the underlying socket of the source is closed.
:param req: incoming request object
:param source: The httplib.Response object this iterator should read
from.
:param node: The node the source is reading from, for logging purposes.
"""
#NOTE: I am not pasting the complete code, but areas where code can create issues.
#NOTE: in case chunk is empty by Object service this iterator will finish without any exception. Rather exit gracefully although complete body of the object was not read.
if not chunk:
break
with ChunkWriteTimeout(self.app.client_timeout, GenericTimeout):
yield chunk
================
Please note in case Object service crashes in between when transferring Body of the object this iterator will not any exception rather it will simply finish and above layer of eventlet.wsgi won't close the connection.
Below is the function which is called in an loop in HTTPServer.py. function is overridden in eventlet.wsgi.HttpProtocol class:
================
def handle_one_request(self):
if self.server.max_http_version:
self.protocol_version = self.server.max_http_version
#NOTE: As per the client it is still expecting chunk from proxy service , but as object service has crashed and iterator finished cleanly it cannot send anymore data to client. But this statement will indefinitely hang now.
try:
self.raw_requestline = self.rfile.readline(self.server.url_length_limit)
if not self.raw_requestline:
self.close_connection = 1
return
================
In a normal Object GET operation, whenever the Client gets all the data from the proxy service it closes the connection from its end and the "readline" call returns empty string immediately. As raw_requestline is empty this will set the close_connection to true and cause the request loop to finish at BaseRequestHandler level
================
class HTTPServer(SocketServer.TCPServer):
def handle(self):
"""Handle multiple requests if necessary."""
self.close_connection = 1
self.handle_one_request()
while not self.close_connection:
self.handle_one_request()
class BaseRequestHandler:
def __init__(self, request, client_address, server):
self.request = request
self.client_address = client_address
self.server = server
try:
self.setup()
self.handle()
self.finish()
================
In the cases where the iterator finishes gracefully without checking that completion of body transfer this issue will occur.
I have below fix to resolve this issue in "_make_app_iter" funtion.
================
def _make_app_iter(self, req, node, source):
+ content_length = source.length or 0
bytes_read_from_source = 0
if not chunk:
+ if (content_length - bytes_read_from_source) > 0:
+ self.app.logger.error(_('Incomplete bytes : %s' %bytes_read_from_source))
+ raise
except GeneratorExit:
if not req.environ.get('swift.non_client_disconnect'):
self.app.logger.warn(_('Client disconnected on read'))
+ if (content_length - bytes_read_from_source) > 0:
+ self.app.logger.error(_('Incomplete bytes : %s' %bytes_read_from_source))
+ raise
================
In case whenever generator exits, it will check whether content length requested is completed or not . In case of not an exception will be generated. |
|
