libra_api hitting timeout limits
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
libra |
Fix Released
|
Critical
|
Andrew Hutchings |
Bug Description
We are seeing errors where libra_api connections timeout and this is causing things to go badly.
After running high stress workloads on a test environment, we see the database contains a number of lb's that are stuck in PENDING_DELETE / PENDING_UPDATE / BUILD states, despite the libra_worker code showing that these actions were successfully completed.
One idea forwarded from discussions was to increase the timeout value (and perhaps to make this configurable).
2013-06-14 10:27:21,648: root - INFO - 15.185.163.106 - - [14/Jun/2013 10:27:21] "PUT /v1.1/loadbalan
2013-06-14 10:27:21,654: root - INFO - (17122) accepted ('15.185.163.106', 50115)
2013-06-14 10:27:51,651: root - INFO - Traceback (most recent call last):
2013-06-14 10:27:51,651: root - INFO - File "/usr/local/
2013-06-14 10:27:51,651: root - INFO - timer()
2013-06-14 10:27:51,651: root - INFO - File "/usr/local/
2013-06-14 10:27:51,651: root - INFO - cb(*args, **kw)
2013-06-14 10:27:51,651: root - INFO - File "/usr/local/
2013-06-14 10:27:51,651: root - INFO - client.
2013-06-14 10:27:51,652: root - INFO - File "/usr/local/
2013-06-14 10:27:51,652: root - INFO - filter(
2013-06-14 10:27:51,652: root - INFO - File "/usr/local/
2013-06-14 10:27:51,652: root - INFO - return list(self)
2013-06-14 10:27:51,652: root - INFO - File "/usr/local/
2013-06-14 10:27:51,652: root - INFO - return self._execute_
2013-06-14 10:27:51,652: root - INFO - File "/usr/local/
2013-06-14 10:27:51,652: root - INFO - close_with_
2013-06-14 10:27:51,652: root - INFO - File "/usr/local/
2013-06-14 10:27:51,652: root - INFO - **kw)
2013-06-14 10:27:51,652: root - INFO - File "/usr/local/
2013-06-14 10:27:51,653: root - INFO - close_with_
2013-06-14 10:27:51,653: root - INFO - File "/usr/local/
2013-06-14 10:27:51,653: root - INFO - return self.transactio
2013-06-14 10:27:51,653: root - INFO - File "/usr/local/
2013-06-14 10:27:51,653: root - INFO - conn = bind.contextual
2013-06-14 10:27:51,653: root - INFO - File "/usr/local/
2013-06-14 10:27:51,653: root - INFO - self.pool.
2013-06-14 10:27:51,653: root - INFO - File "/usr/local/
2013-06-14 10:27:51,653: root - INFO - return _ConnectionFair
2013-06-14 10:27:51,653: root - INFO - File "/usr/local/
2013-06-14 10:27:51,653: root - INFO - rec = self._connectio
2013-06-14 10:27:51,654: root - INFO - File "/usr/local/
2013-06-14 10:27:51,654: root - INFO - (self.size(), self.overflow(), self._timeout))
2013-06-14 10:27:51,654: root - INFO - TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30
2013-06-14 10:27:51,747: root - INFO - Traceback (most recent call last):
File "/usr/local/
result = self.applicatio
File "/usr/lib/
return super(CsAuthPro
File "/usr/lib/
return self.app(env, start_response)
File "/usr/local/
return self.applicatio
File "/usr/local/
app_iter = self.app(environ, replacement_
File "/usr/local/
self.
File "/usr/local/
result = controller(*args, **kwargs)
File "/usr/local/
filter(
File "/usr/local/
ret = list(self[0:1])
File "/usr/local/
return list(res)
File "/usr/local/
return self._execute_
File "/usr/local/
close_
File "/usr/local/
**kw)
File "/usr/local/
close_
File "/usr/local/
return self.transactio
File "/usr/local/
conn = bind.contextual
File "/usr/local/
self.
File "/usr/local/
return _ConnectionFair
File "/usr/local/
rec = self._connectio
File "/usr/local/
(self.size(), self.overflow(), self._timeout))
TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30
http:// stackoverflow. com/questions/ 3360951/ sql-alchemy- connecion- time-out