Under high load, DB Connection Pool get's exhausted.
No matter how high we set the max pool, we consume all the connections,
2015-04-16 15:34:12.946 27446 ERROR oslo_messaging.rpc.dispatcher [req-d1d98dc1-5162-45fa-891d-34693e9e35d9 d5ce18647f3f421b9719fd11faaf3284 abd7ca91f8474abd935f780b3daa547f - - -] Exception during message handling: QueuePool limit of size 500 overflow 10 reached, connection timed out, timeout 30
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/designate/rpc.py", line 178, in _dispatch
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher return super(RPCDispatcher, self)._dispatch(*args, **kwds)
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/designate/central/service.py", line 2132, in update_status
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher self._update_record_status(context, domain_id, status, serial)
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/designate/central/service.py", line 2176, in _update_record_status
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher records = self.storage.find_records(context, criterion=criterion)
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/designate/storage/impl_sqlalchemy/__init__.py", line 650, in find_records
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher sort_dir=sort_dir)
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/designate/storage/impl_sqlalchemy/__init__.py", line 620, in _find_records
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher sort_key, sort_dir)
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/designate/sqlalchemy/base.py", line 251, in _find
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher resultproxy = self.session.execute(query)
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1021, in execute
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher bind, close_with_result=True).execute(clause, params or {})
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 887, in _connection_for_bind
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher conn = engine.contextual_connect(**kw)
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 208, in contextual_connect
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher self._wrap_pool_connect(self.pool.connect, None),
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 215, in _wrap_pool_connect
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher return fn()
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 338, in connect
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher return _ConnectionFairy._checkout(self)
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 645, in _checkout
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher fairy = _ConnectionRecord.checkout(pool)
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 440, in checkout
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher rec = pool._do_get()
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/venvs/designate/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 960, in _do_get
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher (self.size(), self.overflow(), self._timeout))
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher TimeoutError: QueuePool limit of size 500 overflow 10 reached, connection timed out, timeout 30
2015-04-16 15:34:12.946 27446 TRACE oslo_messaging.rpc.dispatcher
Relates to bug 1445123