Heat Stack Causes MariaDB Too Many Connections

[Daneyon Hansen]

OpenStack works just fine until creating a Heat stack. Creating the stack appears too creates too many connections to the DB, causing the DB to reach the max_connection count. Therefore, the DB blocks all connections from the source IP for a specific period of time. Since all OpenStack services are running on the same host, all services are blocked.

From the Heat Engine logs:

heatengine_1 | six.reraise(type(newraise), newraise, exc_info[2])
heatengine_1 | File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 109, in _wrap_pool_connect
heatengine_1 | return fn()
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 338, in connect
heatengine_1 | return _ConnectionFairy._checkout(self)
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 641, in _checkout
heatengine_1 | fairy = _ConnectionRecord.checkout(pool)
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 440, in checkout
heatengine_1 | rec = pool._do_get()
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 961, in _do_get
heatengine_1 | return self._create_connection()
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 285, in _create_connection
heatengine_1 | return _ConnectionRecord(self)
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 411, in __init__
heatengine_1 | self.connection = self.__connect()
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in __connect
heatengine_1 | connection = self.__pool._creator()
heatengine_1 | File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 94, in connect
heatengine_1 | raise original_exception
heatengine_1 | OperationalError: (OperationalError) (1040, 'Too many connections') None None

Increasing the max_connection count fixes this problem.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/192011