It takes up to two minutes to restore a connection to the database. In the worst case scenario a test can fail before the connection is restored.
--- 2015-11-12 15:53:16.247 17115 TRACE nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() 2015-11-12 15:53:16.247 17115 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 525, in get_connection 2015-11-12 15:53:16.247 17115 TRACE nova.servicegroup.drivers.db self.connection = self.__connect() 2015-11-12 15:53:16.247 17115 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 537, in __connect 2015-11-12 15:53:16.247 17115 TRACE nova.servicegroup.drivers.db connection = self.__pool._creator() 2015-11-12 15:53:16.247 17115 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 200, in connect 2015-11-12 15:53:16.247 17115 TRACE nova.servicegroup.drivers.db raise original_exception 2015-11-12 15:53:16.247 17115 TRACE nova.servicegroup.drivers.db DBConnectionError: (OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") None None 2015-11-12 15:53:16.247 17115 TRACE nova.servicegroup.drivers.db 2015-11-12 15:53:26.872 17115 INFO nova.scheduler.host_manager [req-79360310-a2a9-4ada-bf84-e058145484f2 - - - - -] Successfully synced instances from host 'node-4.test.domain.local'. 2015-11-12 15:54:13.195 17115 ERROR nova.servicegroup.drivers.db [req-a3fd5579-559e-4eb7-b56b-364b246f5910 - - - - -] Recovered model server connection! 2015-11-12 15:55:13.181 17115 ERROR sqlalchemy.pool.QueuePool [req-a3fd5579-559e-4eb7-b56b-364b246f5910 - - - - -] Exception closing connection <_mysql.connection closed at 2ab4190> 2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.pool.QueuePool Traceback (most recent call last): 2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.pool.QueuePool File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 250, in _close_connection 2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.pool.QueuePool self._dialect.do_close(connection) 2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.pool.QueuePool File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 412, in do_close 2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.pool.QueuePool dbapi_connection.close() 2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.pool.QueuePool ProgrammingError: closing a closed connection 2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.pool.QueuePool 2015-11-12 15:55:43.456 17115 INFO nova.scheduler.host_manager [req-9d41807c-6267-43e0-94a8-4988463b6561 - - - - -] Successfully synced instances from host 'node-4.test.domain.local'.
It takes up to two minutes to restore a connection to the database. In the worst case scenario a test can fail before the connection is restored.
--- up.drivers. db dbapi_connection = rec.get_ connection( ) up.drivers. db File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ pool.py" , line 525, in get_connection up.drivers. db self.connection = self.__connect() up.drivers. db File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ pool.py" , line 537, in __connect up.drivers. db connection = self.__ pool._creator( ) up.drivers. db File "/usr/lib/ python2. 7/dist- packages/ oslo_db/ sqlalchemy/ compat/ handle_ error.py" , line 200, in connect up.drivers. db raise original_exception up.drivers. db DBConnectionError: (OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") None None up.drivers. db host_manager [req-79360310- a2a9-4ada- bf84-e058145484 f2 - - - - -] Successfully synced instances from host 'node-4. test.domain. local'. up.drivers. db [req-a3fd5579- 559e-4eb7- b56b-364b246f59 10 - - - - -] Recovered model server connection! pool.QueuePool [req-a3fd5579- 559e-4eb7- b56b-364b246f59 10 - - - - -] Exception closing connection <_mysql.connection closed at 2ab4190> pool.QueuePool Traceback (most recent call last): pool.QueuePool File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ pool.py" , line 250, in _close_connection pool.QueuePool self._dialect. do_close( connection) pool.QueuePool File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ engine/ default. py", line 412, in do_close pool.QueuePool dbapi_connectio n.close( ) pool.QueuePool ProgrammingError: closing a closed connection pool.QueuePool host_manager [req-9d41807c- 6267-43e0- 94a8-4988463b65 61 - - - - -] Successfully synced instances from host 'node-4. test.domain. local'.
2015-11-12 15:53:16.247 17115 TRACE nova.servicegro
2015-11-12 15:53:16.247 17115 TRACE nova.servicegro
2015-11-12 15:53:16.247 17115 TRACE nova.servicegro
2015-11-12 15:53:16.247 17115 TRACE nova.servicegro
2015-11-12 15:53:16.247 17115 TRACE nova.servicegro
2015-11-12 15:53:16.247 17115 TRACE nova.servicegro
2015-11-12 15:53:16.247 17115 TRACE nova.servicegro
2015-11-12 15:53:16.247 17115 TRACE nova.servicegro
2015-11-12 15:53:16.247 17115 TRACE nova.servicegro
2015-11-12 15:53:26.872 17115 INFO nova.scheduler.
2015-11-12 15:54:13.195 17115 ERROR nova.servicegro
2015-11-12 15:55:13.181 17115 ERROR sqlalchemy.
2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.
2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.
2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.
2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.
2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.
2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.
2015-11-12 15:55:13.181 17115 TRACE sqlalchemy.
2015-11-12 15:55:43.456 17115 INFO nova.scheduler.