Lost connection to MySQL server

[Oleksii Chuprykov]

Found in nova/neutron/keystone logs:

   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 566, in service_update
     service_ref = service_get(context, service_id)
   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped
     return f(context, *args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 443, in service_get
     result = query.first()
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2659, in first
     ret = list(self[0:1])
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__
     return list(res)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2761, in __iter__
     return self._execute_and_instances(context)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances
     close_with_result=True)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session
     **kw)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
     execution_options=execution_options)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
     engine, execution_options)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
     conn = bind.contextual_connect()
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
     self._wrap_pool_connect(self.pool.connect, None),
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
     e, dialect, self)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
     util.raise_from_cause(newraise, exc_info)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
     reraise(type(exception), exception, tb=exc_tb, cause=cause)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
     return fn()
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
     return _ConnectionFairy._checkout(self)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
     fairy = _ConnectionRecord.checkout(pool)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 485, in checkout
     rec.checkin()
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
     compat.reraise(exc_type, exc_value, exc_tb)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 482, in checkout
     dbapi_connection = rec.get_connection()
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 563, in get_connection
     self.connection = self.__connect()
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
     connection = self.__pool._invoke_creator(self)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
     return dialect.connect(*cargs, **cparams)
   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
     return self.dbapi.connect(*cargs, **cparams)
   File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
     return Connection(*args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 204, in __init__
     super(Connection, self).__init__(*args, **kwargs2)
 DBConnectionError: (_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 11")

Failed job: https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.smoke_neutron/644/

[Alexander Nagovitsyn (gluk12189)]

https://product-ci.infra.mirantis.net/job/9.x.acceptance.ubuntu.mixed_os_components/8/console

We have 1 failure test with neutron MySQL errors.
According to the logs errors occurred at about the same time

Heat log:
2016-09-09 13:54:10.408 2558 ERROR heat.engine.service [req-42c3ccea-8485-44a2-af50-1fc8de7e003c - - - - -] Service 4e3f36de-9232-4cc9-b0fc-230b160f850d update failed: (_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") [SQL: u'SELECT 1']

Neutron log:
2016-09-09 13:54:06.692 18656 ERROR oslo_messaging.rpc.dispatcher DBConnectionError: (_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") [SQL: u'SELECT 1']

Other log:
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)\n", "type": "StackValidationFailed"}, "title": "Bad Request"}
Fri, 09 Sep 2016 13:54:05 GMT', 'Connection': 'close', 'Content-Type': 'application/json; charset=UTF-8', 'Content-Length': '150', 'X-Openstack-Request-Id': 'req-1703418c-f1a8-4528-9d09-e96248ac8ac5'} {"NeutronError": {"message": "Request Failed: internal server error while processing your request.", "type": "HTTPInternalServerError", "detail": ""}}

NeutronError": Fri, 09 Sep 2016 13:54:18
 {"message": "Unable to complete operation on network afd2304c-bfa0-4f6a-a555-dc000bbbf17a. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}}