In our internal environment, we are noticing error whenever haproxy sitting between barbican api server and mysql cluster is restarted.
The issue is related to pooled checked out connection when haproxy restart makes the existing connection stale and cannot be used. So client (barbican-api) executes mysql statement on stale connection and reports error. There is retry logic added in mysql engine connect call but that logic is not invoked.
This kind of issue is addressed in oslo db framework but we don't use this in barbican and that's why will need to have local code to address this. May be we should look into switching to oslo db framework as it provides solution to common problems.
2016-06-01 15:49:43,631 ERROR __init__ handler Secret creation failure seen - please contact site administrator.
Traceback (most recent call last):
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/barbican/api/controllers/__init__.py", line 103, in handler
return fn(inst, *args, **kwargs)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/barbican/api/controllers/__init__.py", line 89, in enforcer
return fn(inst, *args, **kwargs)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/barbican/api/controllers/__init__.py", line 152, in content_types_enforcer
return fn(inst, *args, **kwargs)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/barbican/api/controllers/secrets.py", line 334, in on_post
project = res.get_or_create_project(external_project_id)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/barbican/common/resources.py", line 43, in get_or_create_project
suppress_exception=True)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/barbican/model/repositories.py", line 579, in find_by_external_project_id
entity = query.one()
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2693, in one
ret = list(self)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
return self._execute_and_instances(context)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
return meth(self, multiparams, params)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
compiled_sql, distilled_params
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
context)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
exc_info
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
context)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
cursor.execute(statement, parameters)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'SELECT projects.id AS projects_id, projects.created_at AS projects_created_at, projects.updated_at AS projects_updated_at, projects.deleted_at AS projects_deleted_at, projects.deleted AS projects_deleted, projects.status AS projects_status, projects.external_id AS projects_external_id \nFROM projects \nWHERE projects.external_id = %s'] [parameters: ('d1c1dc7715704dbe89cfc5549ad0d608',)]
(barbican.api.middleware.context): 2016-06-01 15:49:43,638 INFO context process_response Processed request: 500 Internal Server Error - POST https://10.246.27.6:9311/v1/secrets/
Oslo DB framework addresses this issue by listing for engine_connect event and adding ping listener for handling error.
oslo db : https:/