MySQL server has gone away error related to HA event

Bug #1589673 reported by Arun Kant on 2016-06-06
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Barbican
Fix Released
High
Jeremy Liu

Bug Description

In our internal environment, we are noticing error whenever haproxy sitting between barbican api server and mysql cluster is restarted.

The issue is related to pooled checked out connection when haproxy restart makes the existing connection stale and cannot be used. So client (barbican-api) executes mysql statement on stale connection and reports error. There is retry logic added in mysql engine connect call but that logic is not invoked.

This kind of issue is addressed in oslo db framework but we don't use this in barbican and that's why will need to have local code to address this. May be we should look into switching to oslo db framework as it provides solution to common problems.

2016-06-01 15:49:43,631 ERROR __init__ handler Secret creation failure seen - please contact site administrator.
Traceback (most recent call last):
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/barbican/api/controllers/__init__.py", line 103, in handler
    return fn(inst, *args, **kwargs)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/barbican/api/controllers/__init__.py", line 89, in enforcer
    return fn(inst, *args, **kwargs)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/barbican/api/controllers/__init__.py", line 152, in content_types_enforcer
    return fn(inst, *args, **kwargs)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/barbican/api/controllers/secrets.py", line 334, in on_post
    project = res.get_or_create_project(external_project_id)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/barbican/common/resources.py", line 43, in get_or_create_project
    suppress_exception=True)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/barbican/model/repositories.py", line 579, in find_by_external_project_id
    entity = query.one()
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2693, in one
    ret = list(self)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
    return self._execute_and_instances(context)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/opt/stack/service/barbican-api/venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'SELECT projects.id AS projects_id, projects.created_at AS projects_created_at, projects.updated_at AS projects_updated_at, projects.deleted_at AS projects_deleted_at, projects.deleted AS projects_deleted, projects.status AS projects_status, projects.external_id AS projects_external_id \nFROM projects \nWHERE projects.external_id = %s'] [parameters: ('d1c1dc7715704dbe89cfc5549ad0d608',)]
(barbican.api.middleware.context): 2016-06-01 15:49:43,638 INFO context process_response Processed request: 500 Internal Server Error - POST https://10.246.27.6:9311/v1/secrets/

Arun Kant (arunkant-uws) wrote :

Oslo DB framework addresses this issue by listing for engine_connect event and adding ping listener for handling error.

oslo db : https://github.com/openstack/oslo.db/blob/master/oslo_db/sqlalchemy/engines.py#L149

Jeremy Liu (liujiong) wrote :

Currently, I'm encountering the same problem. Any suggestion on switching to oslo.db for barbican, kite?

Changed in barbican:
importance: Undecided → Wishlist
importance: Wishlist → High
milestone: none → pike-1
Dave McCowan (dave-mccowan) wrote :

We should switch to oslo.db in Pike

Jeremy Liu (liujiong) on 2017-03-15
Changed in barbican:
assignee: nobody → Jeremy Liu (liujiong)
status: New → In Progress

Reviewed: https://review.openstack.org/445722
Committed: https://git.openstack.org/cgit/openstack/barbican/commit/?id=314a2046006f153a5588c6002798537e21ae1803
Submitter: Jenkins
Branch: master

commit 314a2046006f153a5588c6002798537e21ae1803
Author: Jeremy Liu <email address hidden>
Date: Tue Mar 14 08:43:15 2017 -0700

    Use oslo.db create_engine instead of SQLAlchemy

    Change-Id: I327c40bcebe8d40cd515038dcf59a1926486f45e
    Closes-Bug: #1589673

Changed in barbican:
status: In Progress → Fix Released

This issue was fixed in the openstack/barbican 5.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers