"Too many connections" with MySQL + sql token driver + v3 auth

[Dolph Mathews]

Reported on the mailing list by Miller, Mark M:

  https://lists.launchpad.net/openstack/msg22307.html

Excerpt:

  (keystone.auth.controllers): ... ERROR (OperationalError) (1040, 'Too many connections') None None
  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py", line 286, in authenticate_for_token
      self.authenticate(context, auth_info, auth_context)
  ...
    File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
      super(Connection, self).__init__(*args, **kwargs2)
  OperationalError: (OperationalError) (1040, 'Too many connections') None None
  (access): ... INFO ... "POST http://15.253.58.165:35357/v3/auth/tokens HTTP/1.0" 401 125
  (eventlet.wsgi.server): ... DEBUG ... "POST /v3/auth/tokens HTTP/1.1" 401 290 0.017322

[Dolph Mathews (dolph)]

Attempted to reproduce this with PKI + sqlite against v2 does not appear to reproduce the issue. However, switching to v3 (as indicated by the above backtrace) revealed a slightly different symptom, documented in bug 1162857. These two bugs may share the same underlying cause.

[Dolph Mathews (dolph)]

I was able to reproduce this with both UUID and PKI tokens against a mysql:// sqlalchemy connection string and explicitly setting the token driver to the sql-based one (which is not the default). This bug appears to affect v3 auth, but not v2 auth.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/25833

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/25831
Committed: http://github.com/openstack/keystone/commit/a2e3a9228afe5bc0a42bb168b728caf29e394b3d
Submitter: Jenkins
Branch: master

commit a2e3a9228afe5bc0a42bb168b728caf29e394b3d
Author: Dolph Mathews <email address hidden>
Date: Mon Apr 1 14:41:26 2013 -0500

    Share one engine for more than just sqlite in-memory

    This fixes both bug 1162857 and bug 1162845.

    Change-Id: Ica5155d3ac03779693d2d54d8e02c2a413c3d3ac

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/25833
Committed: http://github.com/openstack/keystone/commit/93b8f2ddf7f00747d7ccace8401b2b68a11bf98f
Submitter: Jenkins
Branch: milestone-proposed

commit 93b8f2ddf7f00747d7ccace8401b2b68a11bf98f
Author: Dolph Mathews <email address hidden>
Date: Mon Apr 1 14:41:26 2013 -0500

    Share one engine for more than just sqlite in-memory

    This fixes both bug 1162857 and bug 1162845.

    Change-Id: Ica5155d3ac03779693d2d54d8e02c2a413c3d3ac

[Thiago Martins (martinx)]

Guys,

 I am seeing this problem on Kilo.

 What can I do? Re-open it? Or fill another bug report?

Thanks,
Thiago

[Dolph Mathews (dolph)]

Because this bug report has long been closed, it'd be best to open a new bug and reference this one as a regression.

That said, what's your SQL connection limit? We've hit kilo pretty hard and I haven't seen this sort of misbehavior, although it's possible to scale your Keystone cluster beyond SQL's default limits.