keystone eventlet using session is in 'prepared' state

[Brant Knudson]

http://logs.openstack.org/58/257458/5/check/gate-tempest-dsvm-keystone-eventlet-full/2441200/logs/screen-key.txt.gz#_2016-02-24_17_26_26_453

Many operations are failing with an exception:

This session is in 'prepared' state; no further SQL can be emitted within this transaction.

Traceback (most recent call last):
  File "/opt/stack/new/keystone/keystone/common/wsgi.py", line 250, in __call__
    result = method(context, **params)
  File "/usr/local/lib/python2.7/dist-packages/oslo_log/versionutils.py", line 165, in wrapped
    return func_or_cls(*args, **kwargs)
  File "/opt/stack/new/keystone/keystone/token/controllers.py", line 100, in authenticate
    context, auth)
  File "/opt/stack/new/keystone/keystone/token/controllers.py", line 302, in _authenticate_local
    password=password)
  File "/opt/stack/new/keystone/keystone/common/manager.py", line 124, in wrapped
    __ret_val = __f(*args, **kwargs)
  File "/opt/stack/new/keystone/keystone/notifications.py", line 558, in wrapper
    result = f(wrapped_self, context, user_id, *args, **kwargs)
  File "/opt/stack/new/keystone/keystone/identity/core.py", line 425, in wrapper
    return f(self, *args, **kwargs)
  File "/opt/stack/new/keystone/keystone/identity/core.py", line 435, in wrapper
    return f(self, *args, **kwargs)
  File "/opt/stack/new/keystone/keystone/identity/core.py", line 841, in authenticate
    ref = driver.authenticate(entity_id, password)
  File "/opt/stack/new/keystone/keystone/identity/backends/sql.py", line 184, in authenticate
    user_ref = self._get_user(session, user_id)
  File "/opt/stack/new/keystone/keystone/identity/backends/sql.py", line 209, in _get_user
    user_ref = session.query(User).get(user_id)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 831, in get
    return self._get_impl(ident, loading.load_on_ident)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 864, in _get_impl
    return fallback_fn(self, key)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 219, in load_on_ident
    return q.one()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2693, in one
    ret = list(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
    return self._execute_and_instances(context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
    close_with_result=True)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
    **kw)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
    execution_options=execution_options)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
    engine, execution_options)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 313, in _connection_for_bind
    self._assert_active()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 202, in _assert_active
    "This session is in 'prepared' state; no further "
InvalidRequestError: This session is in 'prepared' state; no further SQL can be emitted within this transaction.
2016-02-24 17:26:26.458 14573 ERROR keystone.common.wsgi

Seems like this is related to https://review.openstack.org/#/c/257458/ , Use the new enginefacade from oslo.db

[Steve Martinelli (stevemar)]

see https://review.openstack.org/#/c/284965 for a possible fix

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/284965
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=9535c0908465b626b01fccec034905196f78dc1a
Submitter: Jenkins
Branch: master

commit 9535c0908465b626b01fccec034905196f78dc1a
Author: Davanum Srinivas <email address hidden>
Date: Fri Feb 26 06:16:17 2016 -0800

    Delay using threading.local() to fix check job failure

    After I221232d50821fe2adb9881f237f06714003ce79d was merged,
    the gate-tempest-dsvm-keystone-eventlet-full has been failing
    consistently. We added a threading.local() call and use
    that for the oslo.db engine facade. We need to ensure that
    monkey patching for eventlet is done before we call local().

    Closes-Bug: #1550017
    Change-Id: I363ef507e78c1c601011861ebb589c10a2bc489e

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/keystone 9.0.0.0b3 development milestone.