Reproduce steps:
1. Configure keystone with ldap
2. Send concurrency request to keystone which also calling out to backend LDAP server, like executing "for i in {1,2,3,4,5}; do openstack user list --domain <ldap domain>; done" concurrently in 2 client.
Error message
Fatal Python error: saving thread twice?
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]:
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: Thread 0x00007f0ed0db4700 (most recent call first):
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/ldap/ldapobject.py", line 315 in _ldap_call
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/ldap/ldapobject.py", line 758 in result4
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/ldap/ldapobject.py", line 751 in result3
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/ldap/ldapobject.py", line 446 in simple_bind_s
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/ldap/ldapobject.py", line 1204 in _apply_method_s
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/ldap/ldapobject.py", line 1222 in simple_bind_s
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/ldappool/__init__.py", line 103 in simple_bind_s
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/ldappool/__init__.py", line 230 in _bind
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/ldappool/__init__.py", line 266 in _create_connector
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/ldappool/__init__.py", line 325 in _get_connection
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/ldappool/__init__.py", line 375 in connection
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/contextlib.py", line 112 in __enter__
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/backends/ldap/common.py", line 772 in simple_bind_s
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/backends/ldap/common.py", line 920 in simple_bind_s
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/backends/ldap/common.py", line 1261 in get_connection
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/backends/ldap/core.py", line 70 in authenticate
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/core.py", line 902 in authenticate
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/core.py", line 424 in wrapper
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/core.py", line 414 in wrapper
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/notifications.py", line 570 in wrapper
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/common/manager.py", line 116 in wrapped
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/auth/plugins/password.py", line 36 in authenticate
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/api/_shared/authentication.py", line 153 in authenticate
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/api/_shared/authentication.py", line 186 in authenticate_for_token
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/api/auth.py", line 315 in post
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/server/flask/common.py", line 1066 in wrapper
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/flask_restful/__init__.py", line 573 in dispatch_request
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/flask/views.py", line 89 in view
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/flask_restful/__init__.py", line 458 in wrapper
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/flask/app.py", line 1935 in dispatch_request
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/flask/app.py", line 1949 in full_dispatch_request
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/flask/app.py", line 2446 in wsgi_app
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/werkzeug/middleware/dispatcher.py", line 66 in __call__
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/webob/request.py", line 1278 in call_application
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/webob/request.py", line 1314 in send
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 341 in __call__
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/webob/dec.py", line 193 in call_func
Current thread 0x00007f0ecddae700 (most recent call first):
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/ldap/ldapobject.py", line 315 in _ldap_call
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/ldap/ldapobject.py", line 843 in search_ext
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/backends/ldap/common.py", line 812 in search_ext
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/backends/ldap/common.py", line 1004 in _paged_search_s
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/backends/ldap/common.py", line 952 in search_s
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/backends/ldap/common.py", line 1492 in _ldap_get
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/backends/ldap/common.py", line 1573 in get
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/backends/ldap/common.py", line 1888 in get
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/backends/ldap/core.py", line 309 in get
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/backends/ldap/core.py", line 81 in _get_user
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/backends/ldap/core.py", line 108 in list_groups_for_user
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/core.py", line 1310 in list_groups_for_user
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/core.py", line 424 in wrapper
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/identity/core.py", line 414 in wrapper
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/common/manager.py", line 116 in wrapped
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/assignment/core.py", line 88 in _get_group_ids_for_user_id
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/assignment/core.py", line 851 in _list_effective_role_assignments
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/assignment/core.py", line 996 in list_role_assignments
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/common/manager.py", line 116 in wrapped
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/assignment/core.py", line 129 in get_roles_for_user_and_project
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/dogpile/cache/region.py", line 839 in gen_value
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/dogpile/lock.py", line 179 in _enter_create
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/dogpile/lock.py", line 93 in _enter
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/dogpile/lock.py", line 186 in __enter__
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/dogpile/cache/region.py", line 879 in get_or_create
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/dogpile/cache/region.py", line 1272 in get_or_create_for_user_func
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "</usr/lib/python3.7/site-packages/decorator.py:decorator-gen-3>", line 2 in get_roles_for_user_and_project
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/common/manager.py", line 116 in wrapped
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/models/token_model.py", line 403 in _get_project_roles
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/models/token_model.py", line 441 in roles
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/models/revoke_model.py", line 245 in build_token_values
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/token/provider.py", line 135 in check_revocation_v3
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/common/manager.py", line 116 in wrapped
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/token/provider.py", line 139 in check_revocation
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/common/manager.py", line 116 in wrapped
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/token/provider.py", line 201 in _is_valid_token
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/token/provider.py", line 148 in validate_token
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/common/manager.py", line 116 in wrapped
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 256 in _extract_subject_token_target_data
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 414 in enforce_call
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystone/api/auth.py", line 285 in get
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/flask_restful/__init__.py", line 573 in dispatch_request
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/flask/views.py", line 89 in view
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/flask_restful/__init__.py", line 458 in wrapper
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/flask/app.py", line 1935 in dispatch_request
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/flask/app.py", line 1949 in full_dispatch_request
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/flask/app.py", line 2446 in wsgi_app
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/werkzeug/middleware/dispatcher.py", line 66 in __call__
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/webob/request.py", line 1278 in call_application
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/webob/request.py", line 1314 in send
Nov 06 15:55:34 controller-cf9vqh2fp4 keystone-api[858]: File "/usr/lib/python3.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 341 in __call__
