[ldappool] Downing one of the configured LDAP servers causes a persistent failure

[Grzegorz Grasza]

If a server disconnects after a pooled connection is created, it fails withe an ldap.TIMEOUT which is Unhandled.

The ReconnectLDAPObject used by ldappool only catches the ldap.SERVER_DOWN exception in _apply_method_s which is applied to synchronous methods, whereas ldap.TIMEOUT is properly caught only during the initial connection.

To test this I did the following:

1. Create a new interface with a new IP address for the ldap server

2. set this as the first server in the url list and set pool_connection_timeout in the domain [ldap] configuration

3. Do a couple of:

  time openstack user list --domain=Users

to fill the connections in the pools in all WSGI processes

4. remove the created IP address from the interface

5. try an openstack user list again

Result:

the pooled connection consistently fails with:

gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: CRITICAL keystone [None req-7ad1997a-e23e-436b-a9ad-72f68a52abf8 demo admin] Unhandled error: ldap.TIMEOUT
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone Traceback (most recent call last):
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2091, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.wsgi_app(environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/werkzeug/middleware/proxy_fix.py", line 187, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.app(environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = self.call_func(req, *args, **kw)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.func(req, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/oslo_middleware/base.py", line 124, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone response = req.get_response(self.application)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone status, headers, app_iter = self.call_application(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone app_iter = application(self.environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 143, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return resp(environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = self.call_func(req, *args, **kw)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.func(req, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/oslo_middleware/base.py", line 124, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone response = req.get_response(self.application)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone status, headers, app_iter = self.call_application(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone app_iter = application(self.environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = self.call_func(req, *args, **kw)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.func(req, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/osprofiler/web.py", line 112, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return request.get_response(self.application)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone status, headers, app_iter = self.call_application(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone app_iter = application(self.environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = self.call_func(req, *args, **kw)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.func(req, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/oslo_middleware/request_id.py", line 58, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone response = req.get_response(self.application)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone status, headers, app_iter = self.call_application(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone app_iter = application(self.environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py",[50/1978]
 in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.app(environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = self.call_func(req, *args, **kw)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.func(req, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __c
all__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone response = req.get_response(self._app)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone status, headers, app_iter = self.call_application(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone app_iter = application(self.environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/werkzeug/middleware/dispatcher.py", line 78, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return app(environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2076, in wsgi_app
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone response = self.handle_exception(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone [Previous line repeated 27 more times]
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2073, in wsgi_app
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone response = self.full_dispatch_request()
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1518, in full_dispatch_request
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone rv = self.handle_user_exception(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone [Previous line repeated 27 more times]
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1516, in full_dispatch_request
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone rv = self.dispatch_request()
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1502, in dispatch_request
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 467, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = resource(*args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/views.py", line 84, in view
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = meth(*args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/api/users.py", line 183, in get
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self._list_users()
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/api/users.py", line 214, in _list_users
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone refs = PROVIDERS.identity_api.list_users(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone __ret_val = __f(*args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return f(self, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return f(self, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 1108, in list_users
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone ref_list = self._handle_shadow_and_local_users(driver, hints)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 1091, in _handle_shadow_and_local_users
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return driver.list_users(hints) + fed_res
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 85, in list_users
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.user.get_all_filtered(hints)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 328, in get_all_filtered
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone for user in self.get_all(query, hints)]
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 319, in get_all
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone objs = super(UserApi, self).get_all(ldap_filter=ldap_filter,
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 1934, in get_all
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return super(EnabledEmuMixIn, self).get_all(ldap_filter, hints)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 1622, in get_all
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone for x in self._ldap_get_all(hints, ldap_filter)]
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/common/driver_hints.py", line 42, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return f(self, hints, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 1572, in _ldap_get_all
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone res = conn.search_s(self.tree_dn,
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 985, in search_s
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone ldap_result = self.conn.search_s(base, scope, filterstr,
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 679, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return func(self, conn, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 813, in search_s
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return conn.search_s(base, scope, filterstr, attrlist,
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 631, in search_s
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 1030, in search_ext_s
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 968, in _apply_method_s
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return func(self,*args,**kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 625, in search_ext_s
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.result(msgid,all=1,timeout=timeout)[1]
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 535, in result
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp_type, resp_data, resp_msgid = self.result2(msgid,all,timeout)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 539, in result2
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all,timeout)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 543, in result3
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp_type, resp_data, resp_msgid, decoded_resp_ctrls, retoid, retval = self.result4(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 553, in result4
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 128, in _ldap_call
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone result = func(*args,**kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone ldap.TIMEOUT
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone

[Grzegorz Grasza (xek)]

Also added an issue to python-ldap https://github.com/python-ldap/python-ldap/issues/446

[Sven Kieske (s-kieske)]

Keystone should imho fail gracefully and retry indefinitely if one backend goes down for whatever reason.

Most backend failures are temporary, e.g. during maintenance windows.

[Sven Kieske (s-kieske)]

if I read this correctly:

> There is another issue, which is triggered, when re-using a connection after a ldap.SERVER_DOWN connection. Then the reconnection does not apply as well[..]

https://github.com/python-ldap/python-ldap/pull/267#issuecomment-520436367

a workaround for keystone might be, to use a new connection and not reuse an existing connection, when encountering ldap.SERVER_DOWN.

I don't know if it is feasible though, as I'm not familiar with the keystone codebase.

[Sven Kieske (s-kieske)]

Could at least the logic get changed how keystone uses python-ldap?

Usecase:

I have 4 LDAP Servers configured

3 are down for maintenance.

I get alerts with stacktraces like:

2022-08-24 04:06:44.860 7 ERROR keystonemiddleware.auth_token [req-b4703136-ac14-44d5-bc85-665d852cb41e - c512d8855ab44fd7af3d2ad10583d6e4 - 1831760463a0498e8b0b006eff434cf8 1831760463a0498e8b0b006eff434cf8] Bad response code while vali
dating token: 504 Gateway Timeout (HTTP 504): keystoneauth1.exceptions.http.GatewayTimeout: Gateway Timeout (HTTP 504)
2022-08-24 04:06:44.860 7 WARNING keystonemiddleware.auth_token [req-b4703136-ac14-44d5-bc85-665d852cb41e - c512d8855ab44fd7af3d2ad10583d6e4 - 1831760463a0498e8b0b006eff434cf8 1831760463a0498e8b0b006eff434cf8] Identity response: <html><
body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
: keystoneauth1.exceptions.http.GatewayTimeout: Gateway Timeout (HTTP 504)
2022-08-24 04:06:44.860 7 CRITICAL keystonemiddleware.auth_token [req-b4703136-ac14-44d5-bc85-665d852cb41e - c512d8855ab44fd7af3d2ad10583d6e4 - 1831760463a0498e8b0b006eff434cf8 1831760463a0498e8b0b006eff434cf8] Unable to validate token:
 Failed to fetch token data from identity server: keystonemiddleware.auth_token._exceptions.ServiceError: Failed to fetch token data from identity server

in octavia.

I get this in keystone.log:

2022-08-24 04:07:02.711 31 ERROR ldappool Traceback (most recent call last):
2022-08-24 04:07:02.711 31 ERROR ldappool File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1196, in _apply_method_s
2022-08-24 04:07:02.711 31 ERROR ldappool return func(self,*args,**kwargs)
2022-08-24 04:07:02.711 31 ERROR ldappool File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 443, in simple_bind_s
2022-08-24 04:07:02.711 31 ERROR ldappool msgid = self.simple_bind(who,cred,serverctrls,clientctrls)
2022-08-24 04:07:02.711 31 ERROR ldappool File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 437, in simple_bind
2022-08-24 04:07:02.711 31 ERROR ldappool return self._ldap_call(self._l.simple_bind,who,cred,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls))
2022-08-24 04:07:02.711 31 ERROR ldappool File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 329, in _ldap_call
2022-08-24 04:07:02.711 31 ERROR ldappool reraise(exc_type, exc_value, exc_traceback)
2022-08-24 04:07:02.711 31 ERROR ldappool File "/usr/lib/python3/dist-packages/ldap/compat.py", line 44, in reraise
2022-08-24 04:07:02.711 31 ERROR ldappool raise exc_value
2022-08-24 04:07:02.711 31 ERROR ldappool File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 313, in _ldap_call
2022-08-24 04:07:02.711 31 ERROR ldappool result = func(*args,**kwargs)
2022-08-24 04:07:02.711 31 ERROR ldappool ldap.SERVER_DOWN: {'desc': "Can't contact LDAP server", 'errno': 107, 'info': 'Transport endpoint is not connected'}
2022-08-24 04:07:02.711 31 ERROR ldappool
2022-08-24 04:07:02.711 31 ERROR ldappool During handling of the above exception, another exception occurred:
2022-08-24 04:07:02.711 31 ERROR ldappool
2022-08-24 04:07:02.711 31 ERROR ldappool Traceback (most recent...

[Grzegorz Grasza (xek)]

Looks like this change is related:
https://review.opendev.org/c/openstack/keystone/+/860118

[Sven Kieske (s-kieske)]

so I think I fixed the related change, hopefully someone can look at it soon. I can't remove the "WIP" label though, as I'm not the change owner.