<<< - Next i would like to collect gmr if that can give some hint for the issue. Ok was able to reproduce and collect it in [1][2]. Also did multiple runs disabling dbcounter[3] but it didn't reproduce in the test patch. Since th e issue is random not sure if it's just a coincidence or dbcounter making the issue appear more frequently. We can disable in some jobs and see if it helps in reducing the occurrence as disabling it won't harm. Stuck Thread Traceback:- /opt/stack/new/nova/nova/api/openstack/urlmap.py:305 in __call__ `return app(environ, start_response)` /opt/stack/new/nova/nova/api/openstack/urlmap.py:202 in wrap `return app(environ, start_response)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:129 in __call__ `resp = self.call_func(req, *args, **kw)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:193 in call_func `return self.func(req, *args, **kwargs)` /usr/local/lib/python3.10/dist-packages/oslo_middleware/base.py:124 in __call__ `response = req.get_response(self.application)` /usr/local/lib/python3.10/dist-packages/webob/request.py:1313 in send `status, headers, app_iter = self.call_application(` /usr/local/lib/python3.10/dist-packages/webob/request.py:1278 in call_application `app_iter = application(self.environ, start_response)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:129 in __call__ `resp = self.call_func(req, *args, **kw)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:193 in call_func `return self.func(req, *args, **kwargs)` /usr/local/lib/python3.10/dist-packages/oslo_middleware/base.py:124 in __call__ `response = req.get_response(self.application)` /usr/local/lib/python3.10/dist-packages/webob/request.py:1313 in send `status, headers, app_iter = self.call_application(` /usr/local/lib/python3.10/dist-packages/webob/request.py:1278 in call_application `app_iter = application(self.environ, start_response)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:129 in __call__ `resp = self.call_func(req, *args, **kw)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:193 in call_func `return self.func(req, *args, **kwargs)` /usr/local/lib/python3.10/dist-packages/oslo_middleware/request_id.py:58 in __call__ `response = req.get_response(self.application)` /usr/local/lib/python3.10/dist-packages/webob/request.py:1313 in send `status, headers, app_iter = self.call_application(` /usr/local/lib/python3.10/dist-packages/webob/request.py:1278 in call_application `app_iter = application(self.environ, start_response)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:129 in __call__ `resp = self.call_func(req, *args, **kw)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:193 in call_func `return self.func(req, *args, **kwargs)` /opt/stack/new/nova/nova/api/openstack/__init__.py:95 in __call__ `return req.get_response(self.application)` /usr/local/lib/python3.10/dist-packages/webob/request.py:1313 in send `status, headers, app_iter = self.call_application(` /usr/local/lib/python3.10/dist-packages/webob/request.py:1278 in call_application `app_iter = application(self.environ, start_response)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:129 in __call__ `resp = self.call_func(req, *args, **kw)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:193 in call_func `return self.func(req, *args, **kwargs)` /opt/stack/new/nova/nova/api/openstack/requestlog.py:99 in __call__ `res = req.get_response(self.application)` /usr/local/lib/python3.10/dist-packages/webob/request.py:1313 in send `status, headers, app_iter = self.call_application(` /usr/local/lib/python3.10/dist-packages/webob/request.py:1278 in call_application `app_iter = application(self.environ, start_response)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:143 in __call__ `return resp(environ, start_response)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:129 in __call__ `resp = self.call_func(req, *args, **kw)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:193 in call_func `return self.func(req, *args, **kwargs)` /usr/local/lib/python3.10/dist-packages/osprofiler/web.py:111 in __call__ `return request.get_response(self.application)` /usr/local/lib/python3.10/dist-packages/webob/request.py:1313 in send `status, headers, app_iter = self.call_application(` /usr/local/lib/python3.10/dist-packages/webob/request.py:1278 in call_application `app_iter = application(self.environ, start_response)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:129 in __call__ `resp = self.call_func(req, *args, **kw)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:193 in call_func `return self.func(req, *args, **kwargs)` /usr/local/lib/python3.10/dist-packages/keystonemiddleware/auth_token/__init__.py:340 in __call__ `response = req.get_response(self._app)` /usr/local/lib/python3.10/dist-packages/webob/request.py:1313 in send `status, headers, app_iter = self.call_application(` /usr/local/lib/python3.10/dist-packages/webob/request.py:1278 in call_application `app_iter = application(self.environ, start_response)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:143 in __call__ `return resp(environ, start_response)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:143 in __call__ `return resp(environ, start_response)` /usr/local/lib/python3.10/dist-packages/routes/middleware.py:153 in __call__ `response = self.app(environ, start_response)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:143 in __call__ `return resp(environ, start_response)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:129 in __call__ `resp = self.call_func(req, *args, **kw)` /usr/local/lib/python3.10/dist-packages/webob/dec.py:193 in call_func `return self.func(req, *args, **kwargs)` /opt/stack/new/nova/nova/api/openstack/wsgi.py:486 in __call__ `return self._process_stack(request, action, action_args,` /opt/stack/new/nova/nova/api/openstack/wsgi.py:539 in _process_stack `action_result = self.dispatch(meth, request, action_args)` /opt/stack/new/nova/nova/api/openstack/wsgi.py:624 in dispatch `return method(req=request, **action_args)` /opt/stack/new/nova/nova/api/openstack/wsgi.py:788 in version_select `return func.func(self, *args, **kwargs)` /opt/stack/new/nova/nova/api/openstack/wsgi.py:658 in wrapped `return f(*args, **kwargs)` /opt/stack/new/nova/nova/api/openstack/compute/security_groups.py:287 in create `security_group = security_group_api.get(` /opt/stack/new/nova/nova/network/security_group_api.py:309 in get `group = neutron.show_security_group(id).get('security_group')` /opt/stack/new/nova/nova/network/neutron.py:196 in wrapper `ret = obj(*args, **kwargs)` /usr/local/lib/python3.10/dist-packages/neutronclient/v2_0/client.py:1068 in show_security_group `return self.get(self.security_group_path % (security_group),` /opt/stack/new/nova/nova/network/neutron.py:196 in wrapper `ret = obj(*args, **kwargs)` /usr/local/lib/python3.10/dist-packages/neutronclient/v2_0/client.py:356 in get `return self.retry_request("GET", action, body=body,` /opt/stack/new/nova/nova/network/neutron.py:196 in wrapper `ret = obj(*args, **kwargs)` /usr/local/lib/python3.10/dist-packages/neutronclient/v2_0/client.py:333 in retry_request `return self.do_request(method, action, body=body,` /opt/stack/new/nova/nova/network/neutron.py:196 in wrapper `ret = obj(*args, **kwargs)` /usr/local/lib/python3.10/dist-packages/neutronclient/v2_0/client.py:284 in do_request `resp, replybody = self.httpclient.do_request(action, method, body=body,` /usr/local/lib/python3.10/dist-packages/neutronclient/client.py:342 in do_request `return self.request(url, method, **kwargs)` /usr/local/lib/python3.10/dist-packages/neutronclient/client.py:330 in request `resp = super(SessionClient, self).request(*args, **kwargs)` /usr/local/lib/python3.10/dist-packages/keystoneauth1/adapter.py:257 in request `return self.session.request(url, method, **kwargs)` /usr/local/lib/python3.10/dist-packages/keystoneauth1/session.py:923 in request `resp = send(**kwargs)` /usr/local/lib/python3.10/dist-packages/keystoneauth1/session.py:1014 in _send_request `resp = self.session.request(method, url, **kwargs)` /usr/local/lib/python3.10/dist-packages/requests/sessions.py:587 in request `resp = self.send(prep, **send_kwargs)` /usr/local/lib/python3.10/dist-packages/requests/sessions.py:701 in send `r = adapter.send(request, **kwargs)` /usr/local/lib/python3.10/dist-packages/requests/adapters.py:489 in send `resp = conn.urlopen(` /usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py:692 in urlopen `conn = self._get_conn(timeout=pool_timeout)` /usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py:272 in _get_conn `if conn and is_connection_dropped(conn):` /usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py:28 in is_connection_dropped `return wait_for_read(sock, timeout=0.0)` /usr/local/lib/python3.10/dist-packages/urllib3/util/wait.py:145 in wait_for_read `return wait_for_socket(sock, read=True, timeout=timeout)` /usr/local/lib/python3.10/dist-packages/urllib3/util/wait.py:85 in select_wait_for_socket `rready, wready, xready = _retry_on_intr(fn, timeout)` /usr/local/lib/python3.10/dist-packages/urllib3/util/wait.py:43 in _retry_on_intr `return fn(timeout)` /usr/local/lib/python3.10/dist-packages/eventlet/green/select.py:80 in select `return hub.switch()` /usr/local/lib/python3.10/dist-packages/eventlet/hubs/hub.py:313 in switch `return self.greenlet.switch()` [1] https://review.opendev.org/c/openstack/neutron/+/883629 [2] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f01/883629/1/check/grenade-2/f010362/controller/logs/gmrout/osapi_compute_gurumeditation_20230519055138 [3] https://review.opendev.org/c/openstack/neutron/+/883282