Ubuntu
nova package

[stein] maximum recursion depth exceeded while calling a Python object (File "/usr/lib/python3.6/ssl.py", line 465, in options)

Bug #1827867 reported by Dmitrii Shcherbakov on 2019-05-06

This bug report is a duplicate of: Bug #1808951: python3 + Fedora + SSL + wsgi nova deployment, nova api returns RecursionError: maximum recursion depth exceeded while calling a Python object. Edit Remove

This bug affects 5 people

Affects		Status	Importance	Assigned to	Milestone
	nova (Ubuntu)	Confirmed	Undecided	Unassigned

Bug Description

Deployed stein with 19.04 charms and tried to use OpenStack dashboard (just going through the panes). There were many errors (see the screenshot) which seem to be caused by Nova API errors.

bundle: https://paste.ubuntu.com/p/SFsftWTVzb/

Dashboard error.log:

emote 10.232.46.131:60882] Login successful for user "admin" using domain "admin_domain", remote address 10.232.46.131.
emote 10.232.46.131:60882] Recoverable error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-602a4df0-
coverable error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-af4d8ba8-33ae-4b10-8978-092e06050c07)
emote 10.232.46.131:60882] Recoverable error: local variable 'limits' referenced before assignment
emote 10.232.46.131:60882] Policies ['cloud_admin', 'owner', 'admin_or_owner', 'admin_and_matching_domain_id', 'identity:create_trust'] reference a rule that is not d
emote 10.232.46.131:33692] Recoverable error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-32ce282d-
coverable error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-e397eb7f-e7a1-4115-ba63-5313e7ecf270)
emote 10.232.46.131:33692] Recoverable error: local variable 'limits' referenced before assignment
emote 10.232.46.131:60882] Call to list supported extensions failed. This is likely due to a problem communicating with the Nova endpoint. Host Aggregates panel will
emote 10.232.46.131:33692] Policies ['cloud_admin', 'owner', 'admin_or_owner', 'admin_and_matching_domain_id', 'identity:create_trust'] reference a rule that is not d
emote 10.232.46.131:33692] Call to list supported extensions failed. This is likely due to a problem communicating with the Nova endpoint. Host Aggregates panel will
emote 10.232.46.131:35464] Recoverable error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-0de0f5ee-
emote 10.232.46.131:35464] Policies ['cloud_admin', 'owner', 'admin_or_owner', 'admin_and_matching_domain_id', 'identity:create_trust'] reference a rule that is not d
emote 10.232.46.131:35464] Call to list supported extensions failed. This is likely due to a problem communicating with the Nova endpoint. Host Aggregates panel will
emote 10.232.46.131:35464] Failed to retrieve quota information
emote 10.232.46.131:35464] Traceback (most recent call last):
emote 10.232.46.131:35464] File "/usr/lib/python3/dist-packages/django/template/base.py", line 882, in _resolve_lookup
emote 10.232.46.131:35464] current = current[bit]
emote 10.232.46.131:35464] TypeError: 'InstancesTable' object is not subscriptable

Nova API logs:

2019-05-06 00:27:36.046 1940120 INFO nova.api.openstack [req-d53c7b33-2544-4b20-8d1b-2bbde7219751 - - - - -] https://nova.maas:8774/v2.1/limits?reserved=1 returned with HTTP 500
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack [req-e9708f07-c980-44f1-801d-8fffb60f988c - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack Traceback (most recent call last):
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/nova/api/openstack/__init__.py", line 94, in __call__
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack return req.get_response(self.application)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/request.py", line 1314, in send
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack application, catch_exc_info=False)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/dec.py", line 143, in __call__
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack return resp(environ, start_response)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 330, in __call__
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack response = self.process_request(req)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 609, in process_request
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack resp = super(AuthProtocol, self).process_request(request)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 403, in process_request
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack allow_expired=allow_expired)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 435, in _do_fetch_token
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack data = self.fetch_token(token, **kwargs)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 740, in fetch_token
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack allow_expired=allow_expired)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/_identity.py", line 210, in verify_token
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack auth_ref = self._request_strategy.verify_token(
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/_identity.py", line 161, in _request_strategy
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack strategy_class = self._get_strategy_class()
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/_identity.py", line 183, in _get_strategy_class
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 271, in get_endpoint
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack return self.session.get_endpoint(auth or self.auth, **kwargs)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1139, in get_endpoint
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack return auth.get_endpoint(self, **kwargs)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack allow_version_hack=allow_version_hack, **kwargs)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack service_catalog = self.get_access(session).service_catalog
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack self.auth_ref = self.get_auth_ref(session)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack self._plugin = self._do_create_plugin(session)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack authenticated=False)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack authenticated=authenticated)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 1442, in get_discovery
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack disc = Discover(session, url, authenticated=authenticated)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 526, in __init__
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack authenticated=authenticated)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 101, in get_version_data
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack resp = session.get(url, headers=headers, authenticated=authenticated)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1037, in get
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack return self.request(url, 'GET', **kwargs)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 835, in request
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack resp = send(**kwargs)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 926, in _send_request
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack resp = self.session.request(method, url, **kwargs)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack resp = self.send(prep, **send_kwargs)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack r = adapter.send(request, **kwargs)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack timeout=timeout
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack chunked=chunked)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack self._validate_conn(conn)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 852, in _validate_conn
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack conn.connect()
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 314, in connect
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack cert_reqs=resolve_cert_reqs(self.cert_reqs),
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 269, in create_urllib3_context
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack context.options |= options
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3.6/ssl.py", line 465, in options
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3.6/ssl.py", line 465, in options
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack File "/usr/lib/python3.6/ssl.py", line 465, in options
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack [Previous line repeated 289 more times]
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object
2019-05-06 00:27:39.725 1940121 ERROR nova.api.openstack
2019-05-06 00:27:39.731 1940121 INFO nova.api.openstack [req-e9708f07-c980-44f1-801d-8fffb60f988c - - - - -] https://nova.maas:8774/v2.1/extensions returned with HTTP 500

Tags:

Revision history for this message

Dmitrii Shcherbakov (dmitriis) wrote on 2019-05-06:

06-05-2019-dashboard-errors.png Edit (241.7 KiB, image/png)

Revision history for this message

Launchpad Janitor (janitor) wrote on 2019-05-14:

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in nova (Ubuntu):
status:	New → Confirmed

Revision history for this message

Florian Guitton (f-guitton) wrote on 2019-05-16:

Hello everybody,

This problem also shows up deploying stein on 18.04LTS.

There seems to be mention of this issue in nova, cinder, and eventlet; with a commit made there:
https://github.com/openstack/nova/commit/d0f540742efc1004b4d2c64814dcc6f7b9f0ccf6
However this does not seem to be present in the code shipped in 19.0.0.

This is a blocker for us at this point.
Thank you for time in looking at this.

Best wishes,

Revision history for this message

Jamon Camisso (jamon) wrote on 2019-05-17:

I hit this as well. It looks like it is fixed over in https://bugs.launchpad.net/nova/+bug/1808951

I've tested, and at least for bionic-stein, the problem is resolved with those proposed packages.