| 2018-12-18 09:26:59 |
yatin |
bug |
|
|
added bug |
| 2019-01-24 13:51:19 |
Balazs Gibizer |
tags |
|
api |
|
| 2019-01-25 20:59:29 |
Alex Schultz |
bug task added |
|
tripleo |
|
| 2019-01-25 20:59:37 |
Alex Schultz |
tripleo: status |
New |
Incomplete |
|
| 2019-01-25 20:59:39 |
Alex Schultz |
tripleo: status |
Incomplete |
Triaged |
|
| 2019-01-25 20:59:41 |
Alex Schultz |
tripleo: importance |
Undecided |
High |
|
| 2019-01-25 21:02:22 |
Alex Schultz |
tripleo: milestone |
|
stein-3 |
|
| 2019-01-30 16:40:01 |
OpenStack Infra |
nova: status |
New |
In Progress |
|
| 2019-01-30 16:40:01 |
OpenStack Infra |
nova: assignee |
|
Matthew Booth (mbooth-9) |
|
| 2019-03-14 02:32:57 |
Alex Schultz |
tripleo: milestone |
stein-3 |
stein-rc1 |
|
| 2019-03-21 14:23:47 |
Matthew Booth |
tags |
api |
api stein-rc-potential |
|
| 2019-03-22 22:31:18 |
OpenStack Infra |
nova: status |
In Progress |
Fix Released |
|
| 2019-03-22 22:31:23 |
OpenStack Infra |
bug watch added |
|
https://github.com/eventlet/eventlet/issues/371 |
|
| 2019-03-25 08:31:09 |
Martin Schuppert |
bug |
|
|
added subscriber Martin Schuppert |
| 2019-04-15 18:56:30 |
Alex Schultz |
tripleo: milestone |
stein-rc1 |
train-1 |
|
| 2019-04-25 13:21:50 |
Corey Bryant |
bug task added |
|
nova (Ubuntu) |
|
| 2019-04-25 13:22:04 |
Corey Bryant |
nominated for series |
|
Ubuntu Eoan |
|
| 2019-04-25 13:22:04 |
Corey Bryant |
bug task added |
|
nova (Ubuntu Eoan) |
|
| 2019-04-25 13:22:04 |
Corey Bryant |
nominated for series |
|
Ubuntu Disco |
|
| 2019-04-25 13:22:04 |
Corey Bryant |
bug task added |
|
nova (Ubuntu Disco) |
|
| 2019-04-25 13:22:13 |
Corey Bryant |
nova (Ubuntu Disco): status |
New |
Triaged |
|
| 2019-04-25 13:22:16 |
Corey Bryant |
nova (Ubuntu Disco): importance |
Undecided |
High |
|
| 2019-04-25 13:22:19 |
Corey Bryant |
nova (Ubuntu Eoan): status |
New |
Confirmed |
|
| 2019-04-25 13:22:21 |
Corey Bryant |
nova (Ubuntu Eoan): status |
Confirmed |
Triaged |
|
| 2019-04-25 13:22:23 |
Corey Bryant |
nova (Ubuntu Eoan): importance |
Undecided |
High |
|
| 2019-04-25 13:22:30 |
Corey Bryant |
bug task added |
|
cloud-archive |
|
| 2019-04-25 13:22:40 |
Corey Bryant |
nominated for series |
|
cloud-archive/stein |
|
| 2019-04-25 13:22:40 |
Corey Bryant |
bug task added |
|
cloud-archive/stein |
|
| 2019-04-25 13:22:48 |
Corey Bryant |
cloud-archive/stein: status |
New |
Triaged |
|
| 2019-04-25 13:22:51 |
Corey Bryant |
cloud-archive/stein: importance |
Undecided |
High |
|
| 2019-04-25 13:47:18 |
Corey Bryant |
description |
Description:-
So while testing python3 with Fedora in [1], Found an issue while running nova-api behind wsgi. It fails with below Traceback:-
2018-12-18 07:41:55.364 26870 INFO nova.api.openstack.requestlog [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] 127.0.0.1 "GET /v2.1/servers/detail?all_tenants=True&deleted=True" status: 500 len: 0 microversion: - time: 0.007297
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack Traceback (most recent call last):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/__init__.py", line 94, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._log_req(req, res, start)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.force_reraise()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack raise value
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack res = req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return resp(environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return request.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack response = self.process_request(req)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 622, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = super(AuthProtocol, self).process_request(request)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack data = self.fetch_token(token, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 761, in fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack auth_ref = self._request_strategy.verify_token(
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack strategy_class = self._get_strategy_class()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.session.get_endpoint(auth or self.auth, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return auth.get_endpoint(self, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_version_hack=allow_version_hack, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack service_catalog = self.get_access(session).service_catalog
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.auth_ref = self.get_auth_ref(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._plugin = self._do_create_plugin(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack disc = Discover(session, url, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = session.get(url, headers=headers, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.request(url, 'GET', **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = send(**kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.session.request(method, url, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.send(prep, **send_kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack r = adapter.send(request, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack timeout=timeout
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack chunked=chunked)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._validate_conn(conn)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack conn.connect()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack cert_reqs=resolve_cert_reqs(self.cert_reqs),
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack context.options |= options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [Previous line repeated 280 more times]
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack
2018-12-18 07:41:55.371 26870 INFO nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] https://127.0.0.1:8774/v2.1/servers/detail?all_tenants=True&deleted=True returned with HTTP 500
Environment Details:-
OS: Fedora 28
python: 3.6.7
eventlet: 0.24.1
The issue is around using of eventlet.monkey_patch even if running under wsgi[2]. Eventlet Best practices suggest to not run(skip) monkey_patch when running behind wsgi[3] but with nova it's not the case.
Expected result
===============
Nova api should response successfully.
Actual result
=============
Nova api returns Recursion Error.
[1] https://review.openstack.org/#/c/620561/
[2] https://github.com/openstack/nova/blob/a674241d0f026b546b38ec77ab472e916805c679/nova/api/openstack/wsgi_app.py#L32
[3] https://review.openstack.org/#/c/154642/2/specs/eventlet-best-practices.rst@54 |
Description:-
So while testing python3 with Fedora in [1], Found an issue while running nova-api behind wsgi. It fails with below Traceback:-
2018-12-18 07:41:55.364 26870 INFO nova.api.openstack.requestlog [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] 127.0.0.1 "GET /v2.1/servers/detail?all_tenants=True&deleted=True" status: 500 len: 0 microversion: - time: 0.007297
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack Traceback (most recent call last):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/__init__.py", line 94, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._log_req(req, res, start)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.force_reraise()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack raise value
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack res = req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return resp(environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return request.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack response = self.process_request(req)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 622, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = super(AuthProtocol, self).process_request(request)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack data = self.fetch_token(token, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 761, in fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack auth_ref = self._request_strategy.verify_token(
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack strategy_class = self._get_strategy_class()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.session.get_endpoint(auth or self.auth, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return auth.get_endpoint(self, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_version_hack=allow_version_hack, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack service_catalog = self.get_access(session).service_catalog
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.auth_ref = self.get_auth_ref(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._plugin = self._do_create_plugin(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack disc = Discover(session, url, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = session.get(url, headers=headers, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.request(url, 'GET', **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = send(**kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.session.request(method, url, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.send(prep, **send_kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack r = adapter.send(request, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack timeout=timeout
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack chunked=chunked)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._validate_conn(conn)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack conn.connect()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack cert_reqs=resolve_cert_reqs(self.cert_reqs),
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack context.options |= options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [Previous line repeated 280 more times]
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack
2018-12-18 07:41:55.371 26870 INFO nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] https://127.0.0.1:8774/v2.1/servers/detail?all_tenants=True&deleted=True returned with HTTP 500
Environment Details:-
OS: Fedora 28
python: 3.6.7
eventlet: 0.24.1
The issue is around using of eventlet.monkey_patch even if running under wsgi[2]. Eventlet Best practices suggest to not run(skip) monkey_patch when running behind wsgi[3] but with nova it's not the case.
Expected result
===============
Nova api should response successfully.
Actual result
=============
Nova api returns Recursion Error.
[1] https://review.openstack.org/#/c/620561/
[2] https://github.com/openstack/nova/blob/a674241d0f026b546b38ec77ab472e916805c679/nova/api/openstack/wsgi_app.py#L32
[3] https://review.openstack.org/#/c/154642/2/specs/eventlet-best-practices.rst@54
-------------------------------------------------------------
Ubuntu SRU Details
[Impact]
Please see description above.
[Test Case]
[Regression Potential]
There is certainly regression potential with this patch as it is not a trivial change. The patch has landed in the upstream master branch which is now train, which means it has received iterative reviews and final approval from upstream reviewers and passed gate testing. The patch has not yet landed in the upstream stable/stein but does have one core reviewer's +2 and has passed zuul tests with a +1. The patch has been tested locally and confirmed to fix nova+wsgi+py3+SSL in initial Ubuntu testing. Without this change that combination is broken for Ubuntu Stein. |
|
| 2019-04-25 14:28:36 |
Liam Young |
description |
Description:-
So while testing python3 with Fedora in [1], Found an issue while running nova-api behind wsgi. It fails with below Traceback:-
2018-12-18 07:41:55.364 26870 INFO nova.api.openstack.requestlog [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] 127.0.0.1 "GET /v2.1/servers/detail?all_tenants=True&deleted=True" status: 500 len: 0 microversion: - time: 0.007297
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack Traceback (most recent call last):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/__init__.py", line 94, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._log_req(req, res, start)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.force_reraise()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack raise value
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack res = req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return resp(environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return request.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack response = self.process_request(req)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 622, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = super(AuthProtocol, self).process_request(request)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack data = self.fetch_token(token, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 761, in fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack auth_ref = self._request_strategy.verify_token(
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack strategy_class = self._get_strategy_class()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.session.get_endpoint(auth or self.auth, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return auth.get_endpoint(self, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_version_hack=allow_version_hack, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack service_catalog = self.get_access(session).service_catalog
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.auth_ref = self.get_auth_ref(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._plugin = self._do_create_plugin(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack disc = Discover(session, url, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = session.get(url, headers=headers, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.request(url, 'GET', **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = send(**kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.session.request(method, url, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.send(prep, **send_kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack r = adapter.send(request, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack timeout=timeout
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack chunked=chunked)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._validate_conn(conn)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack conn.connect()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack cert_reqs=resolve_cert_reqs(self.cert_reqs),
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack context.options |= options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [Previous line repeated 280 more times]
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack
2018-12-18 07:41:55.371 26870 INFO nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] https://127.0.0.1:8774/v2.1/servers/detail?all_tenants=True&deleted=True returned with HTTP 500
Environment Details:-
OS: Fedora 28
python: 3.6.7
eventlet: 0.24.1
The issue is around using of eventlet.monkey_patch even if running under wsgi[2]. Eventlet Best practices suggest to not run(skip) monkey_patch when running behind wsgi[3] but with nova it's not the case.
Expected result
===============
Nova api should response successfully.
Actual result
=============
Nova api returns Recursion Error.
[1] https://review.openstack.org/#/c/620561/
[2] https://github.com/openstack/nova/blob/a674241d0f026b546b38ec77ab472e916805c679/nova/api/openstack/wsgi_app.py#L32
[3] https://review.openstack.org/#/c/154642/2/specs/eventlet-best-practices.rst@54
-------------------------------------------------------------
Ubuntu SRU Details
[Impact]
Please see description above.
[Test Case]
[Regression Potential]
There is certainly regression potential with this patch as it is not a trivial change. The patch has landed in the upstream master branch which is now train, which means it has received iterative reviews and final approval from upstream reviewers and passed gate testing. The patch has not yet landed in the upstream stable/stein but does have one core reviewer's +2 and has passed zuul tests with a +1. The patch has been tested locally and confirmed to fix nova+wsgi+py3+SSL in initial Ubuntu testing. Without this change that combination is broken for Ubuntu Stein. |
Description:-
So while testing python3 with Fedora in [1], Found an issue while running nova-api behind wsgi. It fails with below Traceback:-
2018-12-18 07:41:55.364 26870 INFO nova.api.openstack.requestlog [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] 127.0.0.1 "GET /v2.1/servers/detail?all_tenants=True&deleted=True" status: 500 len: 0 microversion: - time: 0.007297
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack Traceback (most recent call last):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/__init__.py", line 94, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._log_req(req, res, start)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.force_reraise()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack raise value
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack res = req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return resp(environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return request.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack response = self.process_request(req)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 622, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = super(AuthProtocol, self).process_request(request)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack data = self.fetch_token(token, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 761, in fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack auth_ref = self._request_strategy.verify_token(
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack strategy_class = self._get_strategy_class()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.session.get_endpoint(auth or self.auth, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return auth.get_endpoint(self, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_version_hack=allow_version_hack, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack service_catalog = self.get_access(session).service_catalog
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.auth_ref = self.get_auth_ref(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._plugin = self._do_create_plugin(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack disc = Discover(session, url, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = session.get(url, headers=headers, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.request(url, 'GET', **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = send(**kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.session.request(method, url, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.send(prep, **send_kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack r = adapter.send(request, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack timeout=timeout
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack chunked=chunked)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._validate_conn(conn)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack conn.connect()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack cert_reqs=resolve_cert_reqs(self.cert_reqs),
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack context.options |= options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [Previous line repeated 280 more times]
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack
2018-12-18 07:41:55.371 26870 INFO nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] https://127.0.0.1:8774/v2.1/servers/detail?all_tenants=True&deleted=True returned with HTTP 500
Environment Details:-
OS: Fedora 28
python: 3.6.7
eventlet: 0.24.1
The issue is around using of eventlet.monkey_patch even if running under wsgi[2]. Eventlet Best practices suggest to not run(skip) monkey_patch when running behind wsgi[3] but with nova it's not the case.
Expected result
===============
Nova api should response successfully.
Actual result
=============
Nova api returns Recursion Error.
[1] https://review.openstack.org/#/c/620561/
[2] https://github.com/openstack/nova/blob/a674241d0f026b546b38ec77ab472e916805c679/nova/api/openstack/wsgi_app.py#L32
[3] https://review.openstack.org/#/c/154642/2/specs/eventlet-best-practices.rst@54
-------------------------------------------------------------
Ubuntu SRU Details
[Impact]
Please see description above.
[Test Case]
[Test Case]
* Deploy OpenStack Stein with tls endpoints
* Attempt to interact with nova api (list flavors for example)
https://paste.ubuntu.com/p/yFPsMqRVRx/
[Regression Potential]
There is certainly regression potential with this patch as it is not a trivial change. The patch has landed in the upstream master branch which is now train, which means it has received iterative reviews and final approval from upstream reviewers and passed gate testing. The patch has not yet landed in the upstream stable/stein but does have one core reviewer's +2 and has passed zuul tests with a +1. The patch has been tested locally and confirmed to fix nova+wsgi+py3+SSL in initial Ubuntu testing. Without this change that combination is broken for Ubuntu Stein. |
|
| 2019-04-25 14:29:59 |
Liam Young |
description |
Description:-
So while testing python3 with Fedora in [1], Found an issue while running nova-api behind wsgi. It fails with below Traceback:-
2018-12-18 07:41:55.364 26870 INFO nova.api.openstack.requestlog [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] 127.0.0.1 "GET /v2.1/servers/detail?all_tenants=True&deleted=True" status: 500 len: 0 microversion: - time: 0.007297
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack Traceback (most recent call last):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/__init__.py", line 94, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._log_req(req, res, start)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.force_reraise()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack raise value
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack res = req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return resp(environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return request.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack response = self.process_request(req)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 622, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = super(AuthProtocol, self).process_request(request)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack data = self.fetch_token(token, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 761, in fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack auth_ref = self._request_strategy.verify_token(
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack strategy_class = self._get_strategy_class()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.session.get_endpoint(auth or self.auth, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return auth.get_endpoint(self, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_version_hack=allow_version_hack, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack service_catalog = self.get_access(session).service_catalog
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.auth_ref = self.get_auth_ref(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._plugin = self._do_create_plugin(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack disc = Discover(session, url, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = session.get(url, headers=headers, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.request(url, 'GET', **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = send(**kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.session.request(method, url, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.send(prep, **send_kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack r = adapter.send(request, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack timeout=timeout
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack chunked=chunked)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._validate_conn(conn)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack conn.connect()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack cert_reqs=resolve_cert_reqs(self.cert_reqs),
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack context.options |= options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [Previous line repeated 280 more times]
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack
2018-12-18 07:41:55.371 26870 INFO nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] https://127.0.0.1:8774/v2.1/servers/detail?all_tenants=True&deleted=True returned with HTTP 500
Environment Details:-
OS: Fedora 28
python: 3.6.7
eventlet: 0.24.1
The issue is around using of eventlet.monkey_patch even if running under wsgi[2]. Eventlet Best practices suggest to not run(skip) monkey_patch when running behind wsgi[3] but with nova it's not the case.
Expected result
===============
Nova api should response successfully.
Actual result
=============
Nova api returns Recursion Error.
[1] https://review.openstack.org/#/c/620561/
[2] https://github.com/openstack/nova/blob/a674241d0f026b546b38ec77ab472e916805c679/nova/api/openstack/wsgi_app.py#L32
[3] https://review.openstack.org/#/c/154642/2/specs/eventlet-best-practices.rst@54
-------------------------------------------------------------
Ubuntu SRU Details
[Impact]
Please see description above.
[Test Case]
[Test Case]
* Deploy OpenStack Stein with tls endpoints
* Attempt to interact with nova api (list flavors for example)
https://paste.ubuntu.com/p/yFPsMqRVRx/
[Regression Potential]
There is certainly regression potential with this patch as it is not a trivial change. The patch has landed in the upstream master branch which is now train, which means it has received iterative reviews and final approval from upstream reviewers and passed gate testing. The patch has not yet landed in the upstream stable/stein but does have one core reviewer's +2 and has passed zuul tests with a +1. The patch has been tested locally and confirmed to fix nova+wsgi+py3+SSL in initial Ubuntu testing. Without this change that combination is broken for Ubuntu Stein. |
Description:-
So while testing python3 with Fedora in [1], Found an issue while running nova-api behind wsgi. It fails with below Traceback:-
2018-12-18 07:41:55.364 26870 INFO nova.api.openstack.requestlog [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] 127.0.0.1 "GET /v2.1/servers/detail?all_tenants=True&deleted=True" status: 500 len: 0 microversion: - time: 0.007297
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack Traceback (most recent call last):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/__init__.py", line 94, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._log_req(req, res, start)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.force_reraise()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack raise value
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack res = req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return resp(environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return request.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack response = self.process_request(req)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 622, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = super(AuthProtocol, self).process_request(request)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack data = self.fetch_token(token, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 761, in fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack auth_ref = self._request_strategy.verify_token(
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack strategy_class = self._get_strategy_class()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.session.get_endpoint(auth or self.auth, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return auth.get_endpoint(self, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_version_hack=allow_version_hack, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack service_catalog = self.get_access(session).service_catalog
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.auth_ref = self.get_auth_ref(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._plugin = self._do_create_plugin(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack disc = Discover(session, url, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = session.get(url, headers=headers, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.request(url, 'GET', **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = send(**kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.session.request(method, url, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.send(prep, **send_kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack r = adapter.send(request, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack timeout=timeout
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack chunked=chunked)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._validate_conn(conn)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack conn.connect()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack cert_reqs=resolve_cert_reqs(self.cert_reqs),
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack context.options |= options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [Previous line repeated 280 more times]
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack
2018-12-18 07:41:55.371 26870 INFO nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] https://127.0.0.1:8774/v2.1/servers/detail?all_tenants=True&deleted=True returned with HTTP 500
Environment Details:-
OS: Fedora 28
python: 3.6.7
eventlet: 0.24.1
The issue is around using of eventlet.monkey_patch even if running under wsgi[2]. Eventlet Best practices suggest to not run(skip) monkey_patch when running behind wsgi[3] but with nova it's not the case.
Expected result
===============
Nova api should response successfully.
Actual result
=============
Nova api returns Recursion Error.
[1] https://review.openstack.org/#/c/620561/
[2] https://github.com/openstack/nova/blob/a674241d0f026b546b38ec77ab472e916805c679/nova/api/openstack/wsgi_app.py#L32
[3] https://review.openstack.org/#/c/154642/2/specs/eventlet-best-practices.rst@54
-------------------------------------------------------------
Ubuntu SRU Details
[Impact]
Please see description above.
[Test Case]
* Deploy OpenStack Stein with tls endpoints
* Attempt to interact with nova api (list flavors for example)
https://paste.ubuntu.com/p/yFPsMqRVRx/
[Regression Potential]
There is certainly regression potential with this patch as it is not a trivial change. The patch has landed in the upstream master branch which is now train, which means it has received iterative reviews and final approval from upstream reviewers and passed gate testing. The patch has not yet landed in the upstream stable/stein but does have one core reviewer's +2 and has passed zuul tests with a +1. The patch has been tested locally and confirmed to fix nova+wsgi+py3+SSL in initial Ubuntu testing. Without this change that combination is broken for Ubuntu Stein. |
|
| 2019-04-25 19:12:55 |
Corey Bryant |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-04-26 15:20:56 |
Launchpad Janitor |
nova (Ubuntu Eoan): status |
Triaged |
Fix Released |
|
| 2019-05-04 22:49:42 |
OpenStack Infra |
cloud-archive/stein: status |
Triaged |
Fix Committed |
|
| 2019-05-08 19:28:21 |
Brian Murray |
nova (Ubuntu Disco): status |
Triaged |
Fix Committed |
|
| 2019-05-08 19:28:26 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2019-05-08 19:28:33 |
Brian Murray |
tags |
api stein-rc-potential |
api stein-rc-potential verification-needed verification-needed-disco |
|
| 2019-05-09 12:04:20 |
Corey Bryant |
tags |
api stein-rc-potential verification-needed verification-needed-disco |
api stein-rc-potential verification-needed verification-needed-disco verification-stein-needed |
|
| 2019-05-17 20:42:38 |
Jamon Camisso |
bug |
|
|
added subscriber Jamon Camisso |
| 2019-05-23 07:58:04 |
Uladzimir Zenkevich |
bug |
|
|
added subscriber Uladzimir Zenkevich |
| 2019-06-07 19:57:14 |
Alex Schultz |
tripleo: milestone |
train-1 |
train-2 |
|
| 2019-06-11 14:15:42 |
Corey Bryant |
tags |
api stein-rc-potential verification-needed verification-needed-disco verification-stein-needed |
api py3 stein-rc-potential verification-needed verification-needed-disco verification-stein-needed |
|
| 2019-06-17 20:05:23 |
David Coronel |
tags |
api py3 stein-rc-potential verification-needed verification-needed-disco verification-stein-needed |
api py3 stein-rc-potential verification-needed verification-needed-disco verification-stein-done |
|
| 2019-06-19 17:50:14 |
David Coronel |
bug |
|
|
added subscriber Canonical Field Critical |
| 2019-06-19 17:54:39 |
David Coronel |
bug |
|
|
added subscriber David Coronel |
| 2019-06-19 18:04:47 |
Dean Henrichsmeyer |
removed subscriber Canonical Field Critical |
|
|
|
| 2019-06-27 12:50:41 |
Sahid Orentino |
tags |
api py3 stein-rc-potential verification-needed verification-needed-disco verification-stein-done |
api py3 stein-rc-potential verification-disco-done verification-needed verification-stein-done |
|
| 2019-06-28 13:29:41 |
Przemyslaw Hausman |
bug |
|
|
added subscriber Przemyslaw Hausman |
| 2019-07-01 13:44:43 |
James Page |
tags |
api py3 stein-rc-potential verification-disco-done verification-needed verification-stein-done |
api py3 stein-rc-potential verification-disco-done verification-done verification-stein-done |
|
| 2019-07-01 19:08:06 |
James Troup |
bug |
|
|
added subscriber Canonical Field Medium |
| 2019-07-02 16:17:43 |
Sahid Orentino |
tags |
api py3 stein-rc-potential verification-disco-done verification-done verification-stein-done |
api py3 stein-rc-potential verification-disco-needed verification-needed verification-stein-done |
|
| 2019-07-03 09:19:10 |
Sahid Orentino |
attachment added |
|
catalog.txt https://bugs.launchpad.net/nova/+bug/1808951/+attachment/5274823/+files/catalog.txt |
|
| 2019-07-03 09:19:26 |
Sahid Orentino |
attachment added |
|
flavor.txt https://bugs.launchpad.net/nova/+bug/1808951/+attachment/5274824/+files/flavor.txt |
|
| 2019-07-03 09:19:42 |
Sahid Orentino |
attachment added |
|
server.txt https://bugs.launchpad.net/nova/+bug/1808951/+attachment/5274825/+files/server.txt |
|
| 2019-07-03 09:20:19 |
Sahid Orentino |
tags |
api py3 stein-rc-potential verification-disco-needed verification-needed verification-stein-done |
api py3 stein-rc-potential verification-disco-done verification-done verification-stein-done |
|
| 2019-07-03 18:08:23 |
Brian Murray |
tags |
api py3 stein-rc-potential verification-disco-done verification-done verification-stein-done |
api py3 stein-rc-potential verification-done verification-done-disco verification-stein-done |
|
| 2019-07-03 18:10:55 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2019-07-03 18:20:27 |
Launchpad Janitor |
nova (Ubuntu Disco): status |
Fix Committed |
Fix Released |
|
| 2019-07-04 04:09:51 |
James Page |
cloud-archive/stein: status |
Fix Committed |
Fix Released |
|
| 2019-07-29 14:17:22 |
Alex Schultz |
tripleo: milestone |
train-2 |
train-3 |
|
| 2019-09-11 21:40:57 |
Alex Schultz |
tripleo: milestone |
train-3 |
ussuri-1 |
|
| 2019-12-19 15:01:46 |
Emilien Macchi |
tripleo: milestone |
ussuri-1 |
ussuri-2 |
|
| 2020-02-10 20:45:06 |
wes hayutin |
tripleo: milestone |
ussuri-2 |
ussuri-3 |
|
| 2020-04-07 04:42:18 |
wes hayutin |
tripleo: status |
Triaged |
Incomplete |
|
| 2020-04-13 17:47:48 |
wes hayutin |
tripleo: milestone |
ussuri-3 |
ussuri-rc3 |
|
| 2020-05-26 20:48:01 |
wes hayutin |
tripleo: milestone |
ussuri-rc3 |
victoria-1 |
|
| 2020-07-28 12:31:36 |
Emilien Macchi |
tripleo: milestone |
victoria-1 |
victoria-3 |
|
| 2021-07-03 01:54:14 |
Billy Olsen |
removed subscriber Canonical Field Medium |
|
|
|
