python3 + Fedora + SSL + wsgi nova deployment, nova api returns RecursionError: maximum recursion depth exceeded while calling a Python object

[yatin]

Description:-

So while testing python3 with Fedora in [1], Found an issue while running nova-api behind wsgi. It fails with below Traceback:-

2018-12-18 07:41:55.364 26870 INFO nova.api.openstack.requestlog [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] 127.0.0.1 "GET /v2.1/servers/detail?all_tenants=True&deleted=True" status: 500 len: 0 microversion: - time: 0.007297
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack Traceback (most recent call last):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/__init__.py", line 94, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._log_req(req, res, start)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.force_reraise()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack raise value
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack res = req.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return resp(environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return request.get_response(self.application)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack response = self.process_request(req)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 622, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = super(AuthProtocol, self).process_request(request)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack data = self.fetch_token(token, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 761, in fetch_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack auth_ref = self._request_strategy.verify_token(
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack strategy_class = self._get_strategy_class()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.session.get_endpoint(auth or self.auth, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return auth.get_endpoint(self, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_version_hack=allow_version_hack, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack service_catalog = self.get_access(session).service_catalog
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.auth_ref = self.get_auth_ref(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._plugin = self._do_create_plugin(session)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=False)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack disc = Discover(session, url, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = session.get(url, headers=headers, authenticated=authenticated)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.request(url, 'GET', **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = send(**kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.session.request(method, url, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.send(prep, **send_kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack r = adapter.send(request, **kwargs)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack timeout=timeout
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack chunked=chunked)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._validate_conn(conn)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack conn.connect()
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack cert_reqs=resolve_cert_reqs(self.cert_reqs),
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack context.options |= options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [Previous line repeated 280 more times]
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack
2018-12-18 07:41:55.371 26870 INFO nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] https://127.0.0.1:8774/v2.1/servers/detail?all_tenants=True&deleted=True returned with HTTP 500

Environment Details:-
OS: Fedora 28
python: 3.6.7
eventlet: 0.24.1

The issue is around using of eventlet.monkey_patch even if running under wsgi[2]. Eventlet Best practices suggest to not run(skip) monkey_patch when running behind wsgi[3] but with nova it's not the case.

Expected result
===============
Nova api should response successfully.

Actual result
=============
Nova api returns Recursion Error.

[1] https://review.openstack.org/#/c/620561/
[2] https://github.com/openstack/nova/blob/a674241d0f026b546b38ec77ab472e916805c679/nova/api/openstack/wsgi_app.py#L32
[3] https://review.openstack.org/#/c/154642/2/specs/eventlet-best-practices.rst@54

-------------------------------------------------------------

Ubuntu SRU Details

[Impact]
Please see description above.

[Test Case]
* Deploy OpenStack Stein with tls endpoints
* Attempt to interact with nova api (list flavors for example)
https://paste.ubuntu.com/p/yFPsMqRVRx/

[Regression Potential]
There is certainly regression potential with this patch as it is not a trivial change. The patch has landed in the upstream master branch which is now train, which means it has received iterative reviews and final approval from upstream reviewers and passed gate testing. The patch has not yet landed in the upstream stable/stein but does have one core reviewer's +2 and has passed zuul tests with a +1. The patch has been tested locally and confirmed to fix nova+wsgi+py3+SSL in initial Ubuntu testing. Without this change that combination is broken for Ubuntu Stein.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/626952

[Alex Schultz (alex-schultz)]

Adding tripleo because this is affecting our fedora28 containers when deployed via an undercloud with ssl enabled.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/626952
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3c5e2b0e9fac985294a949852bb8c83d4ed77e04
Submitter: Zuul
Branch: master

commit 3c5e2b0e9fac985294a949852bb8c83d4ed77e04
Author: Matthew Booth <email address hidden>
Date: Wed Jan 30 15:10:25 2019 +0000

    Eventlet monkey patching should be as early as possible

    We were seeing infinite recursion opening an ssl socket when running
    various combinations of python3, eventlet, and urllib3. It is not
    clear exactly what combination of versions are affected, but for
    background there is an example of this issue documented here:

    https://github.com/eventlet/eventlet/issues/371

    The immediate cause in nova's case was that we were calling
    eventlet.monkey_patch() after importing urllib3. Specifically, change
    Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 introduced the
    nova.utils.monkey_patch() method to make monkey patching common
    between WSGI and non-WSGI services. Unfortunately, before executing
    this method you must first import nova.utils, which imports a large
    number of modules itself. Anything imported (transitively) by
    nova.utils would therefore be imported before monkey patching, which
    included urllib3. This triggers the infinite recursion problem
    described above if you have an affected combination of library
    versions.

    While this specific issue may eventually be worked around or fixed in
    eventlet or urllib3, it remains true that eventlet best practises are
    to monkey patch as early as possible, which we were not doing. To
    avoid this and hopefully future similar issues, this change ensures
    that monkey patching happens as early as possible, and only a minimum
    number of modules are imported first.

    This change fixes monkey patching for both non-wsgi and wsgi callers:

    * Non-WSGI services (nova/cmd)

      This is fixed by using the new monkey_patch module, which has minimal
      dependencies.

    * WSGI services (nova/api/openstack)

      This is fixed both by using the new monkey_patch module, and by moving
      the patching point up one level so that it is done before importing
      anything in nova/api/openstack/__init__.py.

      This move causes issues for some external tools which load this path
      from nova and now monkey patch where they previously did not. However,
      it is unfortunately unavoidable to enable monkey patching for the wsgi
      entry point without major restructuring. This change includes a
      workaround for sphinx to avoid this issue.

    This change has been through several iterations. I started with what
    seemed like the simplest and most obvious change, and moved on as I
    discovered more interactions which broke. It is clear that eventlet
    monkey patching is extremely fragile, especially when done implicitly at
    module load time as we do. I would advocate a code restructure to
    improve this situation, but I think the time would be better spent
    removing the eventlet dependency entirely.

    Co-authored-by: Lee Yarwood <lyarwood@redhat...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.openstack.org/647310

[Corey Bryant (corey.bryant)]

For the Ubuntu SRU, a new version of the package has been uploaded to eoan and disco, and is awaiting review by the SRU team in disco [1].

[1] https://launchpad.net/ubuntu/disco/+queue?queue_state=1&queue_text=nova

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 2:19.0.0-0ubuntu3

---------------
nova (2:19.0.0-0ubuntu3) eoan; urgency=medium

  * d/p/eventlet-monkey-patching-should-be-as-early-as-possible.patch:
    Cherry-picked from upstream master branch to fix py3+wsgi+ssl crash
    (LP: #1808951).

 -- Corey Bryant <email address hidden> Thu, 25 Apr 2019 09:51:42 -0400

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/647310
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d0f540742efc1004b4d2c64814dcc6f7b9f0ccf6
Submitter: Zuul
Branch: stable/stein

commit d0f540742efc1004b4d2c64814dcc6f7b9f0ccf6
Author: Matthew Booth <email address hidden>
Date: Wed Jan 30 15:10:25 2019 +0000

    Eventlet monkey patching should be as early as possible

    We were seeing infinite recursion opening an ssl socket when running
    various combinations of python3, eventlet, and urllib3. It is not
    clear exactly what combination of versions are affected, but for
    background there is an example of this issue documented here:

    https://github.com/eventlet/eventlet/issues/371

    The immediate cause in nova's case was that we were calling
    eventlet.monkey_patch() after importing urllib3. Specifically, change
    Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 introduced the
    nova.utils.monkey_patch() method to make monkey patching common
    between WSGI and non-WSGI services. Unfortunately, before executing
    this method you must first import nova.utils, which imports a large
    number of modules itself. Anything imported (transitively) by
    nova.utils would therefore be imported before monkey patching, which
    included urllib3. This triggers the infinite recursion problem
    described above if you have an affected combination of library
    versions.

    While this specific issue may eventually be worked around or fixed in
    eventlet or urllib3, it remains true that eventlet best practises are
    to monkey patch as early as possible, which we were not doing. To
    avoid this and hopefully future similar issues, this change ensures
    that monkey patching happens as early as possible, and only a minimum
    number of modules are imported first.

    This change fixes monkey patching for both non-wsgi and wsgi callers:

    * Non-WSGI services (nova/cmd)

      This is fixed by using the new monkey_patch module, which has minimal
      dependencies.

    * WSGI services (nova/api/openstack)

      This is fixed both by using the new monkey_patch module, and by moving
      the patching point up one level so that it is done before importing
      anything in nova/api/openstack/__init__.py.

      This move causes issues for some external tools which load this path
      from nova and now monkey patch where they previously did not. However,
      it is unfortunately unavoidable to enable monkey patching for the wsgi
      entry point without major restructuring. This change includes a
      workaround for sphinx to avoid this issue.

    This change has been through several iterations. I started with what
    seemed like the simplest and most obvious change, and moved on as I
    discovered more interactions which broke. It is clear that eventlet
    monkey patching is extremely fragile, especially when done implicitly at
    module load time as we do. I would advocate a code restructure to
    improve this situation, but I think the time would be better spent
    removing the eventlet dependency entirely.

    Co-authored-by: Lee Yarwood <lyarwood@re...

[Brian Murray (brian-murray)]

Hello yatin, or anyone else affected,

Accepted nova into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:19.0.0-0ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello yatin, or anyone else affected,

Accepted nova into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello yatin, or anyone else affected,

Accepted nova into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:19.0.0-0ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Jamon Camisso (jamon)]

I can confirm the python3-nova and associated dependencies from cloud-archive:stein-proposed have stopped the errors like these ones form the original report:

2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [Previous line repeated 280 more times]
2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object

I tested on bionic using the PPA and whereas nova was not accepting any requests before, now it works as expected.

[Corey Bryant (corey.bryant)]

Hi Jamon, If you've tested the packages in -proposed pockets that fix this issue, please can you tag this bug appropriately (ie. verification-done-disco, etc).

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.1 release.

[David Coronel (davecore)]

I confirm the package in stein-proposed fixes the issue.

[David Coronel (davecore)]

I subscribed field-critical to this bug as this is affecting a customer deployment.

[Dean Henrichsmeyer (dean)]

Unsubscribing field-critical

[Przemyslaw Hausman (phausman)]

Hi, I have also verified that the fix works on bionic-stein, with -proposed packages.

https://paste.ubuntu.com/p/nQgKFWxVw4/

[Corey Bryant (corey.bryant)]

Przemyslaw/David, this also needs to get verified on disco before it can land in stein UCA. Any chance you can test that as well? If not please let us know. Thanks!

[David Coronel (davecore)]

Hi Corey, I don't have access to a Disco cloud to test. Can we still get this fix in stein? This is a blocker for a customer deployment.

[Sahid Orentino (sahid-ferdjaoui)]

disco-proposed tested, please consider the results:

  https://paste.ubuntu.com/p/8nBkgzS6YZ/
  https://paste.ubuntu.com/p/nJS75z2rT2/

[James Troup (elmo)]

Subscribed field-medium as we're deploying a cloud with this.

[Sahid Orentino (sahid-ferdjaoui)]

In comment #20 my verification is not right I should have deployed with SSL enabled.

[Sahid Orentino (sahid-ferdjaoui)]

$ dpkg -s nova-compute | grep -i version
Version: 2:19.0.0-0ubuntu2.3

[James Page (james-page)]

To complement Sahid's testing I also ran a tempest smoke against a SSL enabled Disco cloud:

======
Totals
======
Ran: 92 tests in 423.4308 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 592.6270 sec.

All tests passed.

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 2:19.0.0-0ubuntu2.3

---------------
nova (2:19.0.0-0ubuntu2.3) disco; urgency=medium

  * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
    virsh disk attach does not fail silently (LP: #1825882).
  * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
    always disconnect volumes after libvirt exceptions (LP: #1826523).

nova (2:19.0.0-0ubuntu2.2) disco; urgency=medium

  * d/p/xenapi-agent-change-openssl-error-handling.patch: Cherry-picked from
    upstream to ensure xenapi agent only raises a RuntimeError exception
    when openssl returns a non-zero exit code (LP: #1771506).

nova (2:19.0.0-0ubuntu2.1) disco; urgency=medium

  * d/gbp.conf: Create stable/stein branch.
  * d/p/eventlet-monkey-patching-should-be-as-early-as-possible.patch:
    Cherry-picked from upstream stable/stein review to fix py3+wsgi+ssl crash
    (LP: #1808951).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 10:54:46 +0200

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

[Balazs Gibizer (balazs-gibizer)]

There is a chance that we re-introduce this issue by re-enabling greendns in nova with[1]. If you see this happening again then please set the environment variable EVENTLET_NO_GREENDNS=yes before starting the nova-api service.

[1] https://review.opendev.org/c/openstack/nova/+/830966
[2] https://bugs.launchpad.net/nova/+bug/1981080/comments/2