Patched eventlet hub not in use

Bug #1897877 reported by Jacek Tomasiak
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.service
Undecided
Jacek Tomasiak

Bug Description

The `os.environ['EVENTLET_HUB'] = 'oslo_service:service_hub'` line in __init__.py is supposed to install a custom eventlet hub but it is only effective if hub was not requested from eventlet before.

It could happen that hub is assigned indirectly via some imported code and above patch will not be effective.

The problem was detected in nova-compute after upgrading pycrypto/cffi code. It does some plugin loading using system calls and pipes when the modules are loaded. This becomes first use of io code patched by eventlet's green modules:
   File "/usr/bin/nova-compute", line 6, in <module> from nova.cmd.compute import main,
   File "/usr/lib/python2.7/site-packages/nova/__init__.py", line 39, in <module> import oslo_service # noqa,
   File "/usr/lib/python2.7/site-packages/oslo_service/__init__.py", line 16, in <module> import eventlet.patcher,
   File "/usr/lib/python2.7/site-packages/eventlet/__init__.py", line 15, in <module> from eventlet import convenience,
   File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 6, in <module> from eventlet.green import socket,
   File "/usr/lib/python2.7/site-packages/eventlet/green/socket.py", line 28, in <module> from eventlet.support import greendns,
   File "/usr/lib/python2.7/site-packages/eventlet/support/greendns.py", line 65, in <module> setattr(dns, pkg, import_patched(\dns.\ + pkg)),
   File "/usr/lib/python2.7/site-packages/eventlet/support/greendns.py", line 59, in import_patched return patcher.import_patched(module_name, **modules),
   File "/usr/lib/python2.7/site-packages/eventlet/patcher.py", line 119, in import_patched *additional_modules + tuple(kw_additional_modules.items())),
   File "/usr/lib/python2.7/site-packages/eventlet/patcher.py", line 93, in inject module = __import__(module_name, {}, {}, module_name.split(\.\)[:-1]),
   File "/usr/lib/python2.7/site-packages/eventlet/support/dns/dnssec.py", line 428, in <module> import Crypto.PublicKey.RSA,
   File "/usr/lib64/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 42, in <module> from Crypto.Math.Numbers import Integer,
   File "/usr/lib64/python2.7/site-packages/Crypto/Math/Numbers.py", line 42, in <module> from Crypto.Math._Numbers_gmp import Integer,
   File "/usr/lib64/python2.7/site-packages/Crypto/Math/_Numbers_gmp.py", line 95, in <module> lib = load_lib("gmp", gmp_defs),
   File "/usr/lib64/python2.7/site-packages/Crypto/Util/_raw_api.py", line 76, in load_lib lib = ffi.dlopen(name),
   File "/usr/lib64/python2.7/site-packages/cffi/api.py", line 141, in dlopen lib, function_cache = _make_ffi_library(self, name, flags),
   File "/usr/lib64/python2.7/site-packages/cffi/api.py", line 802, in _make_ffi_library backendlib = _load_backend_lib(backend, libname, flags),
   File "/usr/lib64/python2.7/site-packages/cffi/api.py", line 788, in _load_backend_lib path = ctypes.util.find_library(name),
   File "/usr/lib64/python2.7/ctypes/util.py", line 274, in find_library return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name)),
   File "/usr/lib64/python2.7/ctypes/util.py", line 264, in _findSoname_ldconfig env=env),
   File "/usr/lib64/python2.7/subprocess.py", line 424, in __init__ self.stdout = os.fdopen(c2pread, \rb\, bufsize),
   File "/usr/lib/python2.7/site-packages/eventlet/green/os.py", line 28, in fdopen return greenio.GreenPipe(fd, *args, **kw),
   File "/usr/lib/python2.7/site-packages/eventlet/greenio/py2.py", line 41, in __init__ super(GreenPipe, self).__init__(_SocketDuckForFd(fileno), mode, bufsize),
   File "/usr/lib/python2.7/site-packages/eventlet/greenio/py2.py", line 128, in __init__ notify_opened(fileno),
   File "/usr/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 194, in notify_opened hub = get_hub(),

The trace is from Rocky code but similar cases are likely to happen in other versions with right mix of imported modules.

Possible fix: reset eventlet hub right after patching to make sure the new version will be used.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (master)

Fix proposed to branch: master
Review: https://review.opendev.org/755245

Changed in oslo.service:
assignee: nobody → Jacek Tomasiak (skazi)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/760773

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/760774

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/760775

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/760777

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/760778

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/760779

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/760780

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/ussuri)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/oslo.service/+/760774
Reason: Let's wait for the patch to land on master before considering backports

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/victoria)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/oslo.service/+/760773
Reason: Let's wait for the patch to land on master before considering backports

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/train)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/oslo.service/+/760775
Reason: Let's wait for the patch to land on master before considering backports

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/rocky)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/rocky
Review: https://review.opendev.org/c/openstack/oslo.service/+/760778
Reason: Let's wait for the patch to land on master before considering backports

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/queens)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/oslo.service/+/760779
Reason: Let's wait for the patch to land on master before considering backports

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/pike)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/pike
Review: https://review.opendev.org/c/openstack/oslo.service/+/760780
Reason: Let's wait for the patch to land on master before considering backports

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/stein)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/oslo.service/+/760777
Reason: Let's wait for the patch to land on master before considering backports

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers