The `os.environ['EVENTLET_HUB'] = 'oslo_service:service_hub'` line in __init__.py is supposed to install a custom eventlet hub but it is only effective if hub was not requested from eventlet before.
It could happen that hub is assigned indirectly via some imported code and above patch will not be effective.
The problem was detected in nova-compute after upgrading pycrypto/cffi code. It does some plugin loading using system calls and pipes when the modules are loaded. This becomes first use of io code patched by eventlet's green modules:
File "/usr/bin/nova-compute", line 6, in <module> from nova.cmd.compute import main,
File "/usr/lib/python2.7/site-packages/nova/__init__.py", line 39, in <module> import oslo_service # noqa,
File "/usr/lib/python2.7/site-packages/oslo_service/__init__.py", line 16, in <module> import eventlet.patcher,
File "/usr/lib/python2.7/site-packages/eventlet/__init__.py", line 15, in <module> from eventlet import convenience,
File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 6, in <module> from eventlet.green import socket,
File "/usr/lib/python2.7/site-packages/eventlet/green/socket.py", line 28, in <module> from eventlet.support import greendns,
File "/usr/lib/python2.7/site-packages/eventlet/support/greendns.py", line 65, in <module> setattr(dns, pkg, import_patched(\dns.\ + pkg)),
File "/usr/lib/python2.7/site-packages/eventlet/support/greendns.py", line 59, in import_patched return patcher.import_patched(module_name, **modules),
File "/usr/lib/python2.7/site-packages/eventlet/patcher.py", line 119, in import_patched *additional_modules + tuple(kw_additional_modules.items())),
File "/usr/lib/python2.7/site-packages/eventlet/patcher.py", line 93, in inject module = __import__(module_name, {}, {}, module_name.split(\.\)[:-1]),
File "/usr/lib/python2.7/site-packages/eventlet/support/dns/dnssec.py", line 428, in <module> import Crypto.PublicKey.RSA,
File "/usr/lib64/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 42, in <module> from Crypto.Math.Numbers import Integer,
File "/usr/lib64/python2.7/site-packages/Crypto/Math/Numbers.py", line 42, in <module> from Crypto.Math._Numbers_gmp import Integer,
File "/usr/lib64/python2.7/site-packages/Crypto/Math/_Numbers_gmp.py", line 95, in <module> lib = load_lib("gmp", gmp_defs),
File "/usr/lib64/python2.7/site-packages/Crypto/Util/_raw_api.py", line 76, in load_lib lib = ffi.dlopen(name),
File "/usr/lib64/python2.7/site-packages/cffi/api.py", line 141, in dlopen lib, function_cache = _make_ffi_library(self, name, flags),
File "/usr/lib64/python2.7/site-packages/cffi/api.py", line 802, in _make_ffi_library backendlib = _load_backend_lib(backend, libname, flags),
File "/usr/lib64/python2.7/site-packages/cffi/api.py", line 788, in _load_backend_lib path = ctypes.util.find_library(name),
File "/usr/lib64/python2.7/ctypes/util.py", line 274, in find_library return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name)),
File "/usr/lib64/python2.7/ctypes/util.py", line 264, in _findSoname_ldconfig env=env),
File "/usr/lib64/python2.7/subprocess.py", line 424, in __init__ self.stdout = os.fdopen(c2pread, \rb\, bufsize),
File "/usr/lib/python2.7/site-packages/eventlet/green/os.py", line 28, in fdopen return greenio.GreenPipe(fd, *args, **kw),
File "/usr/lib/python2.7/site-packages/eventlet/greenio/py2.py", line 41, in __init__ super(GreenPipe, self).__init__(_SocketDuckForFd(fileno), mode, bufsize),
File "/usr/lib/python2.7/site-packages/eventlet/greenio/py2.py", line 128, in __init__ notify_opened(fileno),
File "/usr/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 194, in notify_opened hub = get_hub(),
The trace is from Rocky code but similar cases are likely to happen in other versions with right mix of imported modules.
Possible fix: reset eventlet hub right after patching to make sure the new version will be used.
Fix proposed to branch: master
Review: https:/