Patched eventlet hub not in use

Bug #1897877 reported by Jacek Tomasiak
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.service
In Progress
Undecided
Jacek Tomasiak

Bug Description

The `os.environ['EVENTLET_HUB'] = 'oslo_service:service_hub'` line in __init__.py is supposed to install a custom eventlet hub but it is only effective if hub was not requested from eventlet before.

It could happen that hub is assigned indirectly via some imported code and above patch will not be effective.

The problem was detected in nova-compute after upgrading pycrypto/cffi code. It does some plugin loading using system calls and pipes when the modules are loaded. This becomes first use of io code patched by eventlet's green modules:
   File "/usr/bin/nova-compute", line 6, in <module> from nova.cmd.compute import main,
   File "/usr/lib/python2.7/site-packages/nova/__init__.py", line 39, in <module> import oslo_service # noqa,
   File "/usr/lib/python2.7/site-packages/oslo_service/__init__.py", line 16, in <module> import eventlet.patcher,
   File "/usr/lib/python2.7/site-packages/eventlet/__init__.py", line 15, in <module> from eventlet import convenience,
   File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 6, in <module> from eventlet.green import socket,
   File "/usr/lib/python2.7/site-packages/eventlet/green/socket.py", line 28, in <module> from eventlet.support import greendns,
   File "/usr/lib/python2.7/site-packages/eventlet/support/greendns.py", line 65, in <module> setattr(dns, pkg, import_patched(\dns.\ + pkg)),
   File "/usr/lib/python2.7/site-packages/eventlet/support/greendns.py", line 59, in import_patched return patcher.import_patched(module_name, **modules),
   File "/usr/lib/python2.7/site-packages/eventlet/patcher.py", line 119, in import_patched *additional_modules + tuple(kw_additional_modules.items())),
   File "/usr/lib/python2.7/site-packages/eventlet/patcher.py", line 93, in inject module = __import__(module_name, {}, {}, module_name.split(\.\)[:-1]),
   File "/usr/lib/python2.7/site-packages/eventlet/support/dns/dnssec.py", line 428, in <module> import Crypto.PublicKey.RSA,
   File "/usr/lib64/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 42, in <module> from Crypto.Math.Numbers import Integer,
   File "/usr/lib64/python2.7/site-packages/Crypto/Math/Numbers.py", line 42, in <module> from Crypto.Math._Numbers_gmp import Integer,
   File "/usr/lib64/python2.7/site-packages/Crypto/Math/_Numbers_gmp.py", line 95, in <module> lib = load_lib("gmp", gmp_defs),
   File "/usr/lib64/python2.7/site-packages/Crypto/Util/_raw_api.py", line 76, in load_lib lib = ffi.dlopen(name),
   File "/usr/lib64/python2.7/site-packages/cffi/api.py", line 141, in dlopen lib, function_cache = _make_ffi_library(self, name, flags),
   File "/usr/lib64/python2.7/site-packages/cffi/api.py", line 802, in _make_ffi_library backendlib = _load_backend_lib(backend, libname, flags),
   File "/usr/lib64/python2.7/site-packages/cffi/api.py", line 788, in _load_backend_lib path = ctypes.util.find_library(name),
   File "/usr/lib64/python2.7/ctypes/util.py", line 274, in find_library return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name)),
   File "/usr/lib64/python2.7/ctypes/util.py", line 264, in _findSoname_ldconfig env=env),
   File "/usr/lib64/python2.7/subprocess.py", line 424, in __init__ self.stdout = os.fdopen(c2pread, \rb\, bufsize),
   File "/usr/lib/python2.7/site-packages/eventlet/green/os.py", line 28, in fdopen return greenio.GreenPipe(fd, *args, **kw),
   File "/usr/lib/python2.7/site-packages/eventlet/greenio/py2.py", line 41, in __init__ super(GreenPipe, self).__init__(_SocketDuckForFd(fileno), mode, bufsize),
   File "/usr/lib/python2.7/site-packages/eventlet/greenio/py2.py", line 128, in __init__ notify_opened(fileno),
   File "/usr/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 194, in notify_opened hub = get_hub(),

The trace is from Rocky code but similar cases are likely to happen in other versions with right mix of imported modules.

Possible fix: reset eventlet hub right after patching to make sure the new version will be used.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (master)

Fix proposed to branch: master
Review: https://review.opendev.org/755245

Changed in oslo.service:
assignee: nobody → Jacek Tomasiak (skazi)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/760773

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/760774

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/760775

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/760777

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/760778

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/760779

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/760780

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/ussuri)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/oslo.service/+/760774
Reason: Let's wait for the patch to land on master before considering backports

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/victoria)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/oslo.service/+/760773
Reason: Let's wait for the patch to land on master before considering backports

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/train)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/oslo.service/+/760775
Reason: Let's wait for the patch to land on master before considering backports

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/rocky)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/rocky
Review: https://review.opendev.org/c/openstack/oslo.service/+/760778
Reason: Let's wait for the patch to land on master before considering backports

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/queens)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/oslo.service/+/760779
Reason: Let's wait for the patch to land on master before considering backports

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/pike)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/pike
Review: https://review.opendev.org/c/openstack/oslo.service/+/760780
Reason: Let's wait for the patch to land on master before considering backports

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/stein)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/oslo.service/+/760777
Reason: Let's wait for the patch to land on master before considering backports

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (master)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/oslo.service/+/755245

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.service (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/oslo.service/+/760774
Committed: https://opendev.org/openstack/oslo.service/commit/5b35903e145a0a1b8c64b7f4ba71fa98ea1d935b
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 5b35903e145a0a1b8c64b7f4ba71fa98ea1d935b
Author: Jacek Tomasiak <email address hidden>
Date: Wed Sep 30 11:26:11 2020 +0200

    [StableOnly] Reset eventlet hub after patching

    Patching the eventlet hub by setting the environment variable will not be
    effective if hub singleton was already set by some imported module.
    Calling `use_hub()` will reset it's value based on the current overrides.

    Change-Id: Ida6756b50266b4b2b2545f82e03e83e1c3be1e9e
    Closes-Bug: #1897877
    (cherry picked from commit 7f75fc1d4e7116b954db8924374e39c9fa8d9f6e)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.service (stable/train)

Reviewed: https://review.opendev.org/c/openstack/oslo.service/+/760775
Committed: https://opendev.org/openstack/oslo.service/commit/f22fe54a9e6c7e719f58eefefc0a44b9e3575eb4
Submitter: "Zuul (22348)"
Branch: stable/train

commit f22fe54a9e6c7e719f58eefefc0a44b9e3575eb4
Author: Jacek Tomasiak <email address hidden>
Date: Wed Sep 30 11:26:11 2020 +0200

    [StableOnly] Reset eventlet hub after patching

    Patching the eventlet hub by setting the environment variable will not be
    effective if hub singleton was already set by some imported module.
    Calling `use_hub()` will reset it's value based on the current overrides.

    Change-Id: Ida6756b50266b4b2b2545f82e03e83e1c3be1e9e
    Closes-Bug: #1897877
    (cherry picked from commit 7f75fc1d4e7116b954db8924374e39c9fa8d9f6e)
    (cherry picked from commit 5b35903e145a0a1b8c64b7f4ba71fa98ea1d935b)

tags: added: in-stable-train
tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.service (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/oslo.service/+/760773
Committed: https://opendev.org/openstack/oslo.service/commit/7f75fc1d4e7116b954db8924374e39c9fa8d9f6e
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 7f75fc1d4e7116b954db8924374e39c9fa8d9f6e
Author: Jacek Tomasiak <email address hidden>
Date: Wed Sep 30 11:26:11 2020 +0200

    [StableOnly] Reset eventlet hub after patching

    Patching the eventlet hub by setting the environment variable will not be
    effective if hub singleton was already set by some imported module.
    Calling `use_hub()` will reset it's value based on the current overrides.

    Change-Id: Ida6756b50266b4b2b2545f82e03e83e1c3be1e9e
    Closes-Bug: #1897877

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/pike)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/pike
Review: https://review.opendev.org/c/openstack/oslo.service/+/760780
Reason: stable/pike has transitioned to End of Life for oslo, open patches need to be abandoned in order to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.service (stable/queens)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/oslo.service/+/760779
Reason: This branch of this project has transitioned to End of Life, open patches need to be abandoned to be able to delete the branch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.