New eventlet library breaks nova-manage

Bug #1407685 reported by Dr. Jens Harbott
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Victor Stinner
Juno
Fix Released
Undecided
Unassigned

Bug Description

This only affects stable/juno and stable/icehouse, which still use the deprecated eventlet.util module:

~# nova-manage service list
2015-01-05 13:13:11.202 29016 ERROR stevedore.extension [-] Could not load 'file': cannot import name util
2015-01-05 13:13:11.202 29016 ERROR stevedore.extension [-] cannot import name util
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension Traceback (most recent call last):
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension File "/opt/cloudbau/nova-virtualenv/lib/python2.7/site-packages/stevedore/extension.py", line 162, in _load_plugins
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension verify_requirements,
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension File "/opt/cloudbau/nova-virtualenv/lib/python2.7/site-packages/stevedore/extension.py", line 178, in _load_one_plugin
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension plugin = ep.load(require=verify_requirements)
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension File "/opt/cloudbau/nova-virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2306, in load
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension return self._load()
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension File "/opt/cloudbau/nova-virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2309, in _load
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension module = __import__(self.module_name, fromlist=['__name__'], level=0)
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension File "/opt/cloudbau/nova-virtualenv/lib/python2.7/site-packages/nova/image/download/file.py", line 23, in <module>
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension import nova.virt.libvirt.utils as lv_utils
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension File "/opt/cloudbau/nova-virtualenv/lib/python2.7/site-packages/nova/virt/libvirt/__init__.py", line 15, in <module>
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension from nova.virt.libvirt import driver
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension File "/opt/cloudbau/nova-virtualenv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 59, in <module>
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension from eventlet import util as eventlet_util
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension ImportError: cannot import name util
2015-01-05 13:13:11.202 29016 TRACE stevedore.extension

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

The changelog from http://eventlet.net/doc/changelog.html says:

- Removed deprecated modules: api, most of coros, pool, proc, processes and util

The master branch isn't affected as https://review.openstack.org/#/c/104930/ has removed the deprecated code.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/145023

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

When someone tries to reproduce this, make sure you are installing eventlet from source and not via pip, as the pip package still contains old files, see https://github.com/eventlet/eventlet/issues/186.

Revision history for this message
Daniel Berrange (berrange) wrote :

The problem appears caused by this libvirt driver.py code

            from eventlet import util as eventlet_util

            sock = eventlet_util.__original_socket__(socket.AF_INET,
                                                     socket.SOCK_STREAM)

It strikes me that eventlet_util.__original_socket__ is somewhat bad practice and could probably just be replaced by use of patcher.original("socket") without any real difficulty.

I think fixing the root cause is better than pinning eventlet to an old version

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/145335

Changed in nova:
assignee: nobody → Victor Stinner (victor-stinner)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/juno)

Change abandoned by Jens Rosenboom (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/145023
Reason: O.k., I thought that this change was some kind of minimal invasive solution for an issue that does not exist in the master branch anymore, but I'll take a look at the other approach, then.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/145356

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/juno)

Change abandoned by Jens Rosenboom (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/145356
Reason: After submitting this, I noticed that this is the same patch as https://review.openstack.org/#/c/145335/, so it'll probably better to backport that one.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/145335
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5793aff19033dd53cefa97fced64a4bf95ea0c72
Submitter: Jenkins
Branch: master

commit 5793aff19033dd53cefa97fced64a4bf95ea0c72
Author: Victor Stinner <email address hidden>
Date: Tue Jan 6 21:50:03 2015 +0100

    Update eventlet API in libvirt driver

    Stop using the deprecated eventlet.util API: use
    eventlet.patcher.original('socket') to access the original socket.socket
    type.

    Change-Id: Idbb9d2b53829dae0e807cd1260dee3dce155d5f3
    Closes-Bug: 1407685

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

The fix works fine for me, but I would like to see it backported to stable/icehouse and juno. @Victor: Will you take care of this?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/145955

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/146096

Revision history for this message
Joe Gordon (jogo) wrote :
Revision history for this message
Victor Stinner (vstinner) wrote :

jogo asked me on IRC a retrospective of the bug.

The module eventlet.util was deprecated. In eventlet 0.16, the module was removed. eventlet 0.15 -> 0.16 changelog:
https://lists.secondlife.com/pipermail/eventletdev/2015-January/001202.html

There was a mistake in the tarball or the wheel package (i don't remember which one), it still contains eventlet.util in the 0.16 release. The release 0.16.1 fixes that (tarball & wheel contain the same thing).
=> https://github.com/eventlet/eventlet/issues/186

I wrote a simple change to stop using eventlet.util in nova (https://review.openstack.org/145335). My change was backported to icehouse & juno, patches are under review.

FYI I was not aware that eventlet.util was deprecated, nor that it was removed. I also contributed to patches to support python 3 in eventlet, especially when using monkey patching, which is unrelated to this issue.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/juno)

Reviewed: https://review.openstack.org/145955
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9f81d9bb3458bca60bcc878e292bfecc3929fd8e
Submitter: Jenkins
Branch: stable/juno

commit 9f81d9bb3458bca60bcc878e292bfecc3929fd8e
Author: Victor Stinner <email address hidden>
Date: Tue Jan 6 21:50:03 2015 +0100

    Update eventlet API in libvirt driver

    Stop using the deprecated eventlet.util API: use
    eventlet.patcher.original('socket') to access the original socket.socket
    type.

    Conflicts:
            nova/virt/libvirt/host.py

    NOTE(mriedem): The conflict is due to the host module being
    newly refactored out of the driver module in Kilo.

    Change-Id: Idbb9d2b53829dae0e807cd1260dee3dce155d5f3
    Closes-Bug: 1407685
    (cherry picked from commit 5793aff19033dd53cefa97fced64a4bf95ea0c72)

tags: added: in-stable-juno
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/icehouse)

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/146096
Reason: Commit 6201bae3a114279bb2d2aa071ac83f86047c475a caps eventlet for stable/icehouse so dropping this.

Thierry Carrez (ttx)
Changed in nova:
milestone: none → kilo-2
status: Fix Committed → Fix Released
no longer affects: openstack-ansible
no longer affects: openstack-ansible/trunk
no longer affects: openstack-ansible/juno
no longer affects: openstack-ansible/icehouse
Thierry Carrez (ttx)
Changed in nova:
milestone: kilo-2 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.