Eventlet seems to be incompatible with multiprocessing/pool.py

Bug #1225275 reported by Joshua Harlow on 2013-09-14
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
taskflow
Critical
Joshua Harlow
Ubuntu
Undecided
Unassigned

Bug Description

The internals of multiprocessing/pool.py seem to use a thread attribute that doesn't exist when eventlet monkey patches the threading module. The error that I am seeing is the following:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/y/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/home/y/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/y/lib/python2.6/multiprocessing/pool.py", line 221, in _handle_tasks
    if thread._state:
AttributeError: '_MainThread' object has no attribute '_state'

Likely the eventlet monkey patching does not patch this attribute (is this attribute is a hidden internal of the threading module?). We likely either need to figure out a different pool and use that or pretty clearly document this case and create a threaded impl that will work with eventlet.

Easy way to hit this:

import eventlet
eventlet.monkey_patch(thread=True)

Then try to use the threaded engine.

Joshua Harlow (harlowja) on 2013-09-14
description: updated
Joshua Harlow (harlowja) on 2013-09-14
Changed in taskflow:
importance: Undecided → Critical
Joshua Harlow (harlowja) on 2013-09-14
Changed in taskflow:
assignee: nobody → Joshua Harlow (harlowja)
Changed in taskflow:
status: New → In Progress

Reviewed: https://review.openstack.org/46603
Committed: http://github.com/stackforge/taskflow/commit/86c60dfa60496eaccf16dc50d13ac39f0cae451b
Submitter: Jenkins
Branch: master

commit 86c60dfa60496eaccf16dc50d13ac39f0cae451b
Author: Joshua Harlow <email address hidden>
Date: Fri Sep 13 20:21:03 2013 -0700

    Use executors instead of pools

    A executor is a more generic concept than a pool and
    also seems to work better with eventlet. This is also
    a more supported model of performing concurrent operations
    than using the mostly unknown multiprocessing thread
    pool implementation.

    Fixes: bug 1225275

    Change-Id: I09e9a9000bc88cc57d51342b83b31f97790a62e9

Changed in taskflow:
status: In Progress → Fix Committed
Changed in taskflow:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers