[SRU] multiprocessing module dysfunctional

Bug #672209 reported by Herbert V. Riedel on 2010-11-07
54
This bug affects 9 people
Affects Status Importance Assigned to Milestone
python2.7 (Ubuntu)
Undecided
Matthias Klose
Maverick
High
Barry Warsaw

Bug Description

* Impact of bug: This bug will prevent common usage of the multiprocessing module in Python 2.7 on Maverick.
* Fix: The bug was addressed in the current version of Ubuntu with a no-change rebuild. Since the original problem was caused by kernel configuration on the buildd's, and (as verified by PPA build) this has been rectified, a simple rebuild is all that's needed
* TEST CASE: see above or: $ python2.7 -c 'from _multiprocessing import SemLock'
* I don't see much chance for regression. Code which relies on multiprocessing in Python 2.7 can't currently run on Maverick. This rebuild will only re-enable standard Python functionality.
Binary package hint: python2.7

On Maverick with python2.7 (2.7-6 deb version), the multiprocessing module fails to operate:

python2.7 -c 'import multiprocessing; multiprocessing.Pool()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/multiprocessing/__init__.py", line 227, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 89, in __init__
    self._setup_queues()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 181, in _setup_queues
    from .queues import SimpleQueue
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 22, in <module>
    from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition
  File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 33, in <module>
    " function, see issue 3770.")
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

Herbert V. Riedel (hvr) wrote :

additional info: installed natty's 2.7-9ubuntu2 amd64 debs on my maverick workstation, but those exhibit the very same problem for me

Herbert V. Riedel (hvr) wrote :

I downloaded and compiled Python-2.7 myself with a simple './configure --prefix /opt/python2.7 && make && make install'...

the resulting custom python2.7 installation doesn't exhibit the described multiprocessing problem

Matthias Klose (doko) wrote :

please recheck with 2.7.1-1 from natty (you should be able to install these packages in maverick too). It works for me, but I still have to find out the cause for the change.

Barry Warsaw (barry) on 2010-11-28
tags: added: python27
Barry Warsaw (barry) wrote :

WFM in natty with 2.7.1-1

Changed in python2.7 (Ubuntu):
status: New → Invalid
Matthias Klose (doko) wrote :

if it works in natty, and not in maverick, then it's not invalid. likely we'll need a fix for maverick to fix it.

Changed in python2.7 (Ubuntu):
status: Invalid → New
Barry Warsaw (barry) wrote :

Works in Natty, fails in Maverick. Works in Maverick with a from-source build of current Python 2.7 trunk.

Barry Warsaw (barry) wrote :

I rebuilt the maverick package in ppa:barry/python, then installed it on my amd64 maverick VM:

@mission[~:1003]% python2.7
Python 2.7.0+ (r27:82500, Jan 10 2011, 23:42:23)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> multiprocessing.Pool()
<multiprocessing.pool.Pool object at 0x17b9f10>
>>>

Changed in python2.7 (Ubuntu):
assignee: nobody → Matthias Klose (doko)
Barry Warsaw (barry) wrote :

Mattias is going to follow up with the admins to check kernel versions on buildds. If they've been upgraded then a rebuild of the Maverick package should fix this.

Barry Warsaw (barry) on 2011-01-11
Changed in python2.7 (Ubuntu):
status: New → Incomplete
Scott Kitterman (kitterman) wrote :

Updated status to be for Maverick only.

Changed in python2.7 (Ubuntu Maverick):
status: New → Incomplete
importance: Undecided → High
assignee: nobody → Matthias Klose (doko)
Changed in python2.7 (Ubuntu):
status: Incomplete → Fix Released
Changed in python2.7 (Ubuntu Maverick):
milestone: none → maverick-updates
Barry Warsaw (barry) wrote :

Matthias, did you ever check with the admins about the buildd kernel versions?

Buck Evan (bukzor) wrote :

I'm easily able to build this from source. Can I help by uploading a package?

What is holding this up, exactly?

Sorry to seem impatient, but it's been over four months.

--Buck

Barry Warsaw (barry) wrote :

Sorry, I'll work on an SRU for Maverick. I think a no-change rebuild will take care of it.

Changed in python2.7 (Ubuntu Maverick):
status: Incomplete → Confirmed
assignee: Matthias Klose (doko) → Barry Warsaw (barry)
Buck Evan (bukzor) wrote :

That would be perfect. Thanks Barry.

Barry Warsaw (barry) on 2011-04-26
summary: - multiprocessing module dysfunctional
+ [SRU] multiprocessing module dysfunctional
Barry Warsaw (barry) wrote :

For the SRU:

* Impact of bug: This bug will prevent common usage of the multiprocessing module in Python 2.7 on Maverick.
* Fix: The bug was addressed in the current version of Ubuntu with a no-change rebuild. Since the original problem was caused by kernel configuration on the buildd's, and (as verified by PPA build) this has been rectified, a simple rebuild is all that's needed
* I'll attach a patch momentarily. It is only a debian/changelog entry.
* TEST CASE: see above or: $ python2.7 -c 'from _multiprocessing import SemLock'
* I don't see much chance for regression. Code which relies on multiprocessing in Python 2.7 can't currently run on Maverick. This rebuild will only re-enable standard Python functionality.

Barry Warsaw (barry) wrote :

Oops, ignore the previous patch.

Barry Warsaw (barry) on 2011-04-26
Changed in python2.7 (Ubuntu Maverick):
status: Confirmed → In Progress
Barry Warsaw (barry) on 2011-04-28
Changed in python2.7 (Ubuntu Maverick):
status: In Progress → Confirmed
Steve Langasek (vorlon) on 2011-05-04
description: updated
Colin Watson (cjwatson) wrote :

Sponsored, thanks Barry. Awaiting ~ubuntu-sru approval.

Changed in python2.7 (Ubuntu Maverick):
status: Confirmed → In Progress

Accepted python2.7 into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in python2.7 (Ubuntu Maverick):
status: In Progress → Fix Committed
tags: added: verification-needed
Barry Warsaw (barry) wrote :

Dang. I'm now seeing this same failure in an oneiric schroot.

Barry Warsaw (barry) wrote :

I had to enable /dev/shm mounting in my chroots. E.g. from schroot/default/fstab:

=== modified file 'schroot/default/fstab'
--- schroot/default/fstab 2011-07-13 20:34:15 +0000
+++ schroot/default/fstab 2011-08-22 18:56:00 +0000
@@ -16,8 +16,8 @@
 # killing services on the host.
 #/run /run none rw,bind 0 0
 #/run/lock /run/lock none rw,bind 0 0
-#/dev/shm /dev/shm none rw,bind 0 0
-#/run/shm /run/shm none rw,bind 0 0
+/dev/shm /dev/shm none rw,bind 0 0
+/run/shm /run/shm none rw,bind 0 0

 # Additions according to
 # https://wiki.ubuntu.com/SecurityTeam/BuildEnvironment

tags: added: testcase
JC Hulce (soaringsky) wrote :

This bug affects Ubuntu 10.10, Maverick Meerkat. Maverick has reached end-of-life and is no longer supported, so I am closing the bugtask for Maverick. Please upgrade to a newer version of Ubuntu.
More information here: https://lists.ubuntu.com/archives/ubuntu-announce/2012-April/000158.html

Changed in python2.7 (Ubuntu Maverick):
status: Fix Committed → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.