[SRU] multiprocessing module dysfunctional

[Herbert V. Riedel]

* Impact of bug: This bug will prevent common usage of the multiprocessing module in Python 2.7 on Maverick.
* Fix: The bug was addressed in the current version of Ubuntu with a no-change rebuild. Since the original problem was caused by kernel configuration on the buildd's, and (as verified by PPA build) this has been rectified, a simple rebuild is all that's needed
* TEST CASE: see above or: $ python2.7 -c 'from _multiprocessing import SemLock'
* I don't see much chance for regression. Code which relies on multiprocessing in Python 2.7 can't currently run on Maverick. This rebuild will only re-enable standard Python functionality.
Binary package hint: python2.7

On Maverick with python2.7 (2.7-6 deb version), the multiprocessing module fails to operate:

python2.7 -c 'import multiprocessing; multiprocessing.Pool()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/multiprocessing/__init__.py", line 227, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 89, in __init__
    self._setup_queues()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 181, in _setup_queues
    from .queues import SimpleQueue
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 22, in <module>
    from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition
  File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 33, in <module>
    " function, see issue 3770.")
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

[Herbert V. Riedel (hvr)]

additional info: installed natty's 2.7-9ubuntu2 amd64 debs on my maverick workstation, but those exhibit the very same problem for me

[Herbert V. Riedel (hvr)]

I downloaded and compiled Python-2.7 myself with a simple './configure --prefix /opt/python2.7 && make && make install'...

the resulting custom python2.7 installation doesn't exhibit the described multiprocessing problem

[Matthias Klose (doko)]

please recheck with 2.7.1-1 from natty (you should be able to install these packages in maverick too). It works for me, but I still have to find out the cause for the change.

[Barry Warsaw (barry)]

WFM in natty with 2.7.1-1

[Matthias Klose (doko)]

if it works in natty, and not in maverick, then it's not invalid. likely we'll need a fix for maverick to fix it.

[Barry Warsaw (barry)]

http://bugs.python.org/issue3770

[Barry Warsaw (barry)]

Works in Natty, fails in Maverick. Works in Maverick with a from-source build of current Python 2.7 trunk.

[Barry Warsaw (barry)]

I rebuilt the maverick package in ppa:barry/python, then installed it on my amd64 maverick VM:

@mission[~:1003]% python2.7
Python 2.7.0+ (r27:82500, Jan 10 2011, 23:42:23)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> multiprocessing.Pool()
<multiprocessing.pool.Pool object at 0x17b9f10>
>>>

[Barry Warsaw (barry)]

Mattias is going to follow up with the admins to check kernel versions on buildds. If they've been upgraded then a rebuild of the Maverick package should fix this.

[Scott Kitterman (kitterman)]

Updated status to be for Maverick only.

[Barry Warsaw (barry)]

Matthias, did you ever check with the admins about the buildd kernel versions?

[Buck Evan (bukzor)]

I'm easily able to build this from source. Can I help by uploading a package?

What is holding this up, exactly?

Sorry to seem impatient, but it's been over four months.

--Buck

[Barry Warsaw (barry)]

Sorry, I'll work on an SRU for Maverick. I think a no-change rebuild will take care of it.

[Buck Evan (bukzor)]

That would be perfect. Thanks Barry.

[Barry Warsaw (barry)]

For the SRU:

* Impact of bug: This bug will prevent common usage of the multiprocessing module in Python 2.7 on Maverick.
* Fix: The bug was addressed in the current version of Ubuntu with a no-change rebuild. Since the original problem was caused by kernel configuration on the buildd's, and (as verified by PPA build) this has been rectified, a simple rebuild is all that's needed
* I'll attach a patch momentarily. It is only a debian/changelog entry.
* TEST CASE: see above or: $ python2.7 -c 'from _multiprocessing import SemLock'
* I don't see much chance for regression. Code which relies on multiprocessing in Python 2.7 can't currently run on Maverick. This rebuild will only re-enable standard Python functionality.

[Barry Warsaw (barry)]

Oops, ignore the previous patch.

[Colin Watson (cjwatson)]

Sponsored, thanks Barry. Awaiting ~ubuntu-sru approval.

[Martin Pitt (pitti)]

Accepted python2.7 into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Barry Warsaw (barry)]

Dang. I'm now seeing this same failure in an oneiric schroot.

[Barry Warsaw (barry)]

I had to enable /dev/shm mounting in my chroots. E.g. from schroot/default/fstab:

=== modified file 'schroot/default/fstab'
--- schroot/default/fstab 2011-07-13 20:34:15 +0000
+++ schroot/default/fstab 2011-08-22 18:56:00 +0000
@@ -16,8 +16,8 @@
 # killing services on the host.
 #/run /run none rw,bind 0 0
 #/run/lock /run/lock none rw,bind 0 0
-#/dev/shm /dev/shm none rw,bind 0 0
-#/run/shm /run/shm none rw,bind 0 0
+/dev/shm /dev/shm none rw,bind 0 0
+/run/shm /run/shm none rw,bind 0 0

 # Additions according to
 # https://wiki.ubuntu.com/SecurityTeam/BuildEnvironment

[JC Hulce (soaringsky)]

This bug affects Ubuntu 10.10, Maverick Meerkat. Maverick has reached end-of-life and is no longer supported, so I am closing the bugtask for Maverick. Please upgrade to a newer version of Ubuntu.
More information here: https://lists.ubuntu.com/archives/ubuntu-announce/2012-April/000158.html