Support for multiprocessing broken in python3.1

Bug #630511 reported by Daniele Pighin on 2010-09-04
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
python3.1 (Ubuntu)
Critical
Unassigned
Lucid
Critical
Canonical Foundations Team
Maverick
Critical
Unassigned

Bug Description

Binary package hint: python3.1

Just upgraded to 10.10beta.

Python3.1 programs trying to use multiprocessing.Pool exit with the following error message:

Traceback (most recent call last):
  File "/usr/lib/python3.1/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock
ImportError: cannot import name SemLock
...
pool = Pool(processes = options.num_processes)
  File "/usr/lib/python3.1/multiprocessing/pool.py", line 84, in __init__
    self._setup_queues()
  File "/usr/lib/python3.1/multiprocessing/pool.py", line 133, in _setup_queues
    from .queues import SimpleQueue
  File "/usr/lib/python3.1/multiprocessing/queues.py", line 22, in <module>
    from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition
  File "/usr/lib/python3.1/multiprocessing/synchronize.py", line 33, in <module>
    " function, see issue 3770.")
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

I was debugging my code (not the multiprocessing part, by the way) just while upgrading to 10.10beta.
Everything worked fine before, suddenly stopped working during the "packages installation" stage of the upgrade.
The problem seems to be related to a wrong build process for the package.

Output of "lsb_release -rd":

Description: Ubuntu maverick (development branch)
Release: 10.10

Output of "apt-cache policy python3.1":

python3.1:
  Installed: 3.1.2+20100707-1ubuntu1
  Candidate: 3.1.2+20100707-1ubuntu1
  Version table:
 *** 3.1.2+20100707-1ubuntu1 0
        500 http://archive.ubuntu.com/ubuntu/ maverick/main amd64 Packages
        100 /var/lib/dpkg/status

Matthias Klose (doko) wrote :

please recheck with 3.1.2+20100915-0ubuntu1 when the package is built and in the archive

Changed in python3.1 (Ubuntu):
status: New → Incomplete
Daniele Pighin (daniele-pighin) wrote :

Just checked, I am afraid it's still there.

$ apt-cache policy python3.1python3.1:
  Installed: 3.1.2+20100915-0ubuntu1
  Candidate: 3.1.2+20100915-0ubuntu1
  Version table:
 *** 3.1.2+20100915-0ubuntu1 0
        500 http://archive.ubuntu.com/ubuntu/ maverick/main amd64 Packages
        100 /var/lib/dpkg/status

On 15.09.2010 21:45, Daniele Pighin wrote:
> Just checked, I am afraid it's still there.

do you have a test case which you could attach?

Daniele Pighin (daniele-pighin) wrote :

I am attaching a test case (straight from python documentation).

Matthias Klose (doko) wrote :

comparing a python3.1 build from sid with one on maverick, same thing when comparing the pyconfig.h from a local build with one from the buildd (both maverick):

--- pyconfig.h.sid 2010-09-09 20:11:48.000000000 +0200
+++ pyconfig.h.mav 2010-07-08 12:19:45.000000000 +0200
@@ -34,9 +34,6 @@
 /* Define if --enable-ipv6 is specified */
 #define ENABLE_IPV6 1

-/* Define if flock needs to be linked with bsd library */
-/* #undef FLOCK_NEEDS_LIBBSD */
-
 /* Define if getpgrp() must be called as getpgrp(0). */
 /* #undef GETPGRP_HAVE_ARG */

@@ -94,7 +91,7 @@
 /* #undef HAVE_BROKEN_PTHREAD_SIGMASK */

 /* define to 1 if your sem_getvalue is broken. */
-/* #undef HAVE_BROKEN_SEM_GETVALUE */
+#define HAVE_BROKEN_SEM_GETVALUE 1

 /* Define this if you have the type _Bool. */
 #define HAVE_C99_BOOL 1
@@ -906,7 +903,7 @@
 /* #undef PACKAGE_VERSION */

 /* Define if POSIX semaphores aren't enabled on your system */
-/* #undef POSIX_SEMAPHORES_NOT_ENABLED */
+#define POSIX_SEMAPHORES_NOT_ENABLED 1

 /* Defined if PTHREAD_SCOPE_SYSTEM supported. */
 #define PTHREAD_SYSTEM_SCHED_SUPPORTED 1

Changed in python3.1 (Ubuntu):
importance: Undecided → Critical
milestone: none → ubuntu-10.10
status: Incomplete → Triaged
Matthias Klose (doko) wrote :

same for lucid :-(

Changed in python3.1 (Ubuntu Lucid):
importance: Undecided → Critical
milestone: none → lucid-updates
status: New → Triaged
Matthias Klose (doko) wrote :

the general issue is that kernel features are checked at build time with a kernel which has nothing to do with the kernel of the release the package is built for.

Matthias Klose (doko) wrote :

the configure checks:

# Multiprocessing check for broken sem_getvalue
AC_MSG_CHECKING(for broken sem_getvalue)
AC_TRY_RUN([
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <semaphore.h>
#include <sys/stat.h>

int main(void){
  sem_t *a = sem_open("/autocftw", O_CREAT, S_IRUSR|S_IWUSR, 0);
  int count;
  int res;
  if(a==SEM_FAILED){
    perror("sem_open");
    return 1;

  }
  res = sem_getvalue(a, &count);
  sem_close(a);
  sem_unlink("/autocftw");
  return res==-1 ? 1 : 0;
}
]
,AC_MSG_RESULT(no),
 if test x`lsb_release -sc` = xlucid; then
 AC_MSG_RESULT(no)
 else
 AC_MSG_RESULT(yes)
  AC_DEFINE(HAVE_BROKEN_SEM_GETVALUE, 1, define to 1 if your sem_getvalue is broken.)
 fi
)

# For multiprocessing module, check that sem_open
# actually works. For FreeBSD versions <= 7.2,
# the kernel module that provides POSIX semaphores
# isn't loaded by default, so an attempt to call
# sem_open results in a 'Signal 12' error.
AC_MSG_CHECKING(whether POSIX semaphores are enabled)
AC_CACHE_VAL(ac_cv_posix_semaphores_enabled,
AC_TRY_RUN([
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <semaphore.h>
#include <sys/stat.h>

int main(void) {
  sem_t *a = sem_open("/autoconf", O_CREAT, S_IRUSR|S_IWUSR, 0);
  if (a == SEM_FAILED) {
    perror("sem_open");
    return 1;
  }
  sem_close(a);
  sem_unlink("/autoconf");
  return 0;
}
], ac_cv_posix_semaphores_enabled=yes,
   ac_cv_posix_semaphores_enabled=no,
   ac_cv_posix_semaphores_enabled=yes)
)
if test x`lsb_release -sc` = xlucid; then
  ac_cv_posix_semaphores_enabled=yes
fi
AC_MSG_RESULT($ac_cv_posix_semaphores_enabled)
if test $ac_cv_posix_semaphores_enabled = no
then
  AC_DEFINE(POSIX_SEMAPHORES_NOT_ENABLED, 1,
            [Define if POSIX semaphores aren't enabled on your system])
fi

Matthias Klose (doko) wrote :

how do we know that other packages are affected?

Matthias Klose (doko) wrote :

when building a package for release N, the buildd should really run the kernel of release N-1

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python3.1 - 3.1.2+20100915-0ubuntu2

---------------
python3.1 (3.1.2+20100915-0ubuntu2) maverick; urgency=low

  * Assume working semaphores, don't rely on running kernel for the check.
    LP: #630511.
 -- Matthias Klose <email address hidden> Thu, 16 Sep 2010 15:19:45 +0200

Changed in python3.1 (Ubuntu Maverick):
status: Triaged → Fix Released

I can confirm that with the current build I am no longer experiencing the problem.

python3.1:
  Installed: 3.1.2+20100915-0ubuntu3
  Candidate: 3.1.2+20100915-0ubuntu3
  Version table:
 *** 3.1.2+20100915-0ubuntu3 0
        500 http://archive.ubuntu.com/ubuntu/ maverick/main amd64 Packages
        100 /var/lib/dpkg/status

Thanks!

Matthias Klose (doko) on 2010-09-20
Changed in python3.1 (Ubuntu Lucid):
status: Triaged → In Progress
Changed in python3.1 (Ubuntu Lucid):
assignee: nobody → Canonical Foundations Team (canonical-foundations)

Is there any progress with this bug?

As far as I can say (and I had written in my reply on Sept. 17 2010) the bug
is fixed.

 Daniele

On Sun, Jan 23, 2011 at 11:48 AM, Sardorbek Pulatov (pROCKrammer) <
<email address hidden>> wrote:

> Is there any progress with this bug?
>
>

Martin Pitt (pitti) wrote :

Any testers of the lucid-proposed package? As this has been in -proposed for about four months, I'll remove the proposed package soon if there is no feedback. Thank you!

Barry Warsaw (barry) wrote :

Upgrade the following packages:
python3.1-minimal [3.1.2-0ubuntu2 (lucid, now) -> 3.1.2-0ubuntu3
(lucid-proposed)]

Score is 81

Accept this solution? [Y/n/q/?]
The following packages will be upgraded:
  python3.1 python3.1-minimal
2 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 5,303kB of archives. After unpacking 0B will be used.
Do you want to continue? [Y/n/?]
Writing extended state information... Done
Get:1 http://archive.ubuntu.com/ubuntu/ lucid-proposed/main python3.1 3.1.2-0ubuntu3 [3,760kB]
Get:2 http://archive.ubuntu.com/ubuntu/ lucid-proposed/main python3.1-minimal 3.1.2-0ubuntu3 [1,543kB]
Fetched 5,303kB in 8s (643kB/s)
(Reading database ... 47982 files and directories currently installed.)
Preparing to replace python3.1 3.1.2-0ubuntu2 (using .../python3.1_3.1.2-0ubuntu3_amd64.deb) ...
Unpacking replacement python3.1 ...
Preparing to replace python3.1-minimal 3.1.2-0ubuntu2 (using .../python3.1-minimal_3.1.2-0ubuntu3_amd64.deb) ...
Unpacking replacement python3.1-minimal ...
Processing triggers for man-db ...
Setting up python3.1-minimal (3.1.2-0ubuntu3) ...

Setting up python3.1 (3.1.2-0ubuntu3) ...

Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done

Current status: 0 updates [-2].
root@spindrift[/etc/apt:54]% python3.1 -c "import multiprocessing; multiprocessing.Pool()"
root@spindrift[/etc/apt:55]% python3.1
Python 3.1.2 (r312:79147, Sep 27 2010, 09:57:50)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> multiprocessing.Pool()
<multiprocessing.pool.Pool object at 0x12806d0>
>>>

Martin Pitt (pitti) wrote :

Thanks for testing!

Changed in python3.1 (Ubuntu Lucid):
status: In Progress → Fix Committed
tags: added: verification-done
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python3.1 - 3.1.2-0ubuntu3

---------------
python3.1 (3.1.2-0ubuntu3) lucid-proposed; urgency=low

  * Assume working semaphores, don't rely on running kernel for the check.
    LP: #630511.
  * Disable test_threading test on powerpc, which fail on the buildds.
 -- Matthias Klose <email address hidden> Mon, 20 Sep 2010 14:31:19 +0200

Changed in python3.1 (Ubuntu Lucid):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments