Crash with pyMPI that doesn't happen with latest upstream

[gpk]

A newer version of openmpi is available upstream: 1.2.8 .

I am using pyMPI-2.5b0 and it crashes and dies if I use the Ubuntu openmpi binary, but runs fine with the newer upstream. This occurs on both 32 and 64-bit Ubuntu, on both Hardy Heron and Intrepid Ibex. I realize (of course) that pyMPI is not packaged by Ubuntu -- I compile it from source. But the point is, it runs nicely when I compile openmpi myself,

At the moment, I have no cluse where the bug is, but presumably the OpenMPI guys (and gals) fixed something since the 1.2.5 version adopted by Ubuntu. Sorry for the lack of detail here.

$ lsb_release -rd
Description: Ubuntu 8.04.1
Release: 8.04
mace:~$

~$ apt-cache policy libopenmpi-dev
libopenmpi-dev:
  Installed: 1.2.5-1ubuntu1.1
  Candidate: 1.2.5-1ubuntu1.1
  Version table:
 *** 1.2.5-1ubuntu1.1 0
        500 http://gb.archive.ubuntu.com hardy-updates/universe Packages
        100 /var/lib/dpkg/status
     1.2.5-1ubuntu1 0
        500 http://gb.archive.ubuntu.com hardy/universe Packages
mace:~$

$ pyMPI
libibverbs: Fatal: couldn't read uverbs ABI version.
--------------------------------------------------------------------------
[0,0,0]: OpenIB on host mace was unable to find any HCAs.
Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
Python 2.5.2 (pyMPI 2.5b0) on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mace/gpk/local/lib/python2.5/site-packages/numpy/__init__.py", line 93, in <module>
    import add_newdocs
  File "/home/mace/gpk/local/lib/python2.5/site-packages/numpy/add_newdocs.py", line 9, in <module>
    from lib import add_newdoc
  File "/home/mace/gpk/local/lib/python2.5/site-packages/numpy/lib/__init__.py", line 4, in <module>
    from type_check import *
  File "/home/mace/gpk/local/lib/python2.5/site-packages/numpy/lib/type_check.py", line 8, in <module>
    import numpy.core.numeric as _nx
  File "/home/mace/gpk/local/lib/python2.5/site-packages/numpy/core/__init__.py", line 5, in <module>
    import multiarray
ImportError: /home/mace/gpk/local/lib/python2.5/site-packages/numpy/core/multiarray.so: undefined symbol: Py_InitModule4
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/apport_python_hook.py", line 38, in apport_excepthook
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python2.5/site-packages/apport/__init__.py", line 1, in <module>
    from apport.report import Report
  File "/usr/lib/python2.5/site-packages/apport/report.py", line 21, in <module>
    import fileutils
  File "/usr/lib/python2.5/site-packages/apport/fileutils.py", line 16, in <module>
    from packaging_impl import impl as packaging
  File "/usr/lib/python2.5/site-packages/apport/packaging_impl.py", line 18, in <module>
    import apt
  File "/usr/lib/python2.5/site-packages/apt/__init__.py", line 2, in <module>
    import apt_pkg
ImportError: /usr/lib/python2.5/site-packages/apt_pkg.so: undefined symbol: Py_InitModule4

Original exception was:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mace/gpk/local/lib/python2.5/site-packages/numpy/__init__.py", line 93, in <module>
    import add_newdocs
  File "/home/mace/gpk/local/lib/python2.5/site-packages/numpy/add_newdocs.py", line 9, in <module>
    from lib import add_newdoc
  File "/home/mace/gpk/local/lib/python2.5/site-packages/numpy/lib/__init__.py", line 4, in <module>
    from type_check import *
  File "/home/mace/gpk/local/lib/python2.5/site-packages/numpy/lib/type_check.py", line 8, in <module>
    import numpy.core.numeric as _nx
  File "/home/mace/gpk/local/lib/python2.5/site-packages/numpy/core/__init__.py", line 5, in <module>
    import multiarray
ImportError: /home/mace/gpk/local/lib/python2.5/site-packages/numpy/core/multiarray.so: undefined symbol: Py_InitModule4
[76571 refs]
>>>

[Olaf Lenz (olenz)]

I can't reproduce you problem with Intrepid on 64bit with openmpi and the current pyMPI (2.5b0).
I compiled pyMPI via configure and make, and also ran make check (1 of 89 tests fails).

> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 8.10
Release: 8.10
Codename: intrepid

> apt-cache policy libopenmpi-dev
libopenmpi-dev:
  Installed: 1.2.7~rc2-1ubuntu2
  Candidate: 1.2.7~rc2-1ubuntu2
  Version table:
 *** 1.2.7~rc2-1ubuntu2 0
        500 http://de.archive.ubuntu.com intrepid/universe Packages
        100 /var/lib/dpkg/status

> ./pyMPI
Python 2.5.2 (r252:60911, Oct 5 2008, 19:49:54)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

[gpk (gpk-kochanski)]

You can't reproduce it because the problem doesn't show up in "make check".

The problem happens when you import numpy from within pyMPI:

mace:~$ pyMPI
# THIS STUFF HERE SEEMS TO BE WARNING MESSAGES -- ANNOYING,
# BUT NOT THE REAL PROBLEM:

libibverbs: Fatal: couldn't read uverbs ABI version.
--------------------------------------------------------------------------
[0,0,0]: OpenIB on host mace was unable to find any HCAs.
Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
Python 2.5.2 (pyMPI 2.5b0) on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import numpy # THIS CAUSES THE PROBLEM

# THIS IS THE REAL PROBLEM:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
....

[gpk (gpk-kochanski)]

I suppose this could be a numpy problem, instead of an openMPI problem. It's not an obvious problem because running
python -c 'import numpy; numpy.test()'
gives no errors.

[Olaf Lenz (olenz)]

Ok, apparently I hadn't read the report carefully enough. Still, everything works fine here:

> sudo apt-cache policy python-numpy
python-numpy:
  Installed: 1:1.1.1-1
  Candidate: 1:1.1.1-1
  Version table:
 *** 1:1.1.1-1 0
        500 http://de.archive.ubuntu.com intrepid/main Packages
        100 /var/lib/dpkg/status

> ./pyMPI
Python 2.5.2 (r252:60911, Oct 5 2008, 19:49:54)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import numpy.random
>>> numpy.random.normal(0.5,0.25)
0.67079544228001975

So I guess your problem is somehow in your system setup.

[gpk (gpk-kochanski)]

It might be something in the system config, but it's not obvious. The trouble is, aside from pyMPI, it's a professionally maintained Ubuntu system, with no customization other than package selection. It's a 64-bit machine with a 32-bit Ubuntu installation.

This seems to be related, perhaps, to the problem discussed at
http://mail.python.org/pipermail/python-list/2007-March/432606.html

The idea is that when built as 64-bit, python doesn't have
Py_InitModule4, but rather has Py_InitModule4_64. So the problem seems to be some kind of 32/64 bit issue.

The computers that exhibit this problem are all 64-bit processors with a 32-bit installation of Ubuntu. There was a suggestion that " Maybe numpy is more accurate in testing
wether it run's on 64 or on 32 bit than the python build script itself and
therefore expect's Py_InitModule4 to be present."

It also seems to have some smlilarities to https://launchpad.net/ubuntu/+source/clearsilver/+bug/86685 .

[Olaf Lenz (olenz)]

Ok, this does make some sort of sense.
I would suggest to move the bug to numpy for now.
The error message of above seems to show that the error occurs when you use pyMPI. Could you please also test whether the problem occurs also when you use pure python instead of pyMPI?

[Olaf Lenz (olenz)]

In the original bug report you stated, that the bug ocurs on 64bit as well as on 32bit. Does this mean that you were talking about 32bit and 64bit Ubuntu, but always on a 64bit machine?
Also, is it connected to pyMPI, or is pure python enough?
And have you already checked whether there are any bug reports at the pyMPI site?

[gpk (gpk-kochanski)]

As of yesterday, when I was re-compiling stuff, I have a
* 64-bit machine with 64-bit Ubuntu 8.10: works fine. I compile openmpi-1.3.?. (I'm running stuff, and not in a position right now to try the Ubuntu package.)
* 64-but machines with 32-bit Ubuntu 8.04: No good, even if I recompile openmpi-1.3.? But the Ubuntu openmpi-dev package is installed too.
* 32-bit machine with 32-bit Debian with Debian openmpi-dev package and python-numpy: Works with Debian openmpi-dev package.
* 32-bit machine with 32-bit Ubuntu 8.04: fail. Ubuntu openmpi package installed.
* 64-bit machine with old 64-bit SUSE linux: Works. (I compile openmpi)

Now, I'd forgotten that the Ubuntu package was also installed on some of these machines. One could imagine that there is some mess-up in the
LD_LIBRARY_PATH or PYTHONPATH or something. I'll look into that.

But all the compiles, both openmpi and numpy are pretty straightforward. Open the box, ./configure --prefix=$HOME/local; make install or python setup.py install --prefix=$HOME/local ,
as appropriate.

python by itself has no problem with numpy. It is only pyMPI.

OOOH! Look at this. The problem isn't numpy after all.
It shows up with other things:

chilli:~$ pyMPI
libibverbs: Fatal: couldn't read uverbs ABI version.
--------------------------------------------------------------------------
[0,0,0]: OpenIB on host chilli was unable to find any HCAs.
Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
Python 2.5.2 (pyMPI 2.5b0) on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import math
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /home/gpk/local/lib/python2.5/lib-dynload/math.so: undefined symbol: Py_InitModule4
[20074 refs]
>>>

Finally, I haven't seen this on the pyMPI sites (via google search), but pyMPI isn't a really big community. And, nobody in their right mind would use a 32-bit operating system on a 64-bit machine if they want to do serious computing with MPI. (So why do am *I* doing it? IT support that likes uniformity, that's why...)

[Olaf Lenz (olenz)]

So finally it looks as though the bug is neither OpenMPI nor numpy-related, but a problem of pyMPI. As there is no pyMPI package for ubuntu, I mark the bug as invalid.
I think you should contact the pyMPI development team if you need to get the bug fixed.
If that doesn't help, you could probably try the python module of the boost.MPI library, maybe that works:
http://www.boost.org/doc/libs/1_38_0/doc/html/mpi.html

Good luck!