build failure on OpenSolaris -- can't find gmp.h

[Zooko Wilcox-O'Hearn]

The build fails when we try the ticket1037 branch on David Abrahams's OpenSolaris builder: http://tahoe-lafs.org/buildbot/builders/David%20A.%20OpenSolaris%20i386/builds/196/steps/build/logs/stdio

It appears to fail due to the absence of gmp.h:

src/_fastmath.c:34:17: gmp.h: No such file or directory

Our options include:

   1. remove OpenSolaris from the list of Supported Platforms to run Tahoe-LAFS
   2. ask Dave Abrahams to manually install gmp.h and leave OpenSolaris listed as a Supported Platform
   3. remove PyCrypto from the standard set of requirements and create an "SFTP"-specific set of requirements which are installed only if users specify that they need the "SFTP" feature of Tahoe-LAFS
   4. try to help the PyCrypto maintainer, Dwayne Litzenberger patch his setup.py [*] and then wait for a new release of PyCrypto which has that patch.
   5. Do option 1 or 2 now while simultaneously working on option 4. People will not then be able to actually install Tahoe-LAFS on OpenSolaris machines that are configured the same way that Dave's is, but they will be able to install Tahoe-LAFS on OpenSolaris machines that are configured differently (with libgmp headers installed), and once a new version of PyCrypto comes out that fixes this problem then they will be able to install on any platform.
   6. ?

[*] It looks like to me that PyCrypto is intended to fallback to a pure-Python implementation (appropriately named "slowmath") when the _fastmath.c file can't be compiled. The fact that the build of PyCrypto fails on David's OpenSolaris machine appears to be a bug in PyCrypto's setup.py. I see from that file that it is checking whether there is a libgmp lib dir present and if so then relying on gmp.h. I infer that on David's OpenSolaris machine that there is a libgmp libdir but not the header files. This could be due to David having the library installed but not the headers installed, which is typical for a packaging system that offers a package of the library separate from the "-dev" package which has the headers. It could also just be a mistake on David's part or (more likely) on the part of the OpenSolaris maintainers. In any case, a more robust way to do this fall-back mechanism in your setup.py file is to attempt to compile the module that you need and then if you get an exception fallback to the pure-Python version. An example of that pattern is simplejson's setup.py. Perhaps the PyCrypto maintainers would like to switch to that pattern by copying the code from simplejson's setup.py. (By the way, this is an example of a more general pattern: Never look before you leap! Never check for good conditions and then proceed if conditions are good. Instead, attempt the thing and then handle failure if it fails. Never look before you leap! Instead just leap without looking and then handle failure gracefully. ;-) Okay so this rule applies better to build scripts than it does to, say, diving in a lake.) Here's a question: just how slow is the slowmath version? If it is too slow for the users of Tahoe-LAFS to rely on in the SFTP implementation, then the fact that PyCrypto does successfully fall back to slowmath on all other systems (that don't have libgmp installed) is a different issue for the Tahoe-LAFS packaging that we also need to deal with... I guess the next step is to write some simple benchmark of the relevant performance which we can run on the buildbots... I'll open a separate ticket for that issue.

[Zooko Wilcox-O'Hearn (zooko)]

Oh I see that when I cut and pasted some text from http://tahoe-lafs.org/trac/tahoe-lafs/ticket/953#comment:17 to here it lost some hyperlinks. In particular, see simplejson's setup.py:

http://code.google.com/p/simplejson/source/browse/trunk/setup.py?spec=svn231&r=230#sl_svn230_63

which has code to fallback to a pure-Python version if a C version fails to compile for any reason. See also this Twisted ticket:

http://twistedmatrix.com/trac/ticket/3586

and this patch attached to that ticket:

http://twistedmatrix.com/trac/attachment/ticket/3586/patch2.txt

Which does the same thing for Twisted.

Dwayne: is the next step for you to try adapting one of these patches to PyCrypto's setup.py? Or is there something else I can do to help?

[Zooko Wilcox-O'Hearn (zooko)]

The operator of the OpenSolaris buildbot for Tahoe-LAFS first uninstalled and then re-installed gmp at our request:

{{{
I installed the SUNWgnu-mp package.

gmp.h is now in /usr/include/gmp/

root@tahoe-test:~# ls /usr/lib/libgmp*
/usr/lib/libgmp.so /usr/lib/libgmp.so.3.4.4 /usr/lib/libgmpxx.so.4
/usr/lib/libgmp.so.3 /usr/lib/libgmpxx.so /usr/lib/libgmpxx.so.4.0.4
}}}

[Zooko Wilcox-O'Hearn (zooko)]

Oh, and the build of PyCrypto still fails, like this:

Searching for pycrypto>=2.0.1
Reading http://pypi.python.org/simple/pycrypto/
Reading http://pycrypto.sourceforge.net
Reading http://www.amk.ca/python/code/crypto
Best match: pycrypto 2.1.0
Downloading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/pycrypto-2.1.0.tar.bz2
Processing pycrypto-2.1.0.tar.bz2
Running pycrypto-2.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-uoZH6d/pycrypto-2.1.0/egg-dist-tmp-5U87GJ
Failure to get the list of managed files from darcs -- if you are building a package with 'setup.py sdist', 'setup.py bdist_egg', or other package-building commands, then the resulting package might be missing some files. If you are not building a package then you can ignore this warning.
In file included from /usr/include/python2.5/Python.h:8,
                 from src/_fastmath.c:32:
/usr/include/python2.5/pyconfig.h:889:1: warning: "_FILE_OFFSET_BITS" redefined
In file included from /usr/include/stdio.h:37,
                 from src/_fastmath.c:30:
/usr/include/sys/feature_tests.h:209:1: warning: this is the location of the previous definition
src/_fastmath.c:34:17: gmp.h: No such file or directory
src/_fastmath.c:37: error: syntax error before "m"
src/_fastmath.c: In function `longObjToMPZ':
src/_fastmath.c:40: error: `mpz_t' undeclared (first use in this function)
src/_fastmath.c:40: error: (Each undeclared identifier is reported only once
src/_fastmath.c:40: error: for each function it appears in.)
src/_fastmath.c:40: error: syntax error before "temp"
src/_fastmath.c:41: error: `temp' undeclared (first use in this function)
src/_fastmath.c:42: error: `temp2' undeclared (first use in this function)
src/_fastmath.c:43: error: `p' undeclared (first use in this function)
src/_fastmath.c:47: error: `m' undeclared (first use in this function)

[Darsey Litzenberger (dlitz)]

Regarding libgmp detection, I looked at the simplejson approach, and wrote an experimental patch for PyCrypto, but I don't like it: I have a machine doing automated builds of PyCrypto every time a new commit gets pushed to the git repo, and this patch would cause actual build failures to be ignored.

Regarding gmp.h being in /usr/include/gmp, it's probably that -I/usr/include/gmp needs to be added to the build process. Of course, gmp.h is under /usr/include on other machines... (Sigh.)

Really, what's needed is some sort of autoconf for Python extension modules, which would allow us to build a small module just for testing that libgmp is available and works, and where it should be included. Do you know of anything like that?

[Zooko Wilcox-O'Hearn (zooko)]

Well, here's a little bit of Python code in pycryptopp's setup.py which does something similar:

http://tahoe-lafs.org/trac/pycryptopp/browser/trunk/setup.py?rev=702#L69

[Zooko Wilcox-O'Hearn (zooko)]

> I have a machine doing automated builds of PyCrypto every time a new commit gets pushed to the git repo, and this patch would cause actual build failures to be ignored.

How could a patch which tries to build libgmp and then falls back to pure-Python cause actual build failures to be ignored? You mean that on some machines you want it to *not* fallback to a pure-Python implementation and instead fail if it can't link to libgmp?

[Darsey Litzenberger (dlitz)]

If _fastmath.c is broken due to a syntax error or some other stupid thing, I want the build to fail.

Why do you care? I honestly don't understand why anyone would *want* to run PyCrypto without libgmp, except maybe on Windows (and PyCrypto builds fine on Windows). Non-GMP PyCrypto is a pain to support---everything in _fastmath.c needs to have a mirrored implementation in pure Python---and usually it just indicates a build misconfiguration. I'm thinking of just dropping support for it altogether.

[Zooko Wilcox-O'Hearn (zooko)]

> If _fastmath.c is broken due to a syntax error or some other stupid thing, I want the build to fail.

I see, so if libgmp isn't installed at all then you want it to fall back to pure-Python math, but if libgmp is installed but the compile fails for some reason then you want the build to fail?

> Why do you care?

I care because I'm a maintainer of Tahoe-LAFS (http://tahoe-lafs.org ) and I want people to be able to install my software easily. Users are encouraged to read the quickstart (http://tahoe-lafs.org/source/tahoe-lafs/trunk/docs/quickstart.html ) which gives a few simple steps to install Tahoe-LAFS.

This quickstart procedure used to work fairly well (not perfectly) on all of our supported platforms in Tahoe-LAFS v1.6.1, but in the Tahoe-LAFS v1.7.0 release we added a dependency on PyCrypto. At that point the quickstart procedure stopped working on *BSD and OpenSolaris, because the quickstart procedure would attempt to build PyCrypto and this attempt would fail on those platforms. The *BSD issue was fixed: https://bugs.launchpad.net/pycrypto/+bug/518852

So now the quickstart procedure works on all platforms except OpenSolaris. If this ticket were fixed then it would work on all platforms.

> I honestly don't understand why anyone would *want* to run PyCrypto without libgmp, except maybe on Windows (and PyCrypto builds fine on Windows). Non-GMP PyCrypto is a pain to support---everything in _fastmath.c needs to have a mirrored implementation in pure Python---and usually it just indicates a build misconfiguration. I'm thinking of just dropping support for it altogether.

My question would be whether the pure-Python version is fast enough for my use. If so, and if it would fall back to the pure-Python version when it can't build the libgmp version (where the subject of this ticket is that it fails to do on on OpenSolaris), then I would like to have it. If it is too slow, or it is too much work to maintain it and make sure it builds right on all platforms, then I would be happy to be rid of it.

[Thorsten Behrens (sbehrens-gmx)]

I've added /usr/include/gmp to the include paths for _fastmath.c as part of the patch I am preparing for Python 3.x / Windows _fastmath support. It's a simple enough change to make.

[Thorsten Behrens (sbehrens-gmx)]

Make that /usr/include. I have some code now that will complain during "setup.py test" if _fastmath exists (it could be built, the file is there) but it can't be imported.Failure to import could be because the dynamic gmp/mpir library can't be found during runtime, or because there was a silly typo in _fastmath.c somewhere.

number.py still falls back silently to slowmath no matter what. The code to detect a failing _fastmath is there but commented out. I think it's friendlier to allow binary dists to fall back to slowmath rather than failing them if _fastmath can't import because the target system lacks gmp, though the build system had it, or a variation on that theme. Like, the build system built on gmp, and the target system has mpir instead, or vice-versa.

Building binary builds with a static gmp/mpir would be the friendly way to go for people who build this stuff. Not that the pycrypto maintainer can influence that.

[Darsey Litzenberger (dlitz)]

Released in PyCrypto 2.4.