test failures on Mac OS X with openssl-0.9.8[gh]

[Zooko Wilcox-O'Hearn]

Thanks for maintaining pyOpenSSL!

I get the following test failures when I run

PYTHONPATH=./build/lib.macosx-10.3-i386-2.5/ trial OpenSSL

[zooko (zooko-tahoe-trac)]

The guy who made the tracdarcs plugin work is K. S. Sreeram. I was idly
looking for alternate python crypto modules when I found ncrypt by K. S.
Sreeram. Then I saw that ncrypt is sponsored by a p2p company, tachyon.in,
which also makes a secure decentralized (?) instant messaging protocol:

It has a very nice straightforward explanation up front:

http://cspace.in/

I was thinking that it might be a nice optional underlay protocol for Foolscap.

Unfortunately it is GPL'ed, so it is a non-starter for Allmydata unless
tachyon.in wants to give us a more permissive licence.

But the OpenSSL Python wrappers that they wrote are permissively licensed:

http://tachyon.in/ncrypt/

And it works well on Windows:

http://tachyon.in/pipermail/ncrypt-users/2007-February/000016.html

Here's K. S. Sreeram's page:

http://sreeram.cc/

So all Python crypto libraries that I know of that do TLS and that have
compatible licences:

{{{
pyOpenSSL
tlslite
M2Crypto
ncrypt
}}}

I vaguely remember that Brian Warner investigated tlslite and had trouble with
it. I've heard bad things about M2Crypto. I would be interested in trying
ncrypt.

By the way, I was reminded while doing this browsing that we need to add "the
OpenSSL+GPL exception" to our licence.

http://en.wikipedia.org/wiki/OpenSSL#The_exception

[zooko (zooko-tahoe-trac)]

I don't like pycrypto either. The official version's ctr mode is too slow, and AMK didn't accept my patch to speed it up. I never received a rejection letter either.

Actually, I *do* kind of like pycrypto, but it doesn't do TLS, and so we can probably replace both of our libraries -- pyOpenSSL and pycrypto -- with one which provides all the crypto functions we need and which is also portable and maintained and so on.

[zooko (zooko-tahoe-trac)]

We've added the OpenSSL exception to our licence.

[zooko (zooko-tahoe-trac)]

There appear to be people hacking on pycrypto other than AMK, as discovered by launchpad. I reported bugs on launchpad.

https://bugs.launchpad.net/pycrypto/+bug/112423

https://bugs.launchpad.net/pycrypto/+bug/112422

[evilrob (evilrob-tahoe-trac)]

fwiw the allmydata.com 'ext' repository has a py24 native build of pyopenssl which was also found from trawling the web

[evilrob (evilrob-tahoe-trac)]

(which I suspect means that we don't have a py25 requirement on windows because of this dependency)

[zooko (zooko-tahoe-trac)]

See also:

http://twistedmatrix.com/trac/ticket/2293

The Twisted folks are planning to fork pyOpenSSL.

[warner (warner-tahoe-trac)]

You know, we could probably ditch pycrypto altogether if we just copied implementations of AES-CTR and SHA-256 into our tree. We'll want RSA sooner or later but that can't be all that big. We certainly don't need any of the other block ciphers or hash algorithms that pycrypto offers.

And if we required python2.5 (which I'm !!!not!!! advocating) then it comes with SHA-256 in the batteries-included 'hashlib' module..

[warner (warner-tahoe-trac)]

I've copied AES-CTR and SHA-256 into our tree, and removed pycrypto (and src/Crypto) altogether.

I also copied RSA in there too, but I've disabled it in setup.py because it requires the GMP package, and I don't want to add another dependency to tahoe until we actually need it (say, when we need RSA for distributed dirnodes and SSK files).

[zooko (zooko-tahoe-trac)]

Our new mutable file design (source:docs/mutable.txt) requires public key cryptography.

The aforementioned (comment:6) thing about someone maintaing pycrypto hasn't panned out -- nobody responded to the patches I submitted.

Oh boy, and now I see that the current version of pycrypto -- v2.0.1 -- has *another* bug which causes SHA-256 to give incorrect results:

http://sourceforge.net/tracker/index.php?func=detail&aid=1740198&group_id=20937&atid=120937

This bug report and accompanying patch has been open since June. This is another demonstration that pycrypto is unmaintained.

This also raises the question: why are we copying our hash function code from pycrypto ? Let's copy hashlib from python 2.5 instead.

Likewise, I'm a bit reluctant to depend on the RSA implementation from pycrypto.

I would be delighted if someone would make a Python wrapper around [http://cryptopp.com Crypto++].

I might try it myself.

Crypto++ has the following features:

 * actively maintained by Wei Dai, who is very smart
 * very portable (see the portability matrix on the front page)
 * high quality code -- the first ever open source sofware to get FIPS 140-2 certification, for example
 * all the algorithms we could ever want, including Tiger hash, elliptic curve signatures, salsa-20, ...
 * extremely high-performance (assembly-implementation) versions of many of the algortihms
 * high-performance (C or C++-implementation) versions of all of the algorithms

It has the following drawback:

 * C++, and not your typical "subset of C++" either, but the real deal with cleverly parameterized templates pouring out of its ears

There are so many ways to make Python wrappers nowadays:

 * hand-rolled
 * pyrex
 * ctypes
 * SWIG

[zooko (zooko-tahoe-trac)]

Apparently the OLPC project has created Python wrappers around libtomcrypt (which is the upstream source for both the pycrypto sha256 and the Python standard library hashlib sha256), but hasn't really packaged or publicized these wrappers:

http://wiki.laptop.org/go/Software_projects/lang-ja#pyltc

Also some person named Larry contributed incomplete python wrappers for libtomcrypt in March of this year:

http://libtom.org/?page=index&newsitems=10000&whatfile=crypt

[warner (warner-tahoe-trac)]

The OLPC wrapper code is [http://dev.laptop.org/git?p=projects/pyltc;a=tree here]. Any idea what
the license is?

Also, it looks like they've got ECC wrappers..

[zooko (zooko-tahoe-trac)]

Oh and just for completeness, there is also a 5th way to wrap C++ code in Python code -- boost.python. Truly, we enjoy an abundance of ways to wrap C/C++ in Python...

[zooko (zooko-tahoe-trac)]

Oh and just for more complete completeness, there is also cython.

So that's seven Ways To Do It.

But I'm using the hand-rolled technique, as per

http://allmydata.org/pipermail/tahoe-dev/2007-October/000215.html

[zooko (zooko-tahoe-trac)]

See also ticket #199.

[zooko (zooko-tahoe-trac)]

Itamar pointed out that M2crypto has been integrated with Twisted, but on the other hand, Guido van Rossum had bad experiences with M2crypto:

http://www.artima.com/forums/flat.jsp?forum=106&thread=95863

[zooko (zooko-tahoe-trac)]

Removing "pycrypto" from the subject line of this ticket, since we have removed our dependency on it by switching to pycryptopp. (This fix isn't committed to trunk yet, but I want to point people at this ticket who are interested in pyOpenSSL specifically.)

[heikki (heikki-tahoe-trac)]

Replying to [comment:21 zooko]:
> Itamar pointed out that M2crypto has been integrated with Twisted, but on the other hand, Guido van Rossum had bad experiences with M2crypto:
>
> http://www.artima.com/forums/flat.jsp?forum=106&thread=95863

I'd like to point out that Guido wrote that almost three years ago! As far as I know, all the issues he experienced were fixed long time ago.

Itamar is also correct: M2Crypto has a Twisted protocol wrapper, which can be used to do SSL instead of pyOpenSSL. It was modeled after similar thing in TLS Lite. We use Twisted in Chandler, and M2Crypto does the SSL part using this wrapper. More on Chandler at http://chandlerproject.org.

If you try M2Crypto and run into any issues, please let me know. I can't fix issues I don't know about. You can find the mailing list and bugzilla info on the M2Crypto homepage at http://chandlerproject.org/Projects/MeTooCrypto.

Heikki Toivonen - M2Crypto maintainer

[zooko (zooko-tahoe-trac)]

Heikki:

Thanks for the post. If M2Crypto is actively maintained, then this is a big advantage that it has over pyOpenSSL!

We could use M2Crypto for both our SSL needs and our filesystem crypto needs, thus removing the need for pyOpenSSL and removing the need for pycryptopp. (This latter part makes me a little sad because I like pycryptopp -- it is my newest baby.)

Brian: are you interested in using M2Crypto for foolscap's SSL layer?

[zooko (zooko-tahoe-trac)]

[yassl http://yassl.com/] is an implementation of SSL written on top of Crypto++. It is licensed under the traditional dual-licence model of "You can use it under the GPL, or you can ask us for a special licence.". I have already asked them for a special licence, and I'm waiting to hear back from them.

[zooko (zooko-tahoe-trac)]

Heikki Toivonen, the M2Crypto maintainer, posted the following comment [http://article.gmane.org/gmane.comp.python.cryptography/494 in a discussion]: "I suspect most
people use something more robust than plain M2Crypto for a server
application (like Apache or Twisted)."

This makes me think that M2Crypto might not be best for allmydata.org.

[zooko (zooko-tahoe-trac)]

The aforementioned discussion leads to two further options:

1. http://www.brontes3d.com/opensource/straightssl.html

yet another openssl wrapper

2. http://pypi.python.org/pypi/ssl

This is a backport of the SSL implementation that is intended to be standard in Python >= v2.6.

[zooko (zooko-tahoe-trac)]

Also, allmydata.com has decided that it would be okay to go ahead and use GPL'ed source code such as yassl, so that opens up some more options.

[zooko (zooko-tahoe-trac)]

Oh, by the way, I forgot to mention that the reason I am looking at this ticket is that I can't compile pyOpenSSL 0.6 on Mac OS X against openssl-0.9.8g. There is a callback defined in openssl that takes (const SSL*, int, int), and pyOpenSSL passes a function that takes (SSL*, int, int). Patching pyOpenSSL to add the const keyword makes the gcc warning/error go away, but when I try to import it I still get:

{{{
HACK wonwin-mcbrootles-computer:~/playground/pyOpenSSL/pyOpenSSL-0.6$ python -c 'import OpenSSL'
Traceback (most recent call last):
  File "<string>", line 1, in ?
  File "/usr/local/stow/pyOpenSSL-0.6/lib/python2.4/site-packages/OpenSSL/__init__.py", line 11, in ?
    import rand, crypto, SSL, tsafe
ImportError: Failure linking new module: /usr/local/lib/python2.4/site-packages/OpenSSL/SSL.so: Symbol not found: _SSL_CTX_set_info_callback
  Referenced from: /usr/local/lib/python2.4/site-packages/OpenSSL/SSL.so
  Expected in: dynamic lookup
}}}

[heikki (heikki-tahoe-trac)]

Replying to [comment:26 zooko]:
> Heikki Toivonen, the M2Crypto maintainer, posted the following comment [http://article.gmane.org/gmane.comp.python.cryptography/494 in a discussion]: "I suspect most
> people use something more robust than plain M2Crypto for a server
> application (like Apache or Twisted)."
>
> This makes me think that M2Crypto might not be best for allmydata.org.

It really depends on what you need. If your website is not a high traffic site, M2Crypto is probably ok. But if you need traffic shaping, load balancing, guaranteed high availability etc. then I believe none of the simple libraries will be robust enough for you. That is why I mentioned Apache etc. which certainly can handle high traffic sites.

[warner (warner-tahoe-trac)]

FYI, I started playing with a port of Foolscap to M2Crypto. The Twisted interface seems pretty well implemented, but the way that you get access to the certificate (and the way you control validation) is pretty different, so I haven't gotten it working yet. When I get back next week I'll post my results and maybe beg Heikki for some help :).

Most of what we need is just reactor.connectTCP, transport.startTLS(), and access to the certificate so we can validate it according to Foolscap's rules instead of the whole CA business (basically: the hash of the cert must match the server you were intending to connect to, no more, no less). My biggest concern about pyopenssl is an apparent memory consumption problem (something like 100kB to 1MB per open connection, although I haven't re-run those tests in a few years). If M2Crypto has comparable speed to pyopenssl, and it isn't too hard for users to obtain/build/install, then I'll seriously consider moving foolscap over to it.

[heikki (heikki-tahoe-trac)]

Replying to [comment:31 warner]:
> FYI, I started playing with a port of Foolscap to M2Crypto. The Twisted interface seems pretty well implemented, but the way that you get access to the certificate (and the way you control validation) is pretty different, so I haven't gotten it working yet. When I get back next week I'll post my results and maybe beg Heikki for some help :).

M2Crypto is used in Chandler, and since the certificates are stored in a database in Chandler, the validation is a bit different. Also, Chandler will present the user with a dialog if there are errors in the SSL connection (the users can choose to ignore these), so this presents additional challenges for the SSL implementation. You can see how it is done in here:

http://svn.osafoundation.org/chandler/trunk/chandler/parcels/osaf/framework/certstore/ssl.py

[zooko (zooko-tahoe-trac)]

Rejoice -- exarkun and bigdog are working on pyOpenSSL. That makes the path of least resistance for us (continuing to use pyOpenSSL) also be the path of future promise, since exarkun is an excellent engineer.

[zooko (zooko-tahoe-trac)]

binary builds from exarkun and company:

http://buildbot.twistedmatrix.com/builds/

[zooko (zooko-tahoe-trac)]

Okay, I like {{{pyOpenSSL}}} now since exarkun et alia are maintaining it. Closing as, um, "fixed".

[zooko (zooko-tahoe-trac)]

Re-opening this since I get unit test failures when I try to use the current pyOpenSSL-0.7 with Tahoe, and different unit test failures when I run pyOpenSSL-0.7's own unit tests. Here's the bug report for the pyOpenSSL project:

https://sourceforge.net/tracker/index.php?func=detail&aid=1978729&group_id=31249&atid=401758

[zooko (zooko-tahoe-trac)]

Another reason why I still don't like pyOpenSSL, and therefore this ticket should remain open, is ticket #402 (bug in Twisted, triggered by pyOpenSSL-0.7).

[Zooko Wilcox-O'Hearn (zooko)]

Thanks for maintaining pyOpenSSL!

I get the following test failures when I run

PYTHONPATH=./build/lib.macosx-10.3-i386-2.5/ trial OpenSSL

[Zooko Wilcox-O'Hearn (zooko)]

By the way, here is the ticket at http://allmydata.org that references
this ticket:

http://allmydata.org/trac/tahoe/ticket/11

[Zooko Wilcox-O'Hearn (zooko)]

I tried it with the brand spankin' new openssl-0.9.8h and got similar errors.

I hereby volunteer a Mac OS X buildslave for pyOpenSSL.

[Zooko Wilcox-O'Hearn (zooko)]

Changing the name to reflect that it isn't specific to one version of openssl.

[Jean-Paul Calderone (exarkun)]

Find me on IRC sometime and I'll get you credentials for a slave.

I wonder if this is as 10.3-specific issue (somehow)? http://buildbot.twistedmatrix.com/waterfall-pyopenssl includes a 10.5 slave which is currently green.

[Zooko Wilcox-O'Hearn (zooko)]

Okay, when I uninstall the openssl-0.9.8h that I compiled myself, and add the following stanza to pyOpenSSL's setup.cfg file, then it compiles and passes tests:

--- old-dw/setup.cfg 2008-06-20 09:26:42.000000000 -0700
+++ new-dw/setup.cfg 2008-06-20 09:26:42.000000000 -0700
@@ -8,3 +8,6 @@
 group = Development/Libraries
 build_script = rpm/build_script
 doc-files = doc/pyOpenSSL.txt doc/pyOpenSSL.ps doc/html
+
+[build_ext]
+include-dirs = /Developer/SDKs/MacOSX10.4u.sdk/usr/include/

So the openssl that comes with Mac OS 10.4 works, and openssl-0.9.8h works on other platforms, but openssl-0.9.8h, as I compiled it, yields these errors. For reference, here is the command that I used to configure and compile openssl-0.9.8h:

./config -shared --prefix=/usr/local/stow/openssl-0.9.8h && make && make check && make install

[Zooko Wilcox-O'Hearn (zooko)]

I spelled that wrong. The actual command line is:

./config -shared --prefix=/usr/local/stow/openssl-0.9.8h && make && make test && make install

[Zooko Wilcox-O'Hearn (zooko)]

Okay, we have a builder showing this compile failure:

http://buildbot.twistedmatrix.com/builders/osx10.4-pyopenssl/builds/16/steps/compile/logs/stdio

Per my earlier experiment, if we can configure pyOpenSSL somehow so that it adds /Developer/SDKs/MacOSX10.4u.sdk/usr/include/ to the include dirs on Mac OS X then I think this build will start working and we'll see tests.

[Jean-Paul Calderone (exarkun)]

I changed the master configuration to set CFLAGS to include that directory. The build succeeded:

http://buildbot.twistedmatrix.com/builders/osx10.4-pyopenssl/builds/20

[zooko (zooko-tahoe-trac)]

Okay, now I like pyOpenSSL. See also #456 (it would be nice if the dependency on OpenSSL could be automatically resolved), but basically ticket #11 can finally be closed, thanks to JP Calderone and bigdog's stewardship of pyOpenSSL.