expose SSL_CTX_set_client_CA_list

[Glyph Lefkowitz]

Right now in order to set the client CA list, you have to load a client CA file.

http://twistedmatrix.com/trac/ticket/2061

[Ziga Seilnacht (zseil)]

Here is a branch that impements this functionality by adding the add_client_CA and set_client_CA_list methods to SSL.Context.

In addition, I also exposed the SSL_get_client_CA_list as get_client_CA_list method of SSL.Connection objects. Without this testing wouldn't be possible.

The branch also contains some cleanup commits and commits with additional error checks. If you think that they need to be separated from this feature request, I'll open a new bug report.

The only thing missing are the docs, I'll work on that tomorrow. Otherwise, the branch should be ready for review.

[Jean-Paul Calderone (exarkun)]

I made a few changes in my client_CA branch, mostly test improvements. Then I accidentally merged this before testing on Windows (I forgot my pyOpenSSL trunk was a checkout). Unfortunately the branch introduced some serious problems into the test suite on Windows. I've tried for a couple days to track down the cause, but have had no luck so far. Any help would be appreciated.

I backed out the branch so that trunk is in a good state. Launchpad doesn't seem to have taken note of this, though.

[Ziga Seilnacht (zseil)]

Hum, I did all my development on Windows, and tests were passing for me. I also tested with your latest changes a few minutes ago, and all tests are still passing for me. My current setup is:
 - Windows XP Professional SP3
 - Python 2.5.2 installed from the official installer
 - Visual Studio .NET 2003
 - OpenSSL 0.9.8k (configured with -zlib and built with Visual Studio)

Due to differences between my OpenSSL installation and the binary installer, I use the following patch to the setup script:

=== modified file 'setup.py'
--- setup.py 2009-07-24 13:37:38 +0000
+++ setup.py 2009-11-10 16:51:59 +0000
@@ -57,9 +57,10 @@
     msvccompiler.MSVCCompiler = makeTellMeIf(msvccompiler.MSVCCompiler, ['libeay32', 'ssleay32'])

     import shutil
- shutil.copy("C:\\OpenSSL\\ssleay32.dll", os.path.split(os.path.abspath(__file__))[0])
- shutil.copy("C:\\OpenSSL\\libeay32.dll", os.path.split(os.path.abspath(__file__))[0])
- package_data = {'': ['ssleay32.dll', 'libeay32.dll']}
+ shutil.copy("C:\\OpenSSL\\bin\\ssleay32.dll", os.path.split(os.path.abspath(__file__))[0])
+ shutil.copy("C:\\OpenSSL\\bin\\libeay32.dll", os.path.split(os.path.abspath(__file__))[0])
+ shutil.copy("C:\\OpenSSL\\bin\\zlib1.dll", os.path.split(os.path.abspath(__file__))[0])
+ package_data = {'': ['ssleay32.dll', 'libeay32.dll', 'zlib1.dll']}
 else:
     Libraries = ['ssl', 'crypto']
     package_data = {}

I run the tests by executing each test module separately, e.g:

> cd test
> python test_crypto.py

Do you have a log of a failed test run?

[Jean-Paul Calderone (exarkun)]

Yep, you can find them on the pyOpenSSL buildbot, <http://buildbot.twistedmatrix.com/waterfall-pyopenssl>. You can restrict the view to just builds of the client_CA branch: <http://buildbot.twistedmatrix.com/waterfall-pyopenssl?branch=client_CA>. If you scroll down a bit (unfortunately these results will continue to move further and further down the page until they disappear) you'll see some purple results on the Windows builders. Click the stdio link and you'll see the test results. What's not visible in those logs is that a "Do you want to debug this process" dialog pops up at the point where the suite hangs. I'm using discover.py to run the full suite. I tried with a debug build of Python and had the crash/hang happen at a different place, so it seems possible or likely that there is some memory corruption going on somewhere which is not immediately fatal.

I'm also unable to reproduce these results on a different Windows machine I have, or on Linux. Valgrind spews tons of warnings for the test suite, I don't know how many of them are spurious because of things CPython does or if some of them are indicating real problems. :/

[Ziga Seilnacht (zseil)]

I managed to reproduce the problem on my computer. It looks like the bug comes from MinGW's linker, it can't resolve the
SSL_CTX_add_client_CA symbol. The problem can also be seen on buildbots, in the first compile step:

Info: resolving _SSL_CTX_add_client_CA by linking to __imp__SSL_CTX_add_client_CA (auto-import)

I did build this branch with MinGW in the past, and I'm almost certain that I ran the tests on it, but I had some local modifications to the setup script. I'll try to see if any of those changes make a difference.

[Ziga Seilnacht (zseil)]

Ok, I managed to build a working PyOpenSSL by renaming the ssleay32.a library in the MinGW library dir to libssl32.a and adding that dir to the lib path:

> copy C:\openssl\lib\MinGW\ssleay32.a C:\openssl\lib\MinGW\libssl32.a
> python setup.py build_ext -c mingw32 -I C:\openssl\include -L C:\openssl\lib\MinGW

With these changes the tests pass, but I don't know if it is worth the effort, we could also just remove the Context.add_client_ca() method.

[Ziga Seilnacht (zseil)]

Here is a patch for the setup script that further automates the build on Windows, and works around the bug in MinGW.

With it, you can use the following combinations for building PyOpenSSL:

 - OpenSSL from binary installer and MinGW or MSVC as the compiler
 - OpenSSL built with MSVC and MSVC
 - OpenSSL built with MinGW and MinGW

There are some other improvements in the patch, OpenSSL's path is not hardcoded anymore and OpenSSL's shared libraries get packaged even on Python 2.3. It also tries to guess if OpenSSL was linked with zlib, but that part can easily be thrown out.

I tested the patch on Windows XP, in the following combinations:

Python 2.3: OpenSSL(installer) + MinGW, OpenSSL(MinGW) + MinGW
Python 2.4: OpenSSL(installer) + MSVC + `from setuptools import setup`
Python 2.5: OpenSSL(installer) + MinGW, OpenSSL(MinGW) + MinGW
            OpenSSL(installer) + MSVC, OpenSSL(MSVC) + MSVC

The builds finished without problems and all tests passed. The patch could probably use some cleanups, but it seems to work.

[Ziga Seilnacht (zseil)]

Ups, I accidentally left the setuptools import in the patch. Please use the latest changeset from the client_CA instead, I've just pushed the change to Launchpad.

[Jean-Paul Calderone (exarkun)]

Which versions of MSVC did you use to build OpenSSL?

Also, it'd be great if you could push the setup.py changes to a new branch of trunk. I'd like to consider them separately from the client ca code.

Thanks a lot.

[Jean-Paul Calderone (exarkun)]

Hi Žiga,

Just a follow-up. I did some testing with the client_CA branch. I built OpenSSL with VS.NET2003 and then built pyOpenSSL for Python 2.3, 2.4, and 2.5 with the same. This all worked, and the tests for all the resulting builds passed. I also used the slproweb.com OpenSSL installer (what I've actually been using for all versions of Python so far) and built pyOpenSSL with VS2008 for Python 2.6 against that. This also worked and produced a passing test suite.

I wasn't able to build pyOpenSSL for Python 2.3, 2.4, or 2.5 with mingw, though. This was the failure:

  gcc: C:\OpenSSL-VS.NET2003\lib\MinGW\ssleay32.a: No such file or directory
  gcc: C:\OpenSSL-VS.NET2003\lib\MinGW\libeay32.a: No such file or directory

I expect this is because building OpenSSL with VS doesn't produce the pieces mingw requires, though. At this point, while I'd be sad to see mingw support go, I'm sick enough of Windows build issues to not feel strongly enough to do much about it. Working VS support is sufficient to provide all the functionality that users have required (ie, a binary Windows installer).

[Ziga Seilnacht (zseil)]

I used Visual Studio .NET 2003 (MSVC 7.1). If you are asking because of the recent buildbot failures, those are caused by buildbot misconfiguration. The library path for build_ext should also be changed when mingw was replaced with msvc (from C:\openssl to C:\openssl\lib).

I pushed the setup change in a sepparate branch, you can get it from here:
lp:~ziga-seilnacht/pyopenssl/windows-setup

[Ziga Seilnacht (zseil)]

Regarding your last comment, yes, with the windows-setup branch you have to build PyOpenSSL with the same compiler that was used for building OpenSSL or use the slproweb.com binary installer, which should work with both of them.

[Jean-Paul Calderone (exarkun)]

Wonderful. Thank you for the windows-setup branch. I've merged it and the client CA branch, and my buildbot is happy. :)

[Ziga Seilnacht (zseil)]

Great, thank you for working on this!

Let me know if you want to get the Python 2.3 Windows buildbot in shape. I think that the problem lies in the Visual Studio
version mismatch, Python 2.3 was compiled with Visual Studio 6.0. Switching to the MinGW compiler should work if the
buildbot has slproweb.com's OpenSSL installed.