please provide binaries

[Zooko Wilcox-O'Hearn]

Please provide binary packages of pyOpenSSL so that people who don't have OpenSSL dev headers installed on their system (or perhaps even don't have a C compiler installed) can install pyOpenSSL. For my own needs right now, binary .egg packages are sufficient, and fortunately they are very easy to build and upload to pypi -- simply execute the following command at the end of your successful unit test runs in the buildbot master.cfg:

{{{
$PYTHON ./setup.py bdist_egg upload
}}}

Other binaries can also be built and uploaded with similar ease, for example:

{{{
$PYTHON ./setup.py bdist_dumb upload
}}}

Although I am not sure if anyone actually wants dumb binaries.

Finally, there is a tool to automate the production of .deb's from setup.py's named {{{stdeb}}}. I know that JP has tried it out. I have written some patches to stdeb and used it on several Python packages and it seems to be working out okay.

Thanks!

[warner (warner-tahoe-trac)]

We should also note that many debian platforms don't have valid egg-info files for their pyopenssl installations. dapper and etch have none at all, edgy has an egg-info for py2.5 only, and gutsy has a "pyOpenSSL.egg-info" (as opposed to the "pyOpenSSL-0.6.egg-info" file that we would expect).

So many of our target platforms will be unable to tell that pyopenssl is installed, and are therefore likely to try to build it anyways. This is unfortunate, as a binary debian package of python-openssl or foolscap (with all of its recommended dependencies) should be sufficient.

We might want to revert the foolscap change that declares this dependency.. it may cause more harm than good.

[Zooko Wilcox-O'Hearn (zooko)]

Ooh, actually there are a couple of reasons why it wouldn't be this easy.

First of all, "bdist_egg" isn't a command of distutils, so we would have to patch pyOpenSSL to use setuptools for that purpose. Secondly, it looks like such bdists are dynamically linked against libcrypto and/or libssl, at least on Mac OS X -- I haven't looked at other platforms yet. Therefore, while doing this *would* relieve users of having to have openssl development headers installed, it would not relieve them of the need to have the right version of libssl and libcrypto .so's installed.

So, further study is warranted...

[zooko (zooko-tahoe-trac)]

The error on the Ubuntu/Gutsy system turned out to be because I had installed [http://www.yassl.com/ yassl] there. Uninstalling yassl made it so that pyOpenSSL could build.

[zooko (zooko-tahoe-trac)]

The same was true on our dapper buildslave.

[warner (warner-tahoe-trac)]

So, I don't know how to fix this. setuptools is not APT, and knows nothing about C libraries. I don't think it is likely or reasonable to expect that the pyopenssl authors will put a full copy of !OpenSSL in their source tree.

Personally, I think this is going to be a "root dependency": something that isn't going to get auto-installed, and we should just document the requirement and oblige the developer to get it installed somehow.

[zooko (zooko-tahoe-trac)]

Changing this from "defect" to "enhancement", and changing title from "pyopenssl depends on non-Python-packaged thing (OpenSSL)" to "it would be nice if the dependency on OpenSSL could be automatically resolved".

We don't know how to solve this nicely, but I would like to leave this ticket open to reflect the fact that I wish it were better.

On Mac OS X, OpenSSL with development headers is installed by default (if you've installed the Mac OS X developer tools), so it isn't a problem on that platform.

Note that we used to have a similar problem with the Crypto++ library, and I "solved" it by including the Crypto++ source code inside the pycryptopp package and building all of it when pycryptopp is built. I doubt that the pyOpenSSL developers will want to do the same thing. (And indeed, this has raised a different problem for pycryptopp: how to facilitate users linking against their own libcryptopp ''instead'' of the version of Crypto++ which is included by source code inside pycryptopp. This would be required by Debian to include pycryptopp in Debian, for example.)

Another possible solution, if you aren't on a platform that comes with OpenSSL lib and headers, and if pyOpenSSL isn't willing to bundle OpenSSL source code itself, would be to stop using pyOpenSSL! There are three reasons why this might be a valid future direction: 1. Tahoe actually doesn't need confidentiality or authentication from its transport layer, for the most part. (There are some cases where it does, but perhaps in the future those will go away.) 2. Foolscap might start using a different crypto layer than SSL. 3. Foolscap might start using a Pythonic SSL implementation.

[Jean-Paul Calderone (exarkun)]

eGenix provides binary packages: http://www.egenix.com/products/python/pyOpenSSL/

For the time being, I think that's sufficient.

[zooko (zooko-tahoe-trac)]

Oh, here is yet another way that the dependency on OpenSSL could be semi-automatically resolved for some people: if binaries of pyOpenSSL which contained the OpenSSL lib itself were automatically installabe on their platforms. [https://bugs.launchpad.net/pyopenssl/+bug/238658 pyOpenSSL bug#238658] says that egenix is now distributing binaries of pyOpenSSL 0.7.

At some point I will investigate whether those pyOpenSSL binaries can satisfy our dependencies on various platforms.

[cgalvan (cgalvan-tahoe-trac)]

It is definitely possible to distribute pyOpenSSL eggs along with the OpenSSL binaries, as this is the approach that my company has taken for our python distribution. The problem you may run into is binary incompatibility between different Linux distros, since currently the platform extension for an egg on Linux would be 'linux_x86_64', for example, no matter what specific distro you were on.

[cgalvan (cgalvan-tahoe-trac)]

Also, even if you can't get these eggs pushed to PyPi, you can add dependency links in your setup.py, which would allow you to host them from your own repo.

[zooko (zooko-tahoe-trac)]

Has your company built eggs for OpenSSL?

[cgalvan (cgalvan-tahoe-trac)]

Yes, we have built eggs for OpenSSL, although we mainly use them as build dependencies so that other packages can find them when they are being built.

[zooko (zooko-tahoe-trac)]

Hm... Would you be interested in supporting binary eggs of pyOpenSSL? By which I mean building them and uploading them to pypi. See this wishlist item on the pyOpenSSL tracker: https://bugs.launchpad.net/pyopenssl/+bug/238658 (please provide binaries)

[Chris Galvan (cgalvan)]

I agree with zooko. It would be best if there were eggs for pyOpenSSL so that people could depend on/install it easily. I work for a company that distributes eggs for pyOpenSSL across many different platforms and so I would be willing to help submit my existing patch for pyOpenSSL's setup.py that allows this to work. We currently distribute eggs for win32, universal os x 10.3-10.5 and red hat 3. The red hat binary would not work with setuptools as-is, since it needs to run a post-install script at the end to fix some rpaths, but on the other platforms I mentioned it works just fine. Who would I talk to about fixing this?

[cgalvan (cgalvan-tahoe-trac)]

Yeah, I just responded to the ticket you mentioned above. At the moment, I'd be able to provide binary(eggs) for win32 a universal egg for OS X 10.3+.

I have also built eggs for some Linux distros(red hat and ubuntu), but they require post-install scripts to be run in order to fix up the rpaths in the binaries, which is not supported natively by setuptools ATM :/ The other problem with providing binary eggs for Linux platforms is that setuptools currently brands all binary linux eggs with <egg-name>-linux-<architecture>.egg, so there would be binary incompatibilities. I have a patch to make the platform branding more specific in the egg names, but it hasn't been looked at yet :/

[Jean-Paul Calderone (exarkun)]

Would the binary eggs be statically linked or dynamically linked? Or would they include both? Does this cause problems if users install a binary egg without a compatible version of OpenSSL installed?

[Chris Galvan (cgalvan)]

On Windows, the binaries are statically linked, which is why we ship the necessary openssl runtime .dll's inside the egg, which allows the extension to find it without the binaries having to be on the system PATH. On OS X, they are also dynamically linked against the system ssl libraries, which I know for a fact are binary compatible between OS X 10.4 and 10.5(intel and ppc), because I had to test my pyOpenSSL egg on all of these platform combinations.

[Chris Galvan (cgalvan)]

On Windows there would not be any problems since the openSSL binaries would be inside the egg itself, and on OS X there would only potentially be problems if they overwrite the standard openSSL libraries that come with the system. If someone had a different version of openSSL that they got through fink or macports it wouldn't interfere with the pyOpenSSL egg.

[Jean-Paul Calderone (exarkun)]

I'd be happy to look at your setup.py patch. I'd rather not make setuptools *required* by pyOpenSSL, but optionally using it for things like building binary eggs is fine. You can attach the patch here or push a branch of trunk which includes it.

[Chris Galvan (cgalvan)]

Here is my patch to take advantage of setuptools in order to create Windows binaries. Sorry about the late response, I have been rather swamped lately :) Let me know what you think.

[cgalvan (cgalvan-tahoe-trac)]

I submitted a patch that will aid in providing binaries for win32 and OS X and it is currently being reviewed.

[zooko (zooko-tahoe-trac)]

Great!

[Zooko Wilcox-O'Hearn (zooko)]

I glanced at the patch. It is nice and simple.

[Jean-Paul Calderone (exarkun)]

One problem this introduces for me is that it breaks `setup.py install --prefix ...´. This makes it harder to test pyOpenSSL (and it's already too hard, I need to start making it easier).

Is there some way to prevent this from happening?

exarkun@boson:~/Projects/pyOpenSSL/trunk$ python setup.py install --prefix /tmp/testing
running install
Checking .pth file support in /tmp/testing/lib/python2.5/site-packages/
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 2] No such file or directory: '/tmp/testing/lib/python2.5/site-packages/test-easy-install-3426.pth'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /tmp/testing/lib/python2.5/site-packages/

This directory does not currently exist. Please create it and try again, or
choose a different installation directory (using the -d or --install-dir
option).

[Zooko Wilcox-O'Hearn (zooko)]

You can add "--single-version-externally-managed" and "--record=anyfilename.txt" to the command-line.

The setup.py could be hacked to added those options automatically.

[Zooko Wilcox-O'Hearn (zooko)]

{{{
import sys
sys.argv.append("--single-version-externally-managed")
sys.argv.append("--record=anyfilename.txt")
}}}

[Zooko Wilcox-O'Hearn (zooko)]

Hello, I would like to let people who want to use allmydata-tahoe on Windows do so without having to manually install OpenSSL and pyOpenSSL themselves. It appears that Chris Galvan's patch would make this work for me. What can I do to accelerate the acceptance of this patch?

If you like, I will write a patch to pyOpenSSL's setup.py so that if someone passes "--prefix=$FOO" as an argument to "install" or "develop", it automatically does the "mkdir -p $FOO/lib/python$VER/site-packages" and the insertion of $FOO/lib/python$VER/site-packages into the sys.path. Or if you like I will make it so that when someone does "--prefix=$FOO" then it automatically does "--single-version-externally-managed" and "--record=log-of-installed-files.txt".

Or, I would be willing to help in some other way.

[Zooko Wilcox-O'Hearn (zooko)]

Here's my proposed code: https://code.launchpad.net/~zooko/pyopenssl/buildbinaries/+merge/1640

[Zooko Wilcox-O'Hearn (zooko)]

Note that I have submitted a patch for setuptools -- http://mail.python.org/pipermail/distutils-sig/2008-November/010488.html -- that would render this "tweak the sys.path" behavior unnecessary. However, could we please go ahead and build pyOpenSSL binaries for Windows so that the next release of Tahoe (which will be called tahoe-lafs-1.3.0), will be installable on Windows?

I would be willing to do some more work to make this happen, such as perhaps writing a buildbot config script to build binaries and upload them to pypi whenever a bzr commit happens.

I would even (ick) be willing to *manually* build a binary and upload it to pypi, at least this one last time (because hopefully in the future such icky manual labor will be automated).

[Jean-Paul Calderone (exarkun)]

The slave which builds two thirds of the Windows pyOpenSSL packages is currently offline. I hope that it will return very soon, but I can't build all of the pyOpenSSL Windows packages until it comes back.

However, this isn't the main obstacle here. Two issues have prevented me from doing anything on this ticket so far:

  - I haven't had a lot of time to work on pyOpenSSL lately
  - Windows isn't my primary platform and I don't have a good understanding of the implications of this change.

You've done a lot to help out with the first issue (by providing patches and a bzr branch) but relatively little has changed since this ticket was filed regarding the second point. Can you do anything about it? I am concerned about several things, although since my understanding of the issue isn't great, I'm also concerned that I'm missing being concerned about something important. In any case:

  - How does this change the experience for a Windows user who was previously satisfied with the current style of pyOpenSSL Windows installers?
  - What are the implications for the importance of the version of OpenSSL (not pyOpenSSL) installed on the host which builds the Windows installer for pyOpenSSL? I have very little control over these machines and I have the (unsubstantiated) feeling that this change makes the version of OpenSSL installed on them more important.
  - This isn't strictly Windows related, and I'd be happy to be told it is a complete red herring: What are the export implications of this change?

With the right answers to these questions, I think I'll feel comfortable about merging your branch.

[Zooko Wilcox-O'Hearn (zooko)]

> "The slave which builds two thirds of the Windows pyOpenSSL packages is currently offline. I hope that it will return very soon, but I can't build all of the pyOpenSSL Windows packages until it comes back."

I am willing to offer a Windows buildslave for pyOpenSSL. It will be the same machine that currently runs Twisted and Nevow Windows buildslaves.

Also I'm willing to manually (ick) use that machine to build packages.

I will now try to answer your questions. Unfortunately I do not know much about the current Windows Installer way to do things, so I will indicate where my answers are guesses.

> "- How does this change the experience for a Windows user who was previously satisfied with the current style of pyOpenSSL Windows installers"

I don't think that this ticket is about removing pyOpenSSL Windows installers, I think it is about adding a bdist_egg. This will have no effect on users who download and run a pyOpenSSL Windows installer. Even if someone for example *both* installs a pyOpenSSL Windows installer and installs a binary egg, I don't think it there will be a problem. I'm not entirely sure about that. Chris Galvan, could you comment?

> "- What are the implications for the importance of the version of OpenSSL (not pyOpenSSL) installed on the host which builds the Windows installer for pyOpenSSL? I have very little control over these machines and I have the (unsubstantiated) feeling that this change makes the version of OpenSSL installed on them more important."

Hm. What will happen is that the "setup.py bdist_egg" process will copy the OpenSSL DLL into the resulting .egg file. Therefore, whatever version of OpenSSL is installed on the build machine is the version that will be included in the binary egg. How is this different from what the current Windows Installers do?

> "- This isn't strictly Windows related, and I'd be happy to be told it is a complete red herring: What are the export implications of this change?"

This change doesn't have any effect on export issues.

[Jean-Paul Calderone (exarkun)]

> Hm. What will happen is that the "setup.py bdist_egg" process will copy the OpenSSL DLL into the resulting .egg file. Therefore, whatever version of OpenSSL is installed on the build machine is the version that will be included in the binary egg. How is this different from what the current Windows Installers do?

The Windows installers don't include the OpenSSL DLL. They link against it, or whatever it is that you do to DLLs on Windows. The user is required to install OpenSSL separately from pyOpenSSL such that the DLLs which are linked against can be found.

[Zooko Wilcox-O'Hearn (zooko)]

I see. Well, as I understand it (Chris Galvan -- please help!), the bdist_egg approach in this ticket will include a DLL inside the pyOpenSSL directory, so it will not interfere with (nor benefit from) an OpenSSL DLL which is installed into a system directory.

[Chris Galvan (cgalvan)]

Sorry for the late reply, I was at school all day today :)

As Zooko already pointed out, the patch would include the OpenSSL DLLs inside the .egg file. On Windows, the Python extensions which are linked against these DLL's dynamically find the necessary DLLs by looking for them either:

1. In the same directory as the extension.
2. In one of the folders on the user's system PATH.
3. I believe that also Python extensions can find DLL's under <python root>/DLLs.

In any case, our proposed patch takes advantage of the (1) method of finding these DLLs. This would allow Windows users to not have to install OpenSSL on their system, since the DLL's would be dynamically found from inside the egg itself. Also, this would not interfere with any existing OpenSSL installation on the users' system because the directory inside the egg which these DLL's would be is not added to your system PATH, so the only thing that will know about these DLLs is the pyOpenSSL extensions.

Hopefully this cleared up some of your questions, is there anything else I can help to clear up?

[Zooko Wilcox-O'Hearn (zooko)]

Okay, what's the next step? I would be happy to set up a Windows buildslave.

[Zooko Wilcox-O'Hearn (zooko)]

Argh. I wasted my morning trying to build openssl and pyOpenSSL myself. It is apparent that both of their build scripts have many hardcoded assumptions which make it difficult for me to build native Windows binaries with cygwin gcc. I'm very frustrated. Feel free to enjoy my angry blog --

http://testgrid.allmydata.org:8123/uri/URI:DIR2-RO:hgvn7nhforxhfxbx3nbej53qoi:yhbnnuxl4o2hr4sxuocoi735t6lcosdin72axkrcboulfslwbfwq/wiki.html#2008-12-01

Anyway, it is increasingly urgent for me to be able to provide my users with a binary of pyOpenSSL. I would very much rather stop using pyOpenSSL and ~OpenSSL entirely in Tahoe, but unfortunately we still sort of rely on it for a couple of things. Please, please build binary eggs for Windows for me. Thank you.

[Jean-Paul Calderone (exarkun)]

Something which hasn't quite been implicit up to this point but which hasn't really been directly stated and which is really critical: there are two things actually being requested here. The first is that pyOpenSSL's distutils junk be changed so that it can build eggs. The second is that such an egg be uploaded to PyPI. One of these will be resolved by a commit to the repository and a subsequent release, the other will only be resolved when someone runs an `upload´ command of some sort.

I think that you (zooko) are more interested in the latter than in the former, except inasmuch as the former facilitates the latter. It occurs to me that your immediate difficulties can be resolved without committing anything to the repository though. If someone locally patches pyOpenSSL 0.8's setup.py so that it can build binary eggs and then runs the upload command, your issue will be resolved (one thing I'm not sure about is whether a subsequent 0.9 release without a binary egg being uploaded would ruin your day or not). Is this accurate? If so, I'm happy for that to happen ASAP so that your life gets easier before I have time to resolve this in a way that I'm confident in committing to trunk.

[Zooko Wilcox-O'Hearn (zooko)]

Dear JP:

Thank you for the prompt offer of help. You are right that all I currently need is for someone to build an egg and make it available to me.

(I remain interested in helping with other related things: making the pyOpenSSL setup.py script capable of building eggs, automating the building of such with buildbot, patching the OpenSSL config scripts so that you can build native OpenSSL libs with cygwin, etc.)

In fact, I don't even require that the egg is hosted on PyPI -- I could host that particular binary egg on allmydata.org. It would seem better for it to be hosted on PyPI, for various reasons.

Thanks!

Regards,

Zooko

[Jean-Paul Calderone (exarkun)]

Hi Zooko,

So can you generate the egg and host it yourself until I "officially" resolve this somehow? I can't generate the egg very easily, as I have no direct access to a Windows machine.

[Zooko Wilcox-O'Hearn (zooko)]

Well, that's what I tried to do this morning. But it appears I can't compile OpenSSL or pyOpenSSL without acquiring a Microsoft compiler.

[Zooko Wilcox-O'Hearn (zooko)]

Maybe Chris Galvan could build one for me.

[Zooko Wilcox-O'Hearn (zooko)]

Okay, I found these instructions:

http://bugpy.blogspot.com/2007/07/pyopenssl-installation.html

Which look like they might work, starting from the openssl official (???) binaries for Windows:

http://www.slproweb.com/products/Win32OpenSSL.html

I really hate redistributing binaries of unknown provenance, that I'm unable to build myself from source, and I really hate manually re-arranging the filesystem and/or patching setup scripts in order to build, but I'm willing to do it this time, in the hopes that it won't be necessary next time.

[Zooko Wilcox-O'Hearn (zooko)]

Okay, I'm sorry that I was so grumpy earlier. OpenSSL *does* come with instructions for how to build it in exactly the way that I want: using cygwin tools to produce native-win32 executables (which is a kind of cross-compiling).

I'm now trying that to generate OpenSSL binaries, then I'll try to patch the pyOpenSSL setup.py to use those binaries.

[Zooko Wilcox-O'Hearn (zooko)]

Ah, those instructions for building native libs using cygwin fail to produce shared libraries -- they can apparently only produce static libraries that way. Of course, static libraries are not sufficient to build pyOpenSSL. Having now spent more or less all day on this project, I give up for now. :-(

[Zooko Wilcox-O'Hearn (zooko)]

Ah, thanks to russians for posting howtos:

http://vitus.wagner.pp.ru/articles/openssl-mingw.html

Okay, I un-give-up. Maybe I should stop updating this ticket quite so often, but I didn't want Chris Galvan to start doing it and then find out that I was doing it.

[Zooko Wilcox-O'Hearn (zooko)]

Here is a letter that I wrote to openssl-dev:

Folks:

I'm trying to build a native win32 openssl-0.9.8i using cygwin gcc (this is a kind of cross-compilation). The goal is to build a binary distribution of the pyOpenSSL package for Python.

The INSTALL.W32 contains instructions, but they don't link shared libs correctly -- exhibit A "openssl-098i-pristine.make.log.txt.bz2". This problem is described on this web page -- http://vitus.wagner.pp.ru/articles/openssl-mingw.html .

I tried the patch to Configure from that page (attached as "Configure.patch.txt") and it did indeed cause the shared libs to be linked correctly and then "make test" passes. However, then when I try to build pyOpenSSL using the openssl-0.9.8i header files I get an error -- "ossl_typ.h:178: error: parse error before numeric constant". Please see attached "gcc.cmdline.txt" and "gcc.log.txt". The relevant part of the pyOpenSSL "connection.c" is visible here:

http://bazaar.launchpad.net/%7Eexarkun/pyopenssl/trunk/annotate/78?start_revid=78&file_id=connection.c-20080219014912-qyb7kjf196jhzlyv-149

It says "#include <openssl/err.h>".

I investigated a bit but I couldn't figure out what is wrong. OCSP_RESPONSE is being defined with a typedef in ossl_typ.h line 178, and that gives this parse error, which presumably has something to do with "DECLARE_ASN1_FUNCTIONS(OCSP_RESPONSE)" over in asn1.h.

Hopefully someone on this list can easily spot the problem.

Thanks!

Regards,

Zooko

[Zooko Wilcox-O'Hearn (zooko)]

Here's just the patch part from the "details.tar.bz2" tarball:

[Zooko Wilcox-O'Hearn (zooko)]

Applying this patch -- http://rt.openssl.org/Ticket/Display.html?id=1749&user=guest&pass=guest -- to openssl-0.9.8i makes it possible to build pyOpenSSL against openssl-0.9.8i with mingw. However, once I've done that, and I produce a binary egg with "python ./setup.py bdist_egg", I get the same failure as I already had with Chris's egg (described on http://allmydata.org/trac/tahoe/ticket/456#comment:21 ):

$ PYTHONPATH=./pyOpenSSL-0.8-py2.5-win32.egg python -c "import OpenSSL"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\Python25\lib\site-packages\pyopenssl-0.8-py2.5-win32.egg\OpenSSL\__init__.py", line 11, in <module>
  File "c:\Python25\lib\site-packages\pyopenssl-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 7, in <module>
  File "c:\Python25\lib\site-packages\pyopenssl-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 6, in __bootstrap__
ImportError: DLL load failed: The specified module could not be found.

[Zooko Wilcox-O'Hearn (zooko)]

A-ha! I cleverly notice that the stack trace that I pasted clearly shows that it is trying to import pyopenssl from C:\Python25, not the one from ./pyOpenSSL-0.8-py25-win32.egg

Ooh, I figured something out. If I add zip_safe=False to setup(), then I get an .egg which still doesn't work when I do "PYTHONPATH=that.egg python -c 'import OpenSSL'", but *does* work if I do "easy_install that.egg".
Something about the zipping of the egg is breaking the import of the rand.pyd and the other .pyd files.

Now proceeding to deploy my home-grown pyopenssl binary egg with zip_safe=False...

Further improvements to this process and a way for it to be zip_safe would be most welcome...

[Zooko Wilcox-O'Hearn (zooko)]

Note that I still need a python 2.4 egg. I can produce one myself, of course, now, but help would be appreciated.

[Zooko Wilcox-O'Hearn (zooko)]

http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:snrfwfxatrci35zdgjnzxxx2ke:unarxv347edtku3xzmefy4mcdmfngxzeb72iyqcadbjzjpczjx5a

^-- build dependencies for tahoe, stored on tahoe!

[Jean-Paul Calderone (exarkun)]

I got this far:

Traceback (most recent call last):
  File "setup.py", line 102, in ?
    data_files = [("OpenSSL", glob(os.path.join(ssl_lib, '*.dll')))]
  File "c:\python23\lib\ntpath.py", line 89, in join
    assert len(path) > 0
AssertionError

I don't have SSL_LIB or SSL_INC set, so that'd be why. I need some more help from Windows people to get any further. The data files we want seem to be cryptoeay32-0.9.8.dll and ssleay32-0.9.8.dll. Zooko got this by building OpenSSL himself following some procedure which I haven't read through in detail. I can't build OpenSSL for Windows without a lot of work (Zooko figured out most of the hard stuff, probably, but I don't even have direct access to a Windows machine) and I'd rather *not* build it myself, anyway. So I need to see if there are other possibilities for getting these dlls. I already have *some* OpenSSL libraries (I can use pyOpenSSL, after all). If there's some way to leverage those, that would be great.

There are a couple Windows people in particular I am going to track down to try to find the answers to these questions, but any input from anyone else is more than welcome.

[Zooko Wilcox-O'Hearn (zooko)]

allmydata.com is working on making a Windows machine accessible to JP over RDP.

[Zooko Wilcox-O'Hearn (zooko)]

Chris Galvan has been building bdist_egg's of pyOpenSSL for Windows for various Python's. See notes here:

http://allmydata.org/trac/tahoe/ticket/456

[Zooko Wilcox-O'Hearn (zooko)]

A new user of Tahoe tried to use it on Windows and got a failure to use the OpenSSL DLL from the binary egg:

http://allmydata.org/trac/tahoe/ticket/664 # Unable to start.

[Jean-Paul Calderone (exarkun)]

I've just released 0.9. I uploaded a Python 2.3 binary egg and a Python 2.5 binary egg to PyPI. It would be nice to hear whether these are working for people or not.

I'm currently unable to build Python 2.4 or Python 2.6 versions, as I lack properly configured Windows machines with those versions of Python installed (I have improperly configured Windows machines with those versions installed, though). If anyone can give me some assistance in getting these machines configured properly, then I should be able to build binary eggs for those versions of Python as well.

[Zooko Wilcox-O'Hearn (zooko)]

Hooray! Good news! I have a Windows machine here that has Python 2.6. I guess I have to take it away from my son who is playing Battle For Wesnoth and reboot it to Windows first, though. Shall I just run "python2.6 ./setup.py bdist_egg" and send you the resulting .egg file?

[Jean-Paul Calderone (exarkun)]

Sounds great

[Zooko Wilcox-O'Hearn (zooko)]

Sigh. No can do. This issue makes it impossible to build pyopenssl using only free software tools on Windows:

http://bugs.python.org/issue3308

Please find someone who is willing to use the Microsoft compiler for this purpose and ask them to build a py2.6 binary. :-(

[Zooko Wilcox-O'Hearn (zooko)]

Oh, there may be another option. http://bugs.python.org/issue3308 suggests that the problem manifests itself only when the extension module needs to link to localtime(). How much work would it be to change pyOpenSSL so it doesn't need that symbol? Ideally by removing code that uses localtime() -- I love removing code -- but possibly also by substituting some other function?

[Jean-Paul Calderone (exarkun)]

Hm. I'm not sure I fully understand, but:

exarkun@charm:~/Projects/pyOpenSSL/trunk$ grep localtime ./ -r --include '*.c'
exarkun@charm:~/Projects/pyOpenSSL/trunk$ grep localtime /home/exarkun/Scratch/Source/openssl-0.9.8g/ -r --include '*.c'
/home/exarkun/Scratch/Source/openssl-0.9.8g/crypto/mem_dbg.c: lcl = localtime(&m->time);
/home/exarkun/Scratch/Source/openssl-0.9.8g/ssl/kssl.c: tm_l = localtime(&now); tl = mktime(tm_l);

[Jean-Paul Calderone (exarkun)]

Well, I still don't really understand, but I built a Python 2.6 binary egg using VSE.

[Zooko Wilcox-O'Hearn (zooko)]

Alas, this doesn't work for me. The following starts with the cmd.exe command shell and uses the standard python 2.6.2 for Windows from python.org:

C:\Documents and Settings\Zooko Brillnonywonx\Desktop>python -i
Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("pyOpenSSL-0.9-py2.6-win32.egg")
>>> import OpenSSL
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyOpenSSL-0.9-py2.6-win32.egg\OpenSSL\__init__.py", line 11, in <module>
  File "pyOpenSSL-0.9-py2.6-win32.egg\OpenSSL\SSL.py", line 7, in <module>
    Object types:
  File "pyOpenSSL-0.9-py2.6-win32.egg\OpenSSL\SSL.py", line 6, in __bootstrap__

ImportError: DLL load failed: The specified module could not be found.
>>>

[Zooko Wilcox-O'Hearn (zooko)]

And here I first unzipped it using winrar:

C:\Documents and Settings\Zooko Brillnonywonx\Desktop\tmp\unzipped>dir
 Volume in drive C is S3A3305D001
 Volume Serial Number is 14EE-EB5A

 Directory of C:\Documents and Settings\Zooko Brillnonywonx\Desktop\tmp\unzipped

02/06/2009 02:42 PM <DIR> .
02/06/2009 02:42 PM <DIR> ..
02/06/2009 02:42 PM <DIR> EGG-INFO
02/06/2009 02:42 PM <DIR> OpenSSL
               0 File(s) 0 bytes
               4 Dir(s) 22,211,915,776 bytes free

C:\Documents and Settings\Zooko Brillnonywonx\Desktop\tmp\unzipped>python -i
Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import OpenSSL
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "OpenSSL\__init__.py", line 11, in <module>
    import rand, crypto, SSL, tsafe
ImportError: DLL load failed: The specified module could not be found.
>>>

[Zooko Wilcox-O'Hearn (zooko)]

Chris Galvan built a pyOpenSSL win32 py2.6 binary egg, attached. He wrote:

"""
I have attached the pyOpenSSL egg I built. The main differences I saw between mine and the one exarkun built is that his was zip-safe and was missing ssleay32.dll. Let me know if mine works for you!
"""

I haven't tried it yet.

[Jean-Paul Calderone (exarkun)]

I wonder how Chris built this. I'm very keen to have build automation which does this stuff right. I don't want to bother anyone when I release 0.10 to build another egg, I just want it to happen. :)

Chris, can you provide details on what you did?

By the way, before Zooko reminded me of the last comment on this ticket, I spent a long time reworking setup.py. I've gotten it to the point where it can build the other kinds of bdists for Windows, but it's presently placing the dlls in the wrong place when bdist_egg is used. I wonder if there is some trick to being able to put the dlls in the right place in all cases.

That work is in <lp:~exarkun/pyopenssl/working-windows-build>, for what it's worth.

[Jean-Paul Calderone (exarkun)]

I think working-windows-build is now in somewhat decent shape. The build it is capable of doing still has certain requirements on the environment in which it is run which I would prefer to avoid, but I think this is still a significant improvement from current trunk. And specifically, I think that at least *I* can use it to build working binary Windows distributions of pyOpenSSL, which I will undertake to do for each release.

I would like someone else to test the binaries created by this branch before I merge it, though. I'll attach several here shortly.

[Jean-Paul Calderone (exarkun)]

Alright, perhaps now this is resolved. The working-windows-build branch should make things better. When I release 0.10, there'll be binaries on PyPI.