Provide wheels for Windows

[Paul Moore]

It would be useful for people using virtualenv if lxml was available as a wheel for Windows (the wininst executables cannot be installed into a virtualenv using pip or by running them directly).

Could wheels be provided on PyPI for Windows users? The extra step is simply a matter of running setup.py bdist_wheel (assuming you have setuptools and wheel installed) as well as setup.py bdist_wininst.

[scoder (scoder)]

You should ask on the mailing list if someone wants to do that.

[Kay Hayen (kayhayen)]

There is a wheel for Python3.2, but ones for Python3.3 are missing. Same for MSI installers BTW.

[nonZero (udioron)]

Some more info:

The compiled wheels can be download from here:
http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml

So it is just a matter of downloading from there and uploading to pypi.

Udi.

[Paul Moore (pmoore)]

Only the project owners can upload wheels to PyPI. The project owners could reasonably ask Christoph Gohlke to allow them to redistribute his builds, but there's no point in 3rd parties doing so.

[John Hagen (johnthagen)]

This is critical for Windows support IMO, and also much more convenient for users. If lxml is a dependency of another package, you are greeted with a cryptic error on Windows.

A great example of a package that does wheels correctly is cryptography: https://pypi.python.org/pypi/cryptography

[Ned Batchelder (nedbatchelder)]

Is there something we can do to help with getting wheels available on PyPI? This is not a difficult thing to do, and would really help installers, on Windows and Mac.

[Paul Moore (pmoore)]

It's frustrating that this has been open for basically 2 years now, and the only response from the project author (scoder, as far as I can tell) is to suggest that I "ask on the mailing list". Which isn't very helpful as I don't know which mailing list, and (as has been pointed out) only the project author can upload wheels anyway!

It would be useful to know whether, if a user stepped up and offered to create wheels, the project would upload them to PyPI. If they will, maybe someone can do the builds. But if not, then there seems little point in anyone wasting any further effort trying to get support from the project itself, and the only option for users will remain that of using Christoph Gohlke's unofficial builds.

[Maximilian Hils (mhils)]

I'd like to second this request. Any package that depends on lxml currently needs to special-case lxml on Windows. (e.g., https://github.com/mitmproxy/mitmproxy/blob/b90579fe4513c1794bb8159880479a05da02cbae/setup.py#L87)

For Python 3.5, the only workaround is to download binaries from http://www.lfd.uci.edu/~gohlke/pythonlibs/ over plain HTTP. :-(

[scoder (scoder)]

Current Windows packages are provided by

https://github.com/zopefoundation/zope.wineggbuilder

If you want to help out, please discuss in this ticket what needs to be done:

https://github.com/zopefoundation/zope.wineggbuilder/issues/3

[Paul Moore (pmoore)]

I can't comment on wineggbuilder directly, but if instructions are available for building for Windows then I can show you how to set up automated builds using Appveyor (which has all the needed compilers available).

The only question really is building the dependencies (libxml) in a form compatible with MSVC (the various versions needed). That's the blocker here. If the project doesn't know how to do that, maybe reach out to Christoph Gohlke (referred to previously in this thread) for help?

[scoder (scoder)]

Paul, I'm really not opposed to providing Windows wheels on PyPI, as long as I don't have to invest any time into it. That's because I have other things to do that have an actual importance for me, which this topic doesn't.

If this is important to you, please go ahead and make it work. If you (or someone else) can come up with a 10-minutes setup guide and/or a pull request with working build scripts that I can trigger from github or so, I'm happy to press that green "bless me" button that you give me. So far, no-one has done that for anything newer than Python 3.2, which is the only reason why this ticket is still not resolved.

[anatoly techtonik (techtonik)]

Just for the reference, repository for GitHub is here https://github.com/lxml/lxml

[Maximilian Hils (mhils)]

Good news: I believe I successfully managed to compile libiconv, libxml2 and libxslt with Visual Studio 2015/ucrt. I'm admittedly all but an expert on this, but at the very least the resulting x86 wheel works fine on a clean Windows 7 machine. \o/

I'll try to document everything and will follow up here.

[Paul Moore (pmoore)]

Nice. Once you have a write-up, I'm happy to see if I can make that into a build process that will generate wheels on Appveyor. If that can be made to work, the result should be automatically generated wheels for Windows. Hopefully that would meet scoder's preference for a zero-effort solution :-)

[Maximilian Hils (mhils)]

Paul: awesome!

Writeup and initial PR are here:
https://github.com/lxml/lxml/pull/196
https://github.com/mhils/libxml2-win-binaries

[Tal Einat (taleinat)]

+1

This is making distributing internal Python tools which use lxml much more difficult than necessary for Windows users.

[Philippe Ombredanne (pombredanne)]

Build a wheel thyself in Windows works fine. I am using a minimal appevyor loop there https://github.com/pombreda/thirdparty (and this is for use to vendor in https://github.com/nexB/scancode-toolkit/) (as an aside, this also builds Linux (and many linux there: https://github.com/pombreda/thirdparty-manylinux/) and Mac using travis.

[Philippe Ombredanne (pombredanne)]

Do you need hand to automate that build + Pypi twine upload process?

[Philippe Ombredanne (pombredanne)]

As a side note: 3.6.4 fails to build wheels on Windows at all: See https://ci.appveyor.com/project/pombreda/thirdparty/build/job/822myjbhkemek4dr

[Maximilian Hils (mhils)]

To keep this thread updated: I also wrote a build script for AppVeyor (https://github.com/lxml/lxml/pull/200). This has been merged, but no wheels have been uploaded to pypi yet.

[scoder (scoder)]

@pombredanne: Note that you have to pass either the env variable "STATIC_DEPS=true" or the option "--static-deps" to setup.py in order to build complete wheels under Windows. ISTM that your failing appveyor build didn't do that.

[Maximilian Hils (mhils)]

I managed to compile x64 binaries for libiconv, libxml2, libxslt and zlib with VS2015, so it is now possible to build both 32 and 64 bit wheels for Python 3.5 on Windows: https://github.com/lxml/lxml/pull/207

[Emil Styrke (emil-styrke)]

Just stumbled on this. Is there anything I can do to make this happen? Or are we just waiting for a new release + pypi upload?

[scoder (scoder)]

I managed to build Python 3.5 wheels for 3.7.1 by manually configuring the build through their web interface. Just providing the .appveyor.yml file (and explicitly referencing it in their web setup) doesn't seem to bother them.

Could someone please test these packages?