PyPI is missing wheels for Mac

[Ned Batchelder]

An old bug (https://bugs.launchpad.net/lxml/+bug/1410270) asked for wheels for Linux and Mac. It was closed as Won't Fix because of the state of Linux wheels.

But the Mac issue was not addressed (and besides, now lxml does provide wheels for Linux).

lxml should provide wheels for Mac. I'm sure people will help if that will move this forward.

[Ned Batchelder (nedbatchelder)]

Any comment on this? Is there a reason for providing linux wheels but not Mac wheels? If someone volunteered to build the wheels, would the project be open to it?

[scoder (scoder)]

Maximilian Hills contributed a setup for building lxml on Windows using AppVeyor, including pre-built library versions. That's why the Windows wheels exist.

Joar Wandborg contributed a build setup for "manylinux" wheels, which I can use myself to build them. That's why the Linux wheels exist.

There were several others during the life of this project who have contributed time and resources for providing binary builds over a certain period, but for a long time now, no-one has contributed a build server that I could use to build Mac wheels. I'd be happy to upload them as long as it's painless, quick and easy for me. I'd also consider giving PyPI access to a trustful person who contributes them independently.

Beyond that, I do not feel in charge of investing much of my private time into this, considering that MacOS-X has a fairly large user base with enough developers and companies in it to "do something". As long as none of them feels like addressing this issue, I'm perfectly ok with the status quo.

[Ned Batchelder (nedbatchelder)]

Thanks for commenting. It's great to hear that you are open to someone else uploading the wheels.

[Matt Bachmann (bachmann.matt)]

I'll look into this and get back to you if I can help. Sometime this week

[Matt Bachmann (bachmann.matt)]

Ok, i've done some research into this.

1. I was able to build a wheel for the project (py2.7) (though --static-deps fails with "IOError: [Errno ftp error] 200 Switching to ASCII mode.")

 So I could put some work into making my laptop able build wheels for all the python versions. But I don't think that would be the most reliable system.

2. The pieces all seem to be there to do this in travis. I have forked your repo and can start experimenting on this with my fork (I think). This will likely take some time because I gotta find slivers of time to work on it ;-)

My basic thinking is we have travis build the linux and osx wheels and dump them into the github repo though the "releases" section. I have dumped the docs for this in here in case I become a ghost and someone takes the baton.

Building on multiple os's
https://docs.travis-ci.com/user/multi-os/

Uploading to github though releases
https://docs.travis-ci.com/user/deployment/releases/

Even deployment to pypi (if you wanted... not sure how this would work bringing the other wheels in line)
https://docs.travis-ci.com/user/deployment/pypi/

My fork
https://github.com/Bachmann1234/lxml

[Matt Bachmann (bachmann.matt)]

Current status (I don't know the Etiquette so please let me know if i'm too spammy)

On a branch I setup split OSX and Linux builds on travis. These builds can deploy wheels to github when successful.

Here is a sample build
https://travis-ci.org/Bachmann1234/lxml/builds/226373796

Since the linux build worked (and I made a tag) a wheel was deployed
https://github.com/Bachmann1234/lxml/releases/tag/wheel_test_2

Todo:

the OSX build is failing with "RuntimeError: ERROR: Trying to build without Cython, but pre-generated 'src/lxml/lxml.etree.c' is not available (pass --without-cython to ignore this error)." so I gotta figure that out. (advise appreciated if anyone is listening and has ideas)

OSX does not have a full travis python environment so right now things are only configured for python3.6. Once I get the builds working I will work backwards to add all supported version. I believe this is doable.

[Matt Bachmann (bachmann.matt)]

Work is being done on this branch btw https://github.com/Bachmann1234/lxml/tree/build_wheels

[scoder (scoder)]

Thanks for working on this! Just keep talking :)

Regarding the Cython error, it looks like you already installed Cython, which is needed to do a source build. I think it would be good to declare a sticky Cython version in requirements.txt, currently 0.25.2 for the 3.7.x release branch (and master). You should then pass "--with-cython" to "setup.py build_ext" to enable its usage.

[Matt Bachmann (bachmann.matt)]

Wooooooo

Good progress today. Here is what was happening. In the travis install script I create and activate a python virtual environment. But when travis starts the actual "run" the context is reset and that virtual environment is no longer active so it was like cython was not installed at all.

But now we have a build that travis says passes. This release has a python3 linux and mac wheel

https://github.com/Bachmann1234/lxml/releases/tag/wheel_test_3

I downloaded the wheel and it seems to work for some basic toying around. So my next step is to get all the python versions working. Travis not having their OSX environment to par with their linux environment will make this annoying but not impossible.

One thing that concerned me was the mac build failed some tests and had several errors, but travis was happy.
```
FAILED (failures=2, errors=39)
make: *** [test_inplace] Error 1

```

Lot of cases of " lxml.etree.ParserError: Unicode parsing is not supported on this platform".

However, I also get this when I run in master on my local machine so im considering it out of scope for this work.

[Matt Bachmann (bachmann.matt)]

Not too bad!

Current status: Multi OS, Multi python build. Have not deployed all the wheels but ill wait till I do the next step.
https://travis-ci.org/Bachmann1234/lxml/builds/227128047

While im playing with this I am going to go ahead and have the linux builds build manylinux wheels cuse... why not?

Only downside is the build is slow. Travis (totally reasonably) limits the amount of parallelization for a build so even though the longest mac builds take ~6-7 minutes the whole build takes closer to 20.

[Matt Bachmann (bachmann.matt)]

Think this work is at a point where I am ready for feedback so I opened https://github.com/lxml/lxml/pull/247

[Matt Bachmann (bachmann.matt)]

Made notes in the PR but figured id keep the bug log up to date. Based on PR feedback I took a different approach.

Repo here
https://github.com/Bachmann1234/lxml-wheels

Successful wheel build here
https://travis-ci.org/Bachmann1234/lxml-wheels/builds/234419129

Artifacts here
https://github.com/Bachmann1234/lxml-wheels/releases/tag/wheel-test-4

[Cosimo Lupo (lupocos)]

It looks like the wheel for python3.7 for Mac is missing from the latest lxml release on PyPI (4.2.3):

https://pypi.org/project/lxml/4.2.3/#files

Only the manylinux1 py37 wheels are available.

However, in the last commit on the lxml/lxml-wheels repository, I see that @scoder updated multibuild setup to also compile the mac 3.7 wheels, not only the linux ones:

https://github.com/lxml/lxml-wheels/commit/dc18b4fc040ba55fb107baecb022c910a6be6819

Maybe they failed to be built or uploaded?
I guess it may be worth retrying now, by updating the multibuild git submodule to the latest master (or even "devel") branch. It should work now.

I'd love to see wheels for Mac (and Windows!) for python3.7 on PyPI.

Thank you!