Missing wheel for macos with M1

[Rehan Khwaja]

There's no wheel on pypi for the M1 chip

[scoder (scoder)]

Could you find out what a good way to build such a wheel is currently?
If it's easy, I'll consider it.

[J Brown (jacksonmaxfield)]

Build infrastructure always scares me and since I know very little about how this project builds itself, I won't try to touch it but I will say, cibuildwheel has been great: https://github.com/pypa/cibuildwheel

To add M1 / arm64 mac building support it is "as simple as adding 'arm64'" to the Mac build environment variables.

Worth a look in my opinion.

[[e-mail] (larsoner)]

Cirrus CI has M1 images you can use nowadays (free for OSS):

https://cirrus-ci.org/examples/#building-pypi-packages

And cibuildwheel has added to their FAQ how to use it (search "cirrus"):

https://cibuildwheel.readthedocs.io/en/stable/faq/#apple-silicon

I have also used cross-compilation via `cibuildwheel` on GitHub actions. So there are at least two good options nowadays I think! But given that you have this:

https://github.com/lxml/lxml/blob/master/.github/workflows/wheels.yml

I would probably transition to `cibuildwheel` for everything.

[scoder (scoder)]

I'd be happy to see a PR that migrates our current wheels.yml to cibuildwheel.

The macOS variant of the build really just calls "setup.py bdist_wheel" (see Makefile). That should be easy to adapt.

[Kapil Thangavelu (hazmat)]

its relatively easy to cross build arch with cibuildwheel in GitHub actions via qemu emulation, here's an example if it helps https://github.com/cloud-custodian/tfparse/blob/main/.github/workflows/release.yml#L31

[scoder (scoder)]

There is a universal2 wheel for Py3.11 now. Could someone please test it and report back if it works?

[Cosimo Lupo (lupocos)]

I just tried to pip install lxml in a fresh virtual environment on my MacBook Pro 16" with M1 Pro chip using CPython 3.11.1. Installing works, pip downloads the universal2 wheel as expected. However attempting to import `from lxml import etree` fails with ImportError: "symbol not found in flat namespace (_xmlFree)"

```
$ python
Python 3.11.1 (main, Dec 14 2022, 11:26:39) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import lxml
>>> from lxml import etree
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/clupo/.pyenv/versions/fonttools-py311/lib/python3.11/site-packages/lxml/etree.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace (_xmlFree)
```

So ... it still doesn't seem to work. :/

[Cosimo Lupo (lupocos)]

xmlFree probably comes from libxml2, but that's strange because according to otool the lxml etree.so only links with libSystem.B.dylib so depedencies (including libxml2) appear to be fully statically linked:

```
$ otool -L /Users/clupo/.pyenv/versions/fonttools-py311/lib/python3.11/site-packages/lxml/etree.cpython-311-darwin.so
/Users/clupo/.pyenv/versions/fonttools-py311/lib/python3.11/site-packages/lxml/etree.cpython-311-darwin.so (architecture x86_64):
 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
/Users/clupo/.pyenv/versions/fonttools-py311/lib/python3.11/site-packages/lxml/etree.cpython-311-darwin.so (architecture arm64):
 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
```

How was the wheel generated?

[Cosimo Lupo (lupocos)]

The error above was with python 3.11.1 compiled from source with pyenv.
So I also tried to download and install the official Python 3.11.1 from python.org (universal2 installer) available at https://www.python.org/downloads/release/python-3111/

I get a similar but slightly different ImportError when I try to import lxml.etree:

```
ImportError: dlopen(/Users/clupo/oss/fontmake/.venv/lib/python3.11/site-packages/lxml/etree.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace (_exsltDateXpathCtxtRegister)
```

[Cosimo Lupo (lupocos)]

If I force this "fat" python3.11 installation to run in emulation mode by prefixing it with `arch -x86_64 /path/to/python3.11` then importing lxml.etree works!

Which leads me to think that the libxml2 library that was statically linked maybe is not "fat" but only contains x86_64 architecture, and no arm64 one?

```
$ arch -x86_64 /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
Python 3.11.1 (v3.11.1:a7a450f84a, Dec 6 2022, 15:24:06) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from lxml import etree
>>> etree
<module 'lxml.etree' from '/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/lxml/etree.cpython-311-darwin.so'>
```

[scoder (scoder)]

> Which leads me to think that the libxml2 library that was statically linked maybe is not "fat" but only contains x86_64 architecture, and no arm64 one?

That's probably the case. Let's see if we can make the libraries fat as well.

https://github.com/scoder/lxml/commit/0766a4a8b23c0b474d235de97fcd2dbd5a874edf

[scoder (scoder)]

Doesn't look good yet. The builds don't pick up the CFLAGS and fail. Could someone try to get this to work?

https://github.com/scoder/lxml/actions/runs/3702410919

[Cosimo Lupo (lupocos)]

it doesn't pick up the CFLAGS because those are already set in the environment before the python setup.py build_ext command is run, see:

https://github.com/scoder/lxml/blob/0766a4a8b23c0b474d235de97fcd2dbd5a874edf/tools/ci-run.sh#L65

You can see how os.environ overwrite the CFLAGS you set in env_default when you call env_default.update(os.environ):

    env_default = {
        'CFLAGS': "-arch arm64 -arch x86_64 -O2",
        'LDFLAGS': "-arch x86_64 -arch arm64",
        'MACOSX_DEPLOYMENT_TARGET': "10.6"
    }
    env_default.update(os.environ)

Also I believe the MACOSX_DEPLOYMENT_TARGET for "fat" x86_64+arm64 wheels should be 10.9, not 10.6. I think that's what cibuildwheel defaults to.

[scoder (scoder)]

This is now the ticket with the highest duplication rate ever. It would be great if a macOS user could step up to solve it.

[Enrico M. (enrico-minack)]

@scoder, you mentioned "There is a universal2 wheel for Py3.11 now.".

With "there", you mean PyPi? There is none. If there is one on PyPi, could you point me to it please?

If you mean GitHub with "there", I have tested that on macOS 12.6.3 (Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz) and it works for me:

https://github.com/EnricoMi/publish-unit-test-result-action/actions/runs/4161663226/jobs/7199902634#step:8:28

This does not work:

    pip install lxml==4.9.2

But this does work:

    pip install https://github.com/lxml/lxml/releases/download/lxml-4.9.2/lxml-4.9.2-cp311-cp311-macosx_10_15_universal2.whl

[scoder (scoder)]

I meant on PyPI, where I had initially uploaded that wheel. I immediately got reports from macOS (not sure which CPU) users that it does not work for them, so I removed it from PyPI again. It seems to work on x86_64, but not on M1, if I understand correctly.

[scoder (scoder)]

> it doesn't pick up the CFLAGS because those are already set in the environment before the python setup.py build_ext command is run

You're looking in the wrong place. The wheel build is set up in .github/workflows/wheels.yml and done by the Makefile. It does not touch the CFLAGS (for macOS – it sets them for Linux, but that's ok).

[scoder (scoder)]

I built a universal wheel for a 5.0 pre-release version:

https://github.com/scoder/lxml/releases/download/lxml-5.0a1/lxml-4.9.2-cp311-cp311-macosx_11_0_universal2.whl

Could macOS users with an Arm64 CPU please give it a try? And could macOS users with an x86_64 CPU please confirm that it also works for them?

[scoder (scoder)]

Here's the change I made:
https://github.com/lxml/lxml/commit/59c328ff388ef04643a9a3949923ce6f35d580b2

[Peter Sobolewski (psobolewskiphd)]

Hi, I tested the cp311 universal2 wheel on my M1. It imports fine and I tried some simple etree stuff. Looks like all the compiled parts are universal:
```
Architectures in the fat file: _elementpath.cpython-311-darwin.so are: x86_64 arm64
Architectures in the fat file: builder.cpython-311-darwin.so are: x86_64 arm64
Architectures in the fat file: etree.cpython-311-darwin.so are: x86_64 arm64
Architectures in the fat file: objectify.cpython-311-darwin.so are: x86_64 arm64
Architectures in the fat file: sax.cpython-311-darwin.so are: x86_64 arm64
```
🎉

[Enrico M. (enrico-minack)]

I have tested this (cp311 universal2) on macOS 12.6.3 (Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz) and it works for me:

https://github.com/EnricoMi/publish-unit-test-result-action/actions/runs/4525139339/jobs/7969433673#step:8:184

[Enrico Minack (enricomi)]

Can we get the universal package added to the release?

[scoder (scoder)]

Probably resolved for Python 3.11 and 3.12.

[Enrico M. (enrico-minack)]

Thanks!