Build instructions insufficient for building on Windows

[Jason R. Coombs]

In https://bugs.launchpad.net/lxml/+bug/1934800, I'm attempting to build lxml from source in order to enable testing of packages that depend on lxml.

The build from source instructions currently have this to say about building on Windows:

> Static linking on Windows
>
> Most operating systems have proper package management that makes installing current versions of libxml2 and libxslt easy. The most famous exception is Microsoft Windows, which entirely lacks these capabilities. To work around the limits of this platform, lxml's installation can download pre-built packages of the dependencies and build statically against them. Assuming you have a proper C compiler setup to build Python extensions, this should work:
>
> python setup.py bdist_wininst --static-deps
>
> It should create a windows installer in the pkg directory.

These instructions fail for me. When I attempt to run the documented command with Microsoft Visual Studio 2019 with C++ compiler installed, I get the following error:

```
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:libs\libxml2-2.9.5.win64\lib /LIBPATH:libs\libxslt-1.1.30.win64\lib /LIBPATH:libs\zlib-1.2.11.win64\lib /LIBPATH:libs\iconv-1.14.win64\lib /LIBPATH:C:\Program Files\Python39\libs /LIBPATH:C:\Program Files\Python39\PCbuild\amd64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\ATLMFC\lib\x64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64 libxslt_a.lib libexslt_a.lib libxml2_a.lib iconv_a.lib zlib.lib WS2_32.lib /EXPORT:PyInit_etree build\temp.win-amd64-3.9\Release\src\lxml\etree.obj /OUT:build\lib.win-amd64-3.9\lxml\etree.cp39-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.9\Release\src\lxml\etree.cp39-win_amd64.lib
fatal error C1047: The object or library file 'libs\iconv-1.14.win64\lib\iconv_a.lib' was created by a different version of the compiler than other objects like 'build\temp.win-amd64-3.9\Release\src\lxml\etree.obj'; rebuild all objects and libraries with the same compiler
LINK : fatal error LNK1257: code generation failed
```

[Jason R. Coombs (jaraco)]

The instructions don't give any context except to say "have a proper C compiler setup", but delving into the INSTALL docs, I see "lxml can use http://www.zlatkovic.com/libxml.en.html" and heading over to that site, it says "All binaries have been built using C/C++ compiler version 13" but it also makes mention of "If you're using Windows 95...", so I'm not super confident in the freshness of the guidance.

At one point in my life (~13 years ago), I wrote a Python script that would build lxml from sources reliably on Windows, but I'm not sure if I still have that code. I suspect I deleted it, presuming that this project had probably formalized a process for building on Windows.

[Jason R. Coombs (jaraco)]

I found the script (https://github.com/jaraco/jaraco.develop/blob/cf2cdcc166bc46bb90823abc8ebede624a463b86/scripts/lxml-build.py) I'd previous used. I see it pulls binaries from php :O

[Jason R. Coombs (jaraco)]

Is there any information on who makes the lxml binary builds for Windows and what setup they use and what commands they run, such that I could repeat the process that goes into building for Windows (in particular this build: https://files.pythonhosted.org/packages/e7/bc/9e7f38333767146191afbb176ffd2c1d553589a11b7e499f12aacc6cf5cd/lxml-4.6.3-cp38-cp38-win_amd64.whl)?

[scoder (scoder)]

The Zlatkovic binaries are hugely outdated.

Maximilian Hils created a project to build the libraries for lxml at some point:

https://github.com/mhils/libxml2-win-binaries

Looks like they haven't been updated for a while. I pinged the owner.

[Jason R. Coombs (jaraco)]

Sadly it looks like that project was abandoned not too long after it was created. :/

I'm pretty close to having a working Docker image with a Visual Studio build environment at https://github.com/jaraco/jaraco.windows/issues/21. If I can get that working, I'll look into updating the mhils technique for later versions, and then creating a continuous integration pipeline to build lxml.

But I can't help but wonder - how is it that this project releases lxml binaries for Windows?

[scoder (scoder)]

> how is it that this project releases lxml binaries for Windows?

Not sure what exactly you're asking, but the libxml2-win-binaries repo only builds the external libraries (or did so a couple of years ago) and lxml then pulls them from their release page and builds its own wheels that have the libraries statically linked.

[Jason R. Coombs (jaraco)]

> > how is it that this project releases lxml binaries for Windows?

> Not sure what exactly you're asking

I mean - how is it that you're able to create this file: https://files.pythonhosted.org/packages/72/d4/426ecb8849c47c3e370c87aa0ac05d85768df917ffea27fcd6686a5e6495/lxml-4.6.3-cp39-cp39-win_amd64.whl#sha256=542d454665a3e277f76954418124d67516c5f88e51a900365ed54a9806122b83 and upload it to PyPI? Does your build process rely on mhils builds? Or do you build those libraries too? If the latter, is there a reason not to publish the code or steps needed to make those builds work?

[scoder (scoder)]

We take the statically built libraries from the mhils repo. I agree that this is suboptimal.

Given that Github Actions supports Windows these days, it should be possible to automate the library build on their side and upload the libraries from a repo in the lxml GH organisation.

A PR for that would be very welcome.