Add support for Python 3.11

[Chillar Anand]

On Python 3.11, installation fails with the following error.

$ p311 -m pip install lxml
Collecting lxml
  Using cached lxml-4.8.0.tar.gz (3.2 MB)
  Preparing metadata (setup.py) ... done
Using legacy 'setup.py install' for lxml, since package 'wheel' is not installed.
Installing collected packages: lxml
  Running setup.py install for lxml ... error
  error: subprocess-exited-with-error

  × Running setup.py install for lxml did not run successfully.
  │ exit code: 1
  ╰─> [125 lines of output]
      Building lxml version 4.8.0.
      Building without Cython.
      Building against libxml2 2.9.4 and libxslt 1.1.29
      running install
      running build
      running build_py
      creating build
      creating build/lib.macosx-12.3-arm64-3.11
      creating build/lib.macosx-12.3-arm64-3.11/lxml
      copying src/lxml/_elementpath.py -> build/lib.macosx-12.3-arm64-3.11/lxml
      copying src/lxml/sax.py -> build/lib.macosx-12.3-arm64-3.11/lxml
      copying src/lxml/pyclasslookup.py -> build/lib.macosx-12.3-arm64-3.11/lxml
      copying src/lxml/__init__.py -> build/lib.macosx-12.3-arm64-3.11/lxml
      copying src/lxml/builder.py -> build/lib.macosx-12.3-arm64-3.11/lxml
      copying src/lxml/doctestcompare.py -> build/lib.macosx-12.3-arm64-3.11/lxml
      copying src/lxml/usedoctest.py -> build/lib.macosx-12.3-arm64-3.11/lxml
      copying src/lxml/cssselect.py -> build/lib.macosx-12.3-arm64-3.11/lxml
      copying src/lxml/ElementInclude.py -> build/lib.macosx-12.3-arm64-3.11/lxml
      creating build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/__init__.py -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      creating build/lib.macosx-12.3-arm64-3.11/lxml/html
      copying src/lxml/html/soupparser.py -> build/lib.macosx-12.3-arm64-3.11/lxml/html
      copying src/lxml/html/defs.py -> build/lib.macosx-12.3-arm64-3.11/lxml/html
      copying src/lxml/html/_setmixin.py -> build/lib.macosx-12.3-arm64-3.11/lxml/html
      copying src/lxml/html/clean.py -> build/lib.macosx-12.3-arm64-3.11/lxml/html
      copying src/lxml/html/_diffcommand.py -> build/lib.macosx-12.3-arm64-3.11/lxml/html
      copying src/lxml/html/html5parser.py -> build/lib.macosx-12.3-arm64-3.11/lxml/html
      copying src/lxml/html/__init__.py -> build/lib.macosx-12.3-arm64-3.11/lxml/html
      copying src/lxml/html/formfill.py -> build/lib.macosx-12.3-arm64-3.11/lxml/html
      copying src/lxml/html/builder.py -> build/lib.macosx-12.3-arm64-3.11/lxml/html
      copying src/lxml/html/ElementSoup.py -> build/lib.macosx-12.3-arm64-3.11/lxml/html
      copying src/lxml/html/_html5builder.py -> build/lib.macosx-12.3-arm64-3.11/lxml/html
      copying src/lxml/html/usedoctest.py -> build/lib.macosx-12.3-arm64-3.11/lxml/html
      copying src/lxml/html/diff.py -> build/lib.macosx-12.3-arm64-3.11/lxml/html
      creating build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron
      copying src/lxml/isoschematron/__init__.py -> build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron
      copying src/lxml/etree.h -> build/lib.macosx-12.3-arm64-3.11/lxml
      copying src/lxml/etree_api.h -> build/lib.macosx-12.3-arm64-3.11/lxml
      copying src/lxml/lxml.etree.h -> build/lib.macosx-12.3-arm64-3.11/lxml
      copying src/lxml/lxml.etree_api.h -> build/lib.macosx-12.3-arm64-3.11/lxml
      copying src/lxml/includes/xmlerror.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/c14n.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/xmlschema.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/__init__.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/schematron.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/tree.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/uri.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/etreepublic.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/xpath.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/htmlparser.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/xslt.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/config.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/xmlparser.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/xinclude.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/dtdvalid.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/relaxng.pxd -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/lxml-version.h -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      copying src/lxml/includes/etree_defs.h -> build/lib.macosx-12.3-arm64-3.11/lxml/includes
      creating build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron/resources
      creating build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron/resources/rng
      copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron/resources/rng
      creating build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron/resources/xsl
      copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron/resources/xsl
      copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron/resources/xsl
      creating build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.macosx-12.3-arm64-3.11/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      warning: build_py: byte-compiling is disabled, skipping.

      running build_ext
      building 'lxml.etree' extension
      creating build/temp.macosx-12.3-arm64-3.11
      creating build/temp.macosx-12.3-arm64-3.11/src
      creating build/temp.macosx-12.3-arm64-3.11/src/lxml
      clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/usr/local/opt/llvm/include -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/Users/chillaranand/.pyenv/versions/3.11-dev/include/python3.11 -c src/lxml/etree.c -o build/temp.macosx-12.3-arm64-3.11/src/lxml/etree.o -w -flat_namespace
      src/lxml/etree.c:265439:14: error: incomplete definition of type 'struct _frame'
                  f->f_back = PyThreadState_GetFrame(tstate);
                  ~^
      /Users/chillaranand/.pyenv/versions/3.11-dev/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'struct _frame'
      typedef struct _frame PyFrameObject;
                     ^
      src/lxml/etree.c:265476:19: error: incomplete definition of type 'struct _frame'
              Py_CLEAR(f->f_back);
                       ~^
      /Users/chillaranand/.pyenv/versions/3.11-dev/include/python3.11/object.h:564:44: note: expanded from macro 'Py_CLEAR'
              PyObject *_py_tmp = _PyObject_CAST(op); \
                                                 ^~
      /Users/chillaranand/.pyenv/versions/3.11-dev/include/python3.11/object.h:107:41: note: expanded from macro '_PyObject_CAST'
      #define _PyObject_CAST(op) ((PyObject*)(op))
                                              ^~
      /Users/chillaranand/.pyenv/versions/3.11-dev/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'struct _frame'
      typedef struct _frame PyFrameObject;
                     ^
      src/lxml/etree.c:265476:19: error: incomplete definition of type 'struct _frame'
              Py_CLEAR(f->f_back);
                       ~^
      /Users/chillaranand/.pyenv/versions/3.11-dev/include/python3.11/object.h:566:14: note: expanded from macro 'Py_CLEAR'
                  (op) = NULL; \
                   ^~
      /Users/chillaranand/.pyenv/versions/3.11-dev/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'struct _frame'
      typedef struct _frame PyFrameObject;
                     ^
      src/lxml/etree.c:268244:5: error: incomplete definition of type 'struct _frame'
          __Pyx_PyFrame_SetLineNumber(py_frame, py_line);
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      src/lxml/etree.c:528:62: note: expanded from macro '__Pyx_PyFrame_SetLineNumber'
        #define __Pyx_PyFrame_SetLineNumber(frame, lineno) (frame)->f_lineno = (lineno)
                                                            ~~~~~~~^
      /Users/chillaranand/.pyenv/versions/3.11-dev/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'struct _frame'
      typedef struct _frame PyFrameObject;
                     ^
      4 errors generated.
      Compile failed: command '/usr/bin/clang' failed with exit code 1
      creating var
      creating var/folders
      creating var/folders/vb
      creating var/folders/vb/3wyl_42s4x1_z_7swgpq9xr40000gq
      creating var/folders/vb/3wyl_42s4x1_z_7swgpq9xr40000gq/T
      cc -I/usr/include/libxml2 -c /var/folders/vb/3wyl_42s4x1_z_7swgpq9xr40000gq/T/xmlXPathInith1ed9w04.c -o var/folders/vb/3wyl_42s4x1_z_7swgpq9xr40000gq/T/xmlXPathInith1ed9w04.o
      cc var/folders/vb/3wyl_42s4x1_z_7swgpq9xr40000gq/T/xmlXPathInith1ed9w04.o -lxml2 -o a.out
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> lxml

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

[scoder (scoder)]

lxml 4.9.0 will use a newer Cython version that supports Py3.11.

[san_ubuntu (gupta-sandeep)]

Does the fix work? I downloaded and compiled the latest python 3.11 (which should have the correct 3.11 compatible cython) and then tried pip install lxml. However I get the same error. Could you tell which cython is compatible with py3.11 and lxml.

[Ewout ter Hoeven (ewouth)]

Over at statsmodels we're working on Python 3.11 support, and noticed that libxml failed compiling. It looked a lot like this error. Could someone validate it's fixed for the versions we're using (they should be the latest)?

  Building lxml version 4.9.1.
  Building without Cython.
  Building against libxml2 2.9.9 and libxslt 1.1.33

The full CI log: https://dev.azure.com/statsmodels/statsmodels-testing/_build/results?buildId=4665&view=logs&j=c203999b-4fb3-5854-a733-83d14a010489&t=84c4483e-70f5-5851-7133-396b4a3130f5&l=596

  building 'lxml.etree' extension
  creating build\temp.win-amd64-3.11
  creating build\temp.win-amd64-3.11\Release
  creating build\temp.win-amd64-3.11\Release\src
  creating build\temp.win-amd64-3.11\Release\src\lxml
  C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.33.31629\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc\lxml\includes -IC:\hostedtoolcache\windows\Python\3.11.0-rc.2\x64\include -IC:\hostedtoolcache\windows\Python\3.11.0-rc.2\x64\Include -IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.33.31629\include -IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.33.31629\ATLMFC\include -IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\VS\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um /Tcsrc\lxml\etree.c /Fobuild\temp.win-amd64-3.11\Release\src\lxml\etree.obj -w
  cl : Command line warning D9025 : overriding '/W3' with '/w'
  etree.c
  C:\Users\VssAdministrator\AppData\Local\Temp\pip-install-9uueqnlg\lxml_51312a02e7e44c24b33029aafed4722a\src\lxml\includes/etree_defs.h(14): fatal error C1083: Cannot open include file: 'libxml/xmlversion.h': No such file or directory
  Compile failed: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Tools\\MSVC\\14.33.31629\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
  C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.33.31629\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I/usr/include/libxml2 -IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.33.31629\include -IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.33.31629\ATLMFC\include -IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\VS\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um /TcC:\Users\VSSADM~1\AppData\Loca...

[scoder (scoder)]

> working on Python 3.11 support, and noticed that libxml failed compiling

I don't think this has anything to do specifically with Py3.11.

BTW, there are newer libxml2/libxslt releases.

[Ewout ter Hoeven (ewouth)]

Thanks for getting back!

I don't think it's Python 3.11 itself, but the fact that there are no Python 3.11 wheels available on PyPI for Windows and macOS: https://pypi.org/project/lxml/4.9.1/#files

So this causes the wheel building process to fail. The easiest solution is to make Python 3.11 wheels available for those OSes, see bug 1977998.

I can't figure out why the wheel building is failing in my environment in the first place. If someone wants to help me figure out, please see this log: https://dev.azure.com/statsmodels/statsmodels-testing/_build/results?buildId=4696&view=logs&j=c203999b-4fb3-5854-a733-83d14a010489&t=84c4483e-70f5-5851-7133-396b4a3130f5&l=416

[Radek Bartoň (blackhexcz)]

I am experiencing the same issue on Arm64 device with Python 3.9.10. Please, let me know whether there is some workaround or if you need some more information to investigate the isssue.

[scoder (scoder)]

> I am experiencing the same issue on Arm64 device with Python 3.9.10.

You are most certainly not. This is a Py3.11 issue, not a Py3.9 issue. You might be looking for a different ticket instead, but didn't give enough information to know what your issue is.

Note that 4.9.x works fine with Py3.11, but lacks macOS binary wheels, see https://bugs.launchpad.net/lxml/+bug/1913032 for that.