segfault when using lxml>=5 on at least python:3.9.12-alpine and almalinux:8 with python 3.8 3.9 and 3.11

[Leon van der Ree]

We are using OneLogin Saml2 in our project and this suddenly crashed in our pipeline.

After some debugging and checking our changes, I found out it wasn't a change at our codebase, but we did not fixate lxml in our requirements.

We are normally running python 3.9 via the official python:3.9.12-alpine image, but I've also tested our code base on almalinux:8 images on which I installed python 3.8, 3.9 and 3.11

```
RUN dnf install -y epel-release yum-utils \
    && dnf config-manager --enable epel \
    && dnf config-manager --set-enabled powertools \
    && dnf update -y \
    && dnf install -y \
    openssl-devel libffi-devel bzip2-devel \
    postgresql-libs postgresql-devel sqlite-devel \
    gcc make wget \
    httpd-devel \
    libxml2-devel xmlsec1-devel xmlsec1-openssl-devel libtool-ltdl-devel \
    git \
    redhat-rpm-config \
    python39-devel \
      python39 \
      postgresql-libs \
      postgresql \
      openssl \
      libxml2 \
      xmlsec1 \
      xmlsec1-openssl \
    && dnf clean all \
    && rm -rf /var/cache/dnf
```

In our project we are using OneLogin Saml2 and when we create our metadata, or got a login request (xml artifact) the entire Python process crashes with a coredump.

With debugging I found out it was for example this line, causing the issue:

https://github.com/SAML-Toolkits/python3-saml/blob/master/src/onelogin/saml2/utils.py#L738
```
        signature = xmlsec.template.create(elem, xmlsec.Transform.EXCL_C14N, sign_algorithm_transform, ns='ds')
```

And after running a `pip freeze` I found out that our lxml library had been updated from `lxml==4.9.3 => lxml==5.1.0`

4.9.3 was working OK, so is 4.9.4
But 5.0.0, 5.0.1 and 5.1.0 are broken; all resulting in a segfault

```
jan 10 15:50:34 localhost.localdomain systemd-coredump[231136]: [🡕] Process 230996 (python3) of user 1000 dumped core.

                                                                Stack trace of thread 48:
                                                                #0 0x00007f71bc5b42e0 n/a (/usr/lib64/libxml2.so.2.9.7 + 0xfd2e0)
                                                                ELF object binary architecture: AMD x86-64
```

I can provide more info if required, like a small test scenario, but I hope this already provides enough info.

[scoder (scoder)]

Without looking into the details, the most likely reason seems the use of xmlsec. If that's based on the system libraries, then the libxml2 version is probably different (and possibly incompatible) for xmlsec and lxml.

Could you try it with a clean source build of lxml against your system libraries? I.e. not using the binary wheels from PyPI?

[Leon van der Ree (lvanderree)]

You are Right!

I've created a minimal script to test with:

```
import sys
import xmlsec
from lxml import etree

print("%-20s: %s" % ('Python', sys.version_info))
print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))
print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))
print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))
print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))
print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))
print("%-20s: %s" % ('xmlsec used', xmlsec.__version__))

template = etree.fromstring('<xml></xml>')
print(xmlsec.template.create(template, xmlsec.Transform.EXCL_C14N, xmlsec.Transform.RSA_SHA256))
```

I don't even have to run it in a container to make it crash, it purely depdends on my VENV containing requirements:

```
xmlsec==1.3.13
lxml==5.0.0 # seg-fault
```

```
Python : sys.version_info(major=3, minor=9, micro=18, releaselevel='final', serial=0)
lxml.etree : (5, 1, 0, 0)
libxml used : (2, 12, 3)
libxml compiled : (2, 12, 3)
libxslt used : (1, 1, 39)
libxslt compiled : (1, 1, 39)
xmlsec used : 1.3.13
Segmentatiefout (geheugendump gemaakt)

```

but lxml==4.9.4 # working
```
Python : sys.version_info(major=3, minor=9, micro=18, releaselevel='final', serial=0)
lxml.etree : (4, 9, 4, 0)
libxml used : (2, 10, 3)
libxml compiled : (2, 10, 3)
libxslt used : (1, 1, 39)
libxslt compiled : (1, 1, 39)
xmlsec used : 1.3.13
<Element {http://www.w3.org/2000/09/xmldsig#}Signature at 0x7f5567f4d380>

```

running on python 3.9.18 (but also higher) on Fedora 39, on which
 - libxml2-devel-2.10.4-3.fc39.x86_64
 - libxslt-devel-1.1.39-1.fc39.x86_64
are installed

When I `pip uninstall lxml`

and do a git checkout of branch lxml-5.1.0

```
Building lxml version 5.1.0.
Building with Cython 3.0.8.
Building against libxml2 2.10.4 and libxslt 1.1.39
```

and after installing this version to my venv it is running:

```
Python : sys.version_info(major=3, minor=9, micro=18, releaselevel='final', serial=0)
lxml.etree : (5, 1, 0, 0)
libxml used : (2, 10, 4)
libxml compiled : (2, 10, 4)
libxslt used : (1, 1, 39)
libxslt compiled : (1, 1, 39)
xmlsec used : 1.3.13
<Element {http://www.w3.org/2000/09/xmldsig#}Signature at 0x7fc4ef70c780>
```

But what do you suggest how to fix this, or deal with this? Should everyone with this combination of xmlsec and livxml compile lxml themselves, of can the prebuild package be changed?

[scoder (scoder)]

> Should everyone with this combination of xmlsec and livxml compile lxml themselves

At least, you have to take care that both use the same version of libxml2, or a compatible one. Otherwise, xmlsec cannot process libxml2 trees created by lxml. I tend to list the library versions provided by the binary wheels in the changelog.

The safest way to make sure both work nicely together is probably to do a source build for both. Or to install both from the Linux distribution rather than PyPI. Or from anaconda/condaforge.

> of can the prebuild package be changed?

The binary wheels of lxml include statically linked libraries so that users don't have to install those themselves. That allows doing "pip install lxml" without further prerequisites. IIUC, xmlsec depends on external libraries being installed. That would be possible also for lxml, but would mean that installing lxml becomes harder and for some users really difficult. It's probably acceptable for most users who want to use lxml yourself, but for many users, it's just a transitive dependency of something else. Making that harder to install is annoying at best.

[scoder (scoder)]

Closing as duplicate of https://bugs.launchpad.net/bugs/1960668