libffi7 missing from Ubuntu (pip's python3-openssl appears to be built against the wrong version of libffi)

[Tom Cook]

Ubuntu groovy and up upgraded to libffi8ubuntu1, thus making Ubuntu incompatible with 3rd-party binaries that desire to use libffi7.

Let's backport and provide libffi7 runtime library only, for those.

Not sure how that would work with ctypes though.

---

I've just upgraded to Ubuntu 20.10 which comes with python3-openssl version 19.0.1-2. It breaks (at least some) Python applications that use the `requests` library to access HTTPS URLS. For instance, this stack trace (note that I have clipped the first few frames from the stack as they are proprietary):

  File "/home/tkcook/.local/lib/python3.8/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/tkcook/.local/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/tkcook/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/tkcook/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/tkcook/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 342, in connect
    self.ssl_context = create_urllib3_context(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 289, in create_urllib3_context
    context.verify_mode = cert_reqs
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 438, in verify_mode
    self._ctx.set_verify(_stdlib_to_openssl_verify[value], _verify_callback)
  File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1119, in set_verify
    self._verify_helper = _VerifyHelper(callback)
  File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 337, in __init__
    self.callback = _ffi.callback(
SystemError: ffi_prep_closure(): bad user_data (it seems that the version of the libffi library seen at runtime is different from the 'ffi.h' file seen at compile-time)

[Tom Cook (tom-k-cook)]

I've confirmed that setting up a virtualenv with the pyopenssl and requests packages from PyPI does not result in the same defect. If I get a chance, I'll try to put together a minimal test case to reproduce this.

[Tom Cook (tom-k-cook)]

A couple more notes:

* I've removed my per-user install of requests but this doesn't help.
* This doesn't kill all HTTPS requests; it seems to be related to the format of the CA certificate being used to verify the server.

[Ben Simpson (thehoagie)]

I am seeing this to while using Lutris v 0.5.8.1

> 2020-12-18 14:10:03,258: Error while completing task <bound method Downloader.async_download of <lutris.util.downloader.Downloader object at 0x7f397c3cb5b0>>: ffi_prep_closure(): bad user_data (it seems that the version of the libffi library seen at runtime is different from the 'ffi.h' file seen at compile-time)

Reinstalling the requirements via pip did not resolve this error.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Tycho Andersen (tycho-s)]

I see the same behavior when starting lutris.

[Nico R (u-nico-c)]

Can confirm with QGIS whenever the plugins "QuickMapServices" or "Google Earth Engine Data Catalog" try to download data from Google servers.

[Dimitri John Ledkov (xnox)]

libffi7 (3.3-5ubuntu1) hirsute; urgency=medium

  * Provide libffi7 runtime library for the 3rd party app
    compatiblity. LP: #1903890

 -- Dimitri John Ledkov <email address hidden> Fri, 05 Feb 2021 13:34:22 +0000

I have now shipped libffi7 in hirsute. Such that if one installs third-party binaries, on hirsute, one can install libffi7 from the archive, and things should "just work". It would be nice if somebody who is affected could verify that.

You can get hirsute trivially with lxd.

Or please provide concrete reproducer examples - i.e. specifically which applications, downloaded from where, and installed how would be useful. I see a few things mentioned in this bug report, but not detailed enough for me to experience the issue myself.

[Jan Kaupe (morodar)]

I just upgraded from Ubuntu 20.10 to Ubuntu 21.04.
Thanks to libffi7, Lutris is working again.

---

I downloaded Lutris using the instructions from https://lutris.net/downloads/

After Installation, I had issues with Lutris and received the error:

```
 Fie "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 337, in __init__
    self.callback = _ffi.callback(

Download failed: ffi_prep_closure(): bad user_data (it seems that the version of the libffi library seen at runtime is different from the 'ffi.h' file seen at compile-time)
```

libffi7 was not available for Ubuntu 20.10.
After the upgrade to 21.04, libffi7 has been installed.

[Tom Cook (tom-k-cook)]

I've confirmed that my use case (the original report of this ticket) is resolved in 21.04.

[Dimitri John Ledkov (xnox)]

I am glad that this worked out fine now.

I am not sure there is time to fix this in 20.10, as it has only a few months of support left. I hope that having libffi7 in 21.04 is enough.

[Brian Murray (brian-murray)]

The Groovy Gorilla has reached end of life, so this bug will not be fixed for that release

[Steve Langasek (vorlon)]

In doing some cleanup of the whitelist of Ubuntu packages for the i386 architecture, I ran across this issue because libffi7 is still in the archive as of kinetic. The backwards-compatibility package has been in the archive now for 3 release cycles, including an LTS; that seems like it should be enough time for third-parties to have updated to not depend on an old version of libffi.

I'm therefore going to go ahead and remove libffi7 from kinetic. If someone still needs libffi7, please provide a complete step-by-step reproducer for it because I'm not able to find any third-party binaries (pip or lutris) that still require libffi7.

Removing packages from kinetic:
 libffi7 3.3-5ubuntu1 in kinetic
  libffi7 3.3-5ubuntu1 in kinetic amd64
  libffi7 3.3-5ubuntu1 in kinetic arm64
  libffi7 3.3-5ubuntu1 in kinetic armhf
  libffi7 3.3-5ubuntu1 in kinetic i386
  libffi7 3.3-5ubuntu1 in kinetic ppc64el
  libffi7 3.3-5ubuntu1 in kinetic riscv64
  libffi7 3.3-5ubuntu1 in kinetic s390x
Comment: Compatibility package, unmaintained, no apparent remaining users; LP: #1903890
1 package successfully removed.