Python docker package does not import, crashes due to urllib3 upgrade

[A. Karl Kornel]

This is on a fresh install of the 24.04 beta that was released on April 12. I used the Desktop installer, into a VMware Fusion VM, on amd64.

Other than the defaults for a fresh Ubuntu Desktop install, all I ran was `apt install -y docker.io python3-docker`.

I ran `python3`, tried to execute `import docker`, and got this:

Python 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/docker/__init__.py", line 2, in <module>
    from .api import APIClient
  File "/usr/lib/python3/dist-packages/docker/api/__init__.py", line 2, in <module>
    from .client import APIClient
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 10, in <module>
    from .. import auth
  File "/usr/lib/python3/dist-packages/docker/auth.py", line 5, in <module>
    from . import credentials
  File "/usr/lib/python3/dist-packages/docker/credentials/__init__.py", line 2, in <module>
    from .store import Store
  File "/usr/lib/python3/dist-packages/docker/credentials/store.py", line 7, in <module>
    from .utils import create_environment_dict
  File "/usr/lib/python3/dist-packages/docker/credentials/utils.py", line 1, in <module>
    import distutils.spawn
ModuleNotFoundError: No module named 'distutils'

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: python3-docker 5.0.3-1ubuntu1
ProcVersionSignature: Ubuntu 6.8.0-22.22-generic 6.8.1
Uname: Linux 6.8.0-22-generic x86_64
ApportVersion: 2.28.0-0ubuntu1
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Tue Apr 16 16:35:20 2024
InstallationDate: Installed on 2024-04-16 (0 days ago)
InstallationMedia: Ubuntu 24.04 LTS "Noble Numbat" - Beta amd64 (20240410.2)
PackageArchitecture: all
ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
SourcePackage: python-docker
UpgradeStatus: No upgrade log present (probably fresh install)

[A. Karl Kornel (akkornel)]

I've been doing some checking, and it looks like this is because Python 3.12 (the default python3) no longer includes distutils. See https://docs.python.org/3.12/whatsnew/3.12.html (the first paragraph under the heading "Summary – Release highlights"), as well as PEP-0632 (https://peps.python.org/pep-0632/).

I figured this out when I tried downgrading the package to the version in Mantic (5.0.3-1ubuntu1): `dpkg` complained that it depends on `python3-distutils`. That led me to searching for the package in Noble, and getting to this point.

(Adding the `regression-release` tag. Apologies if this is the wrong regression tag to apply!)

BTW, this isn't affecting Debian 12 because they're using Python 3.11, the last Python version to include distutils in the standard library. Debian sid is currently on Python 3.11.8, so they haven't hit this issue yet.

As for upstream, looking at PyPi (https://pypi.org/project/docker/#history), 5.0.3 was the last of the 5.x series. Their latest versions are 6.1.3 and 7.0.0. Their changelog page (https://docker-py.readthedocs.io/en/stable/change-log.html) does not explicitly say when they moved away from distutils, but from Git commits it looks like the removal happened in https://github.com/docker/docker-py/commit/42789818bed5d86b487a030e2e60b02bf0cfa284, which was included in the 6.0.0 and later releases.

So, what to do in the meantime? According to PEP-0632, Setuptools adopted distutils, and this started to happen in Setuptools 48 (https://github.com/pypa/setuptools/issues/417#issuecomment-653593796). Since the python3-setuptools package in Noble is newer (68.1.2-2ubuntu1), probably the best workaround is for `python3-docker` to depend on `python3-setuptools`.

== Regression details ==
Discovered in version:5.0.3-1ubuntu1
Last known good version:5.0.3-1

[A. Karl Kornel (akkornel)]

It looks like things are worse than I thought.

Once I install python3-setuptools, even though the `docker` module now imports successfully, running `docker.from_env()` fails with the following:

Python 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
>>> docker.from_env()
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 497, in _make_request
    conn.request(
TypeError: HTTPConnection.request() got an unexpected keyword argument 'chunked'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/docker/client.py", line 96, in from_env
    return cls(
           ^^^^
  File "/usr/lib/python3/dist-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: HTTPConnection.request() got an unexpected keyword argument 'chunked'

This is https://github.com/docker/docker-py/issues/3113, an issue reported about a year ago. Per the urllib3 maintainer, who commented on the bug:

> I can confirm however that docker-py is not compat...

[A. Karl Kornel (akkornel)]

It looks like https://github.com/docker/docker-py/pull/3116 is the pull request which fixes this issue.

I've only just started using this Python package, so I'm not able to do a full test, but I'm at least able to take a pass at making a patch. I'll provide a patch for both issues: The reliance on setuptools code, and the urllib3 2.0 incompatibility.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Remove the calls to distutils, so that the package no longer relies on python3-distutils or python3-setuptools" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]