Zuul fails with either 404 or Invalid version: 'cpython'

[Leonardo Fagundes Luz Serrano]

Brief Description
-----------------
Reviews fail zuul and logs mention either:

- Some package failed download with a 404 error
- "pip._vendor.packaging.version.InvalidVersion: Invalid version: 'cpython'"

Tox doesn't fail locally.

Severity
--------
Critical - No reviews can merge

Steps to Reproduce
------------------
Send a review to one of the affected repos, which include most apps.

Expected Behavior
------------------
Zuul passes if the review can pass tox locally

Actual Behavior
----------------
Reviews failing zuul

Reproducibility
---------------
Reproducible

System Configuration
--------------------
NA

Branch/Pull Time/Commit
-----------------------
20240625

Last Pass
---------
Last week

Timestamp/Logs
--------------

======> 404

2024-06-24 06:56:22.872325 | debian-bullseye | Fetching project page and analyzing links: https://mirror.bhs1.ovh.opendev.org/wheel/debian-11.9-x86_64/tomli/
2024-06-24 06:56:22.872332 | debian-bullseye | Getting page https://mirror.bhs1.ovh.opendev.org/wheel/debian-11.9-x86_64/tomli/
2024-06-24 06:56:22.872340 | debian-bullseye | Found index url https://mirror.bhs1.ovh.opendev.org/wheel/debian-11.9-x86_64/
2024-06-24 06:56:22.872353 | debian-bullseye | Looking up "https://mirror.bhs1.ovh.opendev.org/wheel/debian-11.9-x86_64/tomli/" in the cache
2024-06-24 06:56:22.872362 | debian-bullseye | Request header has "max_age" as 0, cache bypassed
2024-06-24 06:56:22.872369 | debian-bullseye | No cache entry available
2024-06-24 06:56:22.872388 | debian-bullseye | https://mirror.bhs1.ovh.opendev.org:443 "GET /wheel/debian-11.9-x86_64/tomli/ HTTP/1.1" 404 196
2024-06-24 06:56:22.872395 | debian-bullseye | Status code 404 not in (200, 203, 300, 301, 308)
2024-06-24 06:56:22.872402 | debian-bullseye | Could not fetch URL https://mirror.bhs1.ovh.opendev.org/wheel/debian-11.9-x86_64/tomli/: 404 Client Error: Not Found for url: https://mirror.bhs1.ovh.opendev.org/wheel/debian-11.9-x86_64/tomli/ - skipping
2024-06-24 06:56:22.872415 | debian-bullseye | Skipping link: not a file: https://mirror.bhs1.ovh.opendev.org/wheel/debian-11.9-x86_64/tomli/
2024-06-24 06:56:22.872423 | debian-bullseye | Skipping link: not a file: https://mirror.bhs1.ovh.opendev.org/pypi/simple/tomli/
2024-06-24 06:56:22.872430 | debian-bullseye | Given no hashes to check 14 links for project 'tomli': discarding no candidates
2024-06-24 06:56:22.872437 | debian-bullseye | Collecting tomli>=1.1.0 (from pylint->-r /home/zuul/src/opendev.org/starlingx/app-node-interface-metrics-exporter/python3-k8sapp-node-interface-metrics-exporter/k8sapp_node_interface_metrics_exporter/test-requirements.txt (line 19))
2024-06-24 06:56:22.872444 | debian-bullseye | Obtaining dependency information for tomli>=1.1.0 from https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl.metadata
2024-06-24 06:56:22.872452 | debian-bullseye | Created temporary directory: /tmp/pip-unpack-zg4lfl2g
2024-06-24 06:56:22.872458 | debian-bullseye | Found index url https://mirror.bhs1.ovh.opendev.org/pypi/simple/
2024-06-24 06:56:22.872465 | debian-bullseye | Looking up "https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl.metadata" in the cache
2024-06-24 06:56:22.872472 | debian-bullseye | No cache entry available
2024-06-24 06:56:22.872491 | debian-bullseye | No cache entry available
2024-06-24 06:56:22.872498 | debian-bullseye | Resetting dropped connection: mirror.bhs1.ovh.opendev.org
2024-06-24 06:56:22.872517 | debian-bullseye | https://mirror.bhs1.ovh.opendev.org:443 "GET /pypifiles/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl.metadata HTTP/1.1" 200 8875
2024-06-24 06:56:22.872527 | debian-bullseye | Downloading https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl.metadata (8.9 kB)
2024-06-24 06:56:22.872534 | debian-bullseye | Ignoring unknown cache-control directive: immutable
2024-06-24 06:56:22.872542 | debian-bullseye | Updating cache with response from "https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl.metadata"
2024-06-24 06:56:22.872549 | debian-bullseye | etag object cached for 1209600 seconds
2024-06-24 06:56:22.872561 | debian-bullseye | Caching due to etag

===> cpython

ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3070, in _dep_map
    return self.__dep_map
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2863, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/cli/base_command.py", line 179, in exc_logging_wrapper
    status = run_func(*args)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
    return bool(self._sequence)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 174, in __bool__
    return any(self)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 162, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 53, in _iter_built
    candidate = func()
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/factory.py", line 185, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/factory.py", line 231, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 303, in __init__
    super().__init__(
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
    self.dist = self._prepare()
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 247, in _prepare
    self._check_metadata_consistency(dist)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 229, in _check_metadata_consistency
    list(dist.iter_dependencies(list(dist.iter_provided_extras())))
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/metadata/pkg_resources.py", line 250, in iter_provided_extras
    return self._extra_mapping.keys()
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/metadata/pkg_resources.py", line 94, in _extra_mapping
    canonicalize_name(extra): extra for extra in self._dist.extras
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3026, in extras
    return [dep for dep in self._dep_map if dep]
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3072, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3089, in _compute_dependencies
    common = types.MappingProxyType(dict.fromkeys(reqs_for_extra(None)))
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3086, in reqs_for_extra
    if not req.marker or req.marker.evaluate({'extra': extra}):
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/packaging/markers.py", line 325, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/packaging/markers.py", line 225, in _evaluate_markers
    groups[-1].append(_eval_op(lhs_value, op, rhs_value))
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/packaging/markers.py", line 183, in _eval_op
    return spec.contains(lhs, prereleases=True)
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/packaging/specifiers.py", line 552, in contains
    normalized_item = _coerce_version(item)
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/packaging/specifiers.py", line 28, in _coerce_version
    version = Version(version)
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/packaging/version.py", line 202, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
pip._vendor.packaging.version.InvalidVersion: Invalid version: 'cpython'

Test Activity
-------------
Posting commits for review

Workaround
----------
Unknown

[Leonardo Fagundes Luz Serrano (lfagunde)]

The "cpython" issue was reproduced locally. Turns out to be an incompatibility between pip 24.1 and pyzmq 20.0.0

Steps to reproduce:

docker run -it debian:bullseye bash

apt-get update
apt-get install python3
apt-get install pip
pip install pip==24.1
pip install --root-user-action=ignore pyzmq==20.0.0

Version 21.0.2 does not reproduce the error.

[Leonardo Fagundes Luz Serrano (lfagunde)]

Related bug report:
https://bugs.launchpad.net/kolla/+bug/2006727

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/922771
Committed: https://opendev.org/starlingx/root/commit/79677da02bb5fb9012930d688eadfd8d4eb53b52
Submitter: "Zuul (22348)"
Branch: master

commit 79677da02bb5fb9012930d688eadfd8d4eb53b52
Author: Leonardo Fagundes Luz Serrano <email address hidden>
Date: Tue Jun 25 17:13:05 2024 -0300

    Zuul: Fix pyzmq - pip version incompatibility

    pip has released version 24.1 recently and
    it is incompatible with pyzmq 20.0.0

    To work around this, changed pyzmq versions used
    to 21.0.2, which does not present this error.
    This version is also available in zuul mirrors:
    https://mirror.dfw.rax.opendev.org/wheel/debian-11-x86_64/pyzmq/

    Reference:
    https://bugs.launchpad.net/kolla/+bug/2006727

    Test Plan:
    pass: pip install pip==24.1;
          pip install --root-user-action=ignore pyzmq==21.0.2

    Closes-Bug: 2070407

    Change-Id: I7a61dd0ed2be592e7550a36c800c8923fbc6fd60
    Signed-off-by: Leonardo Fagundes Luz Serrano <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/root/+/923165