openstack-tox-pep8 failure in stable/wallaby: import name 'OrderedDict' from 'typing_extensions'

[Damien Ciabrini]

During the setup of the openstack-tox-pep8 job on stable/wallaby, a Python exception is raised when starting ansible-lint:

[1966] /home/zuul/src/opendev.org/openstack/tripleo-heat-templates$ /home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/bin/ansible-lint -v
Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/bin/ansible-lint", line 5, in <module>
    from ansiblelint.__main__ import _run_cli_entrypoint
  File "/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/lib/python3.8/site-packages/ansiblelint/__main__.py", line 35, in <module>
    from enrich.console import should_do_markup
  File "/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/lib/python3.8/site-packages/enrich/console.py", line 6, in <module>
    import rich.console as rich_console
  File "/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/lib/python3.8/site-packages/rich/console.py", line 46, in <module>
    from ._log_render import FormatTimeCallable, LogRender
  File "/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/lib/python3.8/site-packages/rich/_log_render.py", line 5, in <module>
    from .text import Text, TextType
  File "/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/lib/python3.8/site-packages/rich/text.py", line 5, in <module>
    from rich.emoji import EmojiVariant
  File "/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/lib/python3.8/site-packages/rich/emoji.py", line 4, in <module>
    from .jupyter import JupyterMixin
  File "/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/lib/python3.8/site-packages/rich/jupyter.py", line 4, in <module>
    from .segment import Segment
  File "/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/lib/python3.8/site-packages/rich/segment.py", line 19, in <module>
    from .cells import (
  File "/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/lib/python3.8/site-packages/rich/cells.py", line 6, in <module>
    from ._lru_cache import LRUCache
  File "/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/lib/python3.8/site-packages/rich/_lru_cache.py", line 8, in <module>
    from typing_extensions import OrderedDict
ImportError: cannot import name 'OrderedDict' from 'typing_extensions' (/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/lib/python3.8/site-packages/typing_extensions.py)
ERROR: InvocationError for command /home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/bin/ansible-lint -v (exited with code 1)
pep8 finish: run-test after 51.86 seconds
pep8 start: run-test-post

[Rabi Mishra (rabi)]

Sounds like typing_extensions uc for wallaby is 3.7.4.3. However, ansible-lint expects 4.1.1[2].
The issue has surfaced after openstacksdk was bumped to use proper constraints[3].

We can probably try pinning ansible-lint < 6.0.0.

[1] https://github.com/openstack/requirements/blob/stable/wallaby/upper-constraints.txt#L574
[2] https://github.com/ansible/ansible-lint/blob/v6.0.1/requirements.txt#L57
[3] https://github.com/openstack/requirements/commit/8b04528e19dbc0c9ebc2b0674eebc3f2bcc86a4a

[Rabi Mishra (rabi)]

> We can probably try pinning ansible-lint < 6.0.0.

That does not help.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836453

[Rabi Mishra (rabi)]

I tried a few things.

- Adding basepython = python3.9 for pep8 target fixes the issue. However, 3.9 is not supported for wallaby[1].

- Pinning rich as done in https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836453 works but the constraints check fails as rich is not in openstack requirements/uc.

I'm kind of out of ideas on how we can fix it (without bumping typing_extensions) in openstack/releases uc. Someone familiar with ansible-lint and how it's expected to work should help fix it.

[1]https://github.com/openstack/governance/blob/master/reference/runtimes/wallaby.rst

[Marios Andreou (marios-b)]

thanks Rabi.. i saw this yesterday and filed the duplicate at https://bugs.launchpad.net/tripleo/+bug/1968065 ... i tried the pinning... pin down works but then causes requirements check to fail, and pin up introduces more deps issues (comments 1 and 2 on /+bug/1968065) and I was going to dig into those today.

However, I think removing uc from the pep8 check is a fine idea ;) lets go with https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836453 \o/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836453
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/7f094f32075800cd37ccb6f68c1d18ef7800214e
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7f094f32075800cd37ccb6f68c1d18ef7800214e
Author: rabi <email address hidden>
Date: Thu Apr 7 09:02:06 2022 +0530

    Don't use upper-constraints for pep8 target

    We've ansible-lint in test requirements which is not constrained[1]
    by uc and it's dependencies when constrained create issues.

    [1] https://github.com/openstack/requirements/blob/master/blacklist.txt#L4

    Closes-Bug: #1967788
    Change-Id: I34dcd9aef9b236ec75b1efbfe0f1c503473e0ad9