openstack-tox-molecule jobs are failing with "ImportError: cannot import name 'ansible_version'"

[Ronelle Landy]

openstack-tox-molecule jobs are failing jobs on tripleo-quickstart and tripleo-quickstart-extras with the following error:

2020-10-15 17:29:10.656828 | ubuntu-bionic | [2893] /home/zuul/src/opendev.org/openstack/tripleo-quickstart$ /home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/bin/python -m pytest --color=yes --html=/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/log/reports.html --self-contained-html
2020-10-15 17:29:11.230249 | ubuntu-bionic | Traceback (most recent call last):
2020-10-15 17:29:11.230324 | ubuntu-bionic | File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
2020-10-15 17:29:11.230340 | ubuntu-bionic | "__main__", mod_spec)
2020-10-15 17:29:11.230353 | ubuntu-bionic | File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
2020-10-15 17:29:11.230365 | ubuntu-bionic | exec(code, run_globals)
2020-10-15 17:29:11.230378 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/pytest/__main__.py", line 7, in <module>
2020-10-15 17:29:11.230391 | ubuntu-bionic | raise SystemExit(pytest.console_main())
2020-10-15 17:29:11.230404 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/_pytest/config/__init__.py", line 180, in console_main
2020-10-15 17:29:11.230416 | ubuntu-bionic | code = main()
2020-10-15 17:29:11.230428 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/_pytest/config/__init__.py", line 136, in main
2020-10-15 17:29:11.230440 | ubuntu-bionic | config = _prepareconfig(args, plugins)
2020-10-15 17:29:11.230452 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/_pytest/config/__init__.py", line 314, in _prepareconfig
2020-10-15 17:29:11.230469 | ubuntu-bionic | pluginmanager=pluginmanager, args=args
2020-10-15 17:29:11.230481 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__
2020-10-15 17:29:11.230498 | ubuntu-bionic | return self._hookexec(self, self.get_hookimpls(), kwargs)
2020-10-15 17:29:11.230510 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec
2020-10-15 17:29:11.230523 | ubuntu-bionic | return self._inner_hookexec(hook, methods, kwargs)
2020-10-15 17:29:11.230535 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/pluggy/manager.py", line 87, in <lambda>
2020-10-15 17:29:11.230547 | ubuntu-bionic | firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
2020-10-15 17:29:11.230573 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/pluggy/callers.py", line 203, in _multicall
2020-10-15 17:29:11.230587 | ubuntu-bionic | gen.send(outcome)
2020-10-15 17:29:11.230599 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/_pytest/helpconfig.py", line 99, in pytest_cmdline_parse
2020-10-15 17:29:11.230612 | ubuntu-bionic | config = outcome.get_result() # type: Config
2020-10-15 17:29:11.230624 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
2020-10-15 17:29:11.230639 | ubuntu-bionic | raise ex[1].with_traceback(ex[2])
2020-10-15 17:29:11.230651 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
2020-10-15 17:29:11.230663 | ubuntu-bionic | res = hook_impl.function(*args)
2020-10-15 17:29:11.230675 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/_pytest/config/__init__.py", line 932, in pytest_cmdline_parse
2020-10-15 17:29:11.230687 | ubuntu-bionic | self.parse(args)
2020-10-15 17:29:11.230699 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/_pytest/config/__init__.py", line 1204, in parse
2020-10-15 17:29:11.230711 | ubuntu-bionic | self._preparse(args, addopts=addopts)
2020-10-15 17:29:11.230723 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/_pytest/config/__init__.py", line 1097, in _preparse
2020-10-15 17:29:11.230735 | ubuntu-bionic | self.pluginmanager.load_setuptools_entrypoints("pytest11")
2020-10-15 17:29:11.230746 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/pluggy/manager.py", line 299, in load_setuptools_entrypoints
2020-10-15 17:29:11.230759 | ubuntu-bionic | plugin = ep.load()
2020-10-15 17:29:11.230771 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 105, in load
2020-10-15 17:29:11.230783 | ubuntu-bionic | module = import_module(match.group('module'))
2020-10-15 17:29:11.230794 | ubuntu-bionic | File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
2020-10-15 17:29:11.230807 | ubuntu-bionic | return _bootstrap._gcd_import(name[level:], package, level)
2020-10-15 17:29:11.230819 | ubuntu-bionic | File "<frozen importlib._bootstrap>", line 994, in _gcd_import
2020-10-15 17:29:11.230833 | ubuntu-bionic | File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2020-10-15 17:29:11.230845 | ubuntu-bionic | File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2020-10-15 17:29:11.230856 | ubuntu-bionic | File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
2020-10-15 17:29:11.230868 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
2020-10-15 17:29:11.230880 | ubuntu-bionic | exec(co, module.__dict__)
2020-10-15 17:29:11.230907 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/lib/python3.6/site-packages/pytest_molecule/__init__.py", line 19, in <module>
2020-10-15 17:29:11.230922 | ubuntu-bionic | from molecule.config import ansible_version
2020-10-15 17:29:11.230934 | ubuntu-bionic | ImportError: cannot import name 'ansible_version'
2020-10-15 17:29:11.271558 | ubuntu-bionic | ERROR: InvocationError for command /home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/bin/python -m pytest --color=yes --html=/home/zuul/src/opendev.org/openstack/tripleo-quickstart/.tox/molecule/log/reports.html --self-contained-html (exited with code 1)
2020-10-15 17:29:1

The job started failing on 10/15:
https://zuul.openstack.org/builds?job_name=openstack-tox-molecule

[Ronelle Landy (rlandy)]

Using ansible==2.9.14 - same as on passing jobs

[Ronelle Landy (rlandy)]

pytest-molecule==1.3.1 pytest-molecule==1.3.0

[Ronelle Landy (rlandy)]

passing tests have pytest-molecule==1.3.0, failing tests have pytest-molecule==1.3.1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/758502

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/758503

[Ronelle Landy (rlandy)]

tripleo-quickstart-extras fix:

https://review.opendev.org/758503 Restrict pytest-molecule to < 1.3.1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/758503
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=4b5e68d3bd0bdc4cc70d7db318e02cb2ca462ed1
Submitter: Zuul
Branch: master

commit 4b5e68d3bd0bdc4cc70d7db318e02cb2ca462ed1
Author: Ronelle Landy <email address hidden>
Date: Thu Oct 15 17:36:59 2020 -0400

    Restrict pytest-molecule to < 1.3.1

    With pytest-molecule==1.3.1, openstack-tox-molecule
    tests fail with:
    ImportError: cannot import name 'ansible_version'

    Locking the pytest-molecule until the version bug is
    addressed.

    Change-Id: If33f664a73498b66f12550f7aacf92be192f8951
    Related-Bug: #1900033

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/758502
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=d0e3b33cdcd1633c6e6437ee8de9a9149e378cdc
Submitter: Zuul
Branch: master

commit d0e3b33cdcd1633c6e6437ee8de9a9149e378cdc
Author: Sorin Sbarnea <email address hidden>
Date: Fri Oct 16 11:38:37 2020 +0100

    Temporary pin-down with newer pytest-molecule

    Temporary workaround until new release of molecule/pytest-molecule
    bug is fixed.

    Change-Id: I2721e0a30b6ca63b279fe9c388eed2e4c0ffc347
    Partial-Bug: #1900033

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/759270

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759270
Committed: https://git.openstack.org/cgit/openstack/tripleo-operator-ansible/commit/?id=e6cbed6a99dad3928eb397f8ebbd0ea580e80015
Submitter: Zuul
Branch: master

commit e6cbed6a99dad3928eb397f8ebbd0ea580e80015
Author: Alex Schultz <email address hidden>
Date: Thu Oct 22 08:07:53 2020 -0600

    Fix molecule requirements

    There's a pytest-molecule bug around ansible version in the latest
    version. Pin it to <1.30 for now.

    Additionally move the requirements out of tox.ini into it's own file to
    match other repositories.

    Change-Id: I516b38fa0766a9aaa15c00495cae4a781b769c4c
    Partial-Bug: #1900033

[Sorin Sbarnea (ssbarnea)]

Please note that pinning down is no longer needed since last weekend as all molecule related tools were upgraded to avoid depending on ansible directly.

[Alex Schultz (alex-schultz)]

Except I hit it yesterday, so I don't think that's correct. You may need to double check what you think fixed it

[wes hayutin (weshayutin)]

http://dashboard-ci.tripleo.org/d/3-DYSmOGk/jobs-exploration?orgId=1&from=now-7d&to=now&var-influxdb_filter=job_name%7C%3D%7Copenstack-tox-molecule

I'm not seeing any failures.. closing.