wallaby gate blocker tripleo-ci-centos-8-standalone ERROR: Cannot install stackviz

[Marios Andreou]

At [1][2] the tripleo-ci-centos-8-standalone fails for stable/wallaby jobs and blocks the gate. During the os_tempest role execution, the python_venv_build role runs and fails during python package installs with conflict - trace like:

        2022-05-30 08:55:06.326662 | primary | fatal: [undercloud]: FAILED! => {"attempts": 5, "changed": false, "cmd": ["/home/zuul/stackviz_venv/bin/pip", "install", "-U", "--constraint", "/home/zuul/stackviz_venv/global-constraints.txt", "--constraint", "/home/zuul/stackviz_venv/constraints.txt", "--pre", "--log", "/var/log/python_venv_build.log", "--isolated", "https://tarballs.opendev.org/openstack/stackviz/dist/stackviz-latest.tar.gz"], "msg": "stdout: Looking in indexes: https://mirror.mtl01.iweb.opendev.org/pypi/simple, https://mirror.mtl01.iweb.opendev.org/wheel/centos-8-x86_64\nCollecting https://tarballs.opendev.org/openstack/stackviz/dist/stackviz-latest.tar.gz\n Using cached https://tarballs.opendev.org/openstack/stackviz/dist/stackviz-latest.tar.gz (764 kB)\n Preparing metadata (setup.py): started\n Preparing metadata (setup.py): finished with status 'done'\nCollecting python-subunit>=0.0.18\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/88/de/be89fff5c30810afb101bdac2c8706764fafa8e950dc6ffa22092ad81035/python_subunit-1.4.0-py2.py3-none-any.whl (106 kB)\nCollecting testtools>=0.9.30\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/c0/49/b2b4956528cca6954cb3a8016a8283282ccd1a1d66ab1c2d1bbde3f66946/testtools-2.5.0-py3-none-any.whl (181 kB)\nCollecting testrepository>=0.0.18\n Using cached https://mirror.mtl01.iweb.opendev.org/wheel/centos-8-x86_64/testrepository/testrepository-0.0.20-py3-none-any.whl (118 kB)\nCollecting subunit2sql>=0.8.0\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/0a/1b/e1aba9b1ea76c1818d1e629f2071b9604321d8bf85fef1128e6b4c7aaaa6/subunit2sql-1.10.0-py2.py3-none-any.whl (98 kB)\nCollecting six\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (11 kB)\nCollecting extras\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/03/e9/e915af1f97914cd0bc021e125fd1bfd4106de614a275e4b6866dd9a209ac/extras-1.0.0-py2.py3-none-any.whl (7.3 kB)\nCollecting stevedore>=1.20.0\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/7a/bc/fcce9e50da73ea23af6d236e05e15db8a02da1099a5e0a479451bcea3833/stevedore-3.5.0-py3-none-any.whl (49 kB)\nCollecting python-dateutil>=2.4.2\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)\nCollecting alembic>=0.8.10\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/b3/e2/8d48220731b7279911c43e95cd182961a703b939de6822b00de3ea0d3159/alembic-1.7.7-py3-none-any.whl (210 kB)\nCollecting oslo.db>=4.24.0\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/29/85/bbb2d8d0cc79c45d3d61ccaf257cbac722628ad602e2d9551658b456a06b/oslo.db-11.3.0-py3-none-any.whl (175 kB)\nCollecting SQLAlchemy!=1.1.5,!=1.1.6,!=1.1.7,!=1.1.8,>=1.0.10\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/2d/57/b2bf4c7adb045ce05f4f2648e7938aef96bc2909be433b1c650071f6a466/SQLAlchemy-1.4.36-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)\nCollecting oslo.config>=4.0.0\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/c5/4c/6cfd9274f3fb665f276ec41bd7c98fce20d5ec903bac7b4cd2c30f5832fe/oslo.config-8.8.0-py3-none-any.whl (128 kB)\nCollecting pbr!=2.1.0,>=2.0.0\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/1b/ef/0621aa0e422dda72dc89b61972e5a28d5a947c22fdff83a5d69fdbc1949a/pbr-5.9.0-py2.py3-none-any.whl (112 kB)\nCollecting fixtures\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/a8/28/7eed6bf76792f418029a18d5b2ace87ce7562927cdd00f1cefe481cd148f/fixtures-3.0.0-py2.py3-none-any.whl (67 kB)\nINFO: pip is looking at multiple versions of testtools to determine which version is compatible with other requirements. This could take a while.\nINFO: pip is looking at multiple versions of testrepository to determine which version is compatible with other requirements. This could take a while.\nINFO: pip is looking at multiple versions of six to determine which version is compatible with other requirements. This could take a while.\nINFO: pip is looking at multiple versions of subunit2sql to determine which version is compatible with other requirements. This could take a while.\nCollecting subunit2sql>=0.8.0\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/30/ec/ba2964b2040f6700429f640632f16bce1dacd6d2ee254592bc2ba42fa6c3/subunit2sql-1.9.0-py2.py3-none-any.whl (99 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/35/bd/cfb11e9d44af8684c929f1cea930211f38a9339adeb7f34931425d0da938/subunit2sql-1.8.0-py2.py3-none-any.whl (95 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/f5/17/1b5f9e0aa269b66f2f807f1e0d339ec0973637e325ebd82fa49ab34d6e0a/subunit2sql-1.7.0-py2.py3-none-any.whl (92 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/da/06/eb6cfc28025642bf1ebf60c07ee8c49c94d310b4c51f64e924dc036fad68/subunit2sql-1.6.0-py2.py3-none-any.whl (91 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/dd/4c/35d6af4579f3cd45a11f492a2fe437a0d906185425525dbe26c6a1a2487a/subunit2sql-1.5.0-py2.py3-none-any.whl (90 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/06/9d/2323fed7e01d7ebf4dce99939ba9b69ace397b8f2caa4837bc74e5460344/subunit2sql-1.4.0-py2.py3-none-any.whl (89 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/41/a5/bca5b2a165e89790a590f2419baa8b4f59fc56baafcb5de6721adce48b12/subunit2sql-1.3.0-py2.py3-none-any.whl (87 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/90/48/1dee5c9e2aaadaedcd44892e6ba9a34b44fda1808cf492d9a04a73942db9/subunit2sql-1.2.0-py2.py3-none-any.whl (86 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/6d/5f/e8872f7bc5c83b96760101e196c226ebaa385407ac5e4aae2eb8d7f4ac8b/subunit2sql-1.1.1-py2.py3-none-any.whl (80 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/d7/f0/49d1b41e6b2bd0965788b22d44568f41839e12ced09b0a9e9bf2a2eb9107/subunit2sql-1.1.0-py2.py3-none-any.whl (79 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/43/2e/117aaf93d62edccc9504edc9ee1a40cff4d6a353330395d3497de3889336/subunit2sql-1.0.2-py2.py3-none-any.whl (79 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/cc/7e/5b2af1012b0db866b01d4f1d3820f70796e29f85fbec7d839bf28aebe405/subunit2sql-1.0.1-py2.py3-none-any.whl (78 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/dc/51/ec67d653e737d1f599cb9dbb86966d3adbad9ffbf735d093d2e4c390f4d4/subunit2sql-1.0.0-py2.py3-none-any.whl (77 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/ad/a4/bdb40b5ff8195f124db94f22917dd1b5fdbeb8b8f4b77f59df732cb54136/subunit2sql-0.12.0-py2.py3-none-any.whl (69 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/19/a8/dd791dff6915abc0d27fcfdb1b4a41d1ddb2713ed2ef70b9cb6d86567913/subunit2sql-0.11.0-py2.py3-none-any.whl (67 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/b1/fa/8c73ce63c08ec08acee63d9d46ba13f327d661ad9ebe6cf397f1aed82541/subunit2sql-0.10.0-py2.py3-none-any.whl (62 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/1f/6e/8d367e3835bcec033bd326898ead6be6de590670a92fbf7de4ba797df6e4/subunit2sql-0.9.0-py2.py3-none-any.whl (60 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/a2/0d/b312dc04d10331b1ef218f5b1bcd87b22c581bf3e8dbcca628ed567ab9a1/subunit2sql-0.8.1-py2.py3-none-any.whl (59 kB)\n Using cached https://mirror.mtl01.iweb.opendev.org/pypifiles/packages/f6/51/2dd80a612450b9d929d8f6981f0353edf6ae4dc485f6116b615c1eab94c8/subunit2sql-0.8.0-py2.py3-none-any.whl (59 kB)\nINFO: pip is looking at multiple versions of python-subunit to determine which version is compatible with other requirements. This could take a while.\nINFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.\nINFO: pip is looking at multiple versions of stackviz to determine which version is compatible with other requirements. This could take a while.\n\nThe conflict is caused by:\n subunit2sql 1.5.0 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 1.4.0 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 1.3.0 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 1.2.0 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 1.1.1 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 1.1.0 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 1.0.2 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 1.0.1 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 1.0.0 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 0.12.0 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 0.11.0 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 0.10.0 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 0.9.0 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 0.8.1 depends on oslo.db!=1.12.0 and <2.0.0\n subunit2sql 0.8.0 depends on oslo.db!=1.12.0 and <2.0.0\n The user requested (constraint) oslo-db===11.3.0\n\nTo fix this you could try to:\n1. loosen the range of package versions you've specified\n2. remove package versions to allow pip attempt to solve the dependency conflict\n\n\n:stderr: ERROR: Cannot install stackviz because these package versions have conflicting dependencies.\nERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies\n"}

This issue is also reported at [3].

Thanks @ysandeep for the info indeed this must be related to py36 support.

For master we dropped it with [4]. For stable/wallaby we need it for centos8 jobs, or we need to find another way to overcome that conflict.

[1] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3af/840477/1/check/tripleo-ci-centos-8-standalone/3af6916/job-output.txt
[2] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b7b/839184/3/gate/tripleo-ci-centos-8-standalone/b7bc8d9/job-output.txt
[3] https://bugzilla.redhat.com/show_bug.cgi?id=2091502
[4] https://review.opendev.org/q/topic:tripleo-zed-jobs

[Marios Andreou (marios-b)]

Looks like we're using the *master* upper constraints here [1]

        * oslo.db===11.3.0

But we should be using the wallaby one at [2]

        * oslo.db===8.5.1

[1] https://github.com/openstack/requirements/blob/7ac0e631833ee8c33b42ddbcc23be89c05dcc658/upper-constraints.txt#L478

[2] https://github.com/openstack/requirements/blob/20be78e7b3ef4c3e43ad06745e5ffcb0f1a87576/upper-constraints.txt#L466

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/843841

[Marios Andreou (marios-b)]

On the ansible-os_tempest role the default at [1] is pulling in master

i am trying to override it for the tripleo side with zuul.branch at [2] and testing with [3]

[1] https://opendev.org/openstack/openstack-ansible-os_tempest/src/commit/1869a9f138bcbbdbb53f2686cd567f71e073472d/defaults/main.yml#L61-L62

[2] https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/843841

[3] https://review.opendev.org/c/openstack/tripleo-ansible/+/843842

[Marios Andreou (marios-b)]

no this is not coming from ansible-os_tempest but from python _venv_build (called by the tempest role):

2022-05-30 08:54:00.578554 | primary | TASK [python_venv_build : Install python packages into the venv] ***************

so still digging...

[Marios Andreou (marios-b)]

coming from there https://opendev.org/openstack/ansible-role-python_venv_build/src/commit/e7307f5d01db07cc504eb619654a0829c9d4e92c/tasks/python_venv_install.yml#L152

[Marios Andreou (marios-b)]

we may end up disabling stackviz to unblock the gate

the code in ansible-role-python_venv is a bit convoluted wrt the defaults e.g.

- name: Install python packages into the venv
  block:
    - name: Install python packages into the venv
      pip:
        name: "{{ _venv_install_pip_packages }}"
        state: "{{ venv_pip_package_state }}"
        virtualenv: "{{ venv_install_destination_path }}"
        extra_args: >-
          --constraint {{ venv_install_destination_path }}/global-constraints.txt
          --constraint {{ venv_install_destination_path }}/constraints.txt

i've tried to trace how the _venv_install_pip_packages (actually defined in this same file [1]) and constraints.txt and global-constraints.txt are formed and it is not obvious to follow :/

As there are other things that need pressing attention I am contemplating disabling this/stackviz as we already have for c9 already there [2] (thanks chkumar for reminder)

But not even sure if that will work. The earlier fix we have at [2] is for os_tempest itself whereas the failing task i nthis case is from ansible-role-python-venv_build

[1] https://opendev.org/openstack/ansible-role-python_venv_build/src/commit/e7307f5d01db07cc504eb619654a0829c9d4e92c/tasks/python_venv_install.yml#L29
[2] https://opendev.org/openstack/tripleo-quickstart-extras/src/commit/be5e48aeedf5ecd5ee4acdbed7c4fa1be2db0d50/playbooks/tasks/tempest.yml#L87

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/843856

[Marios Andreou (marios-b)]

trying https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/843856 & test @ https://review.opendev.org/c/openstack/tripleo-ansible/+/843857 (for disable)

also posted blocker warning to list at http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028750.html [tripleo] stable/wallaby gate blocker for centos-8 standalone - please hold recheck

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/843856
Committed: https://opendev.org/openstack/tripleo-quickstart-extras/commit/8bcf4bfc0f2596776ca6abf284cce6337e097e3d
Submitter: "Zuul (22348)"
Branch: master

commit 8bcf4bfc0f2596776ca6abf284cce6337e097e3d
Author: Marios Andreou <email address hidden>
Date: Mon May 30 16:06:35 2022 +0300

    Disable stackviz for tempest centos 8 to unblock gate

    As commented in related bug (see comments/6) we need to disable
    installation of stackviz as there are issues with the constraints
    used. This is not a proper fix but if it works can unblock for now

    Related-Bug: 1976247

    Change-Id: I8c5e168a9fd18332c073223726ad3a3e338af04d

[Marios Andreou (marios-b)]

we are unblocked for now with https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/843856

moving to fix released for now

there will be more investigation needed if we want to re-enable stackviz

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ghanshyam <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/843841
Reason: TrieplO project is retiring now, for details, please see https://review.opendev.org/c/openstack/governance/+/905145 or reach out to OpenStack TC.