tripleo-ci-centos-7-containers-multinode-train/stein/stein/queens failing with ERROR: Package 'python-heatclient' requires a different Python: 2.7.5 not in '>=3.6'.

[Sandeep Yadav]

Below jobs are failing with ERROR: Package 'python-heatclient' requires a different Python: 2.7.5 not in '>=3.6'.

tripleo-ci-centos-7-containers-multinode-train
tripleo-ci-centos-7-containers-multinode-stein
tripleo-ci-centos-7-containers-multinode-stein
tripleo-ci-centos-7-containers-multinode-queens

Error snippet:-

~~~
2020-03-26 06:50:27.769014 | primary | ERROR: Package 'python-heatclient' requires a different Python: 2.7.5 not in '>=3.6'
2020-03-26 06:50:28.384140 | primary | ERROR
2020-03-26 06:50:28.384576 | primary | {
2020-03-26 06:50:28.384693 | primary | "delta": "0:00:45.148427",
2020-03-26 06:50:28.384803 | primary | "end": "2020-03-26 06:50:27.812948",
2020-03-26 06:50:28.384909 | primary | "msg": "non-zero return code",
2020-03-26 06:50:28.385012 | primary | "rc": 1,
2020-03-26 06:50:28.385120 | primary | "start": "2020-03-26 06:49:42.664521"
2020-03-26 06:50:28.385222 | primary | }
2020-03-26 06:50:28.599240 |
~~~

URL:-

~~~

https://130486e6097cfc48efe3-316ead88afa2be91e0f5b80937dadc42.ssl.cf2.rackcdn.com/714609/5/check/tripleo-ci-centos-7-containers-multinode-train/b6d1495/job-output.txt

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_932/714609/5/check/tripleo-ci-centos-7-containers-multinode-stein/932032a/job-output.txt

https://a54d2fc98a60fcd93973-00f5302f444d38fb0cb9bb58094cf5c2.ssl.cf5.rackcdn.com/714609/5/check/tripleo-ci-centos-7-containers-multinode-rocky/52ca577/job-output.txt

https://c76ef7bd3445ba6e591d-9b787c7d8fce93ea012fc251d0fca893.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/tripleo-quickstart-extras/master/tripleo-ci-centos-7-containers-multinode-queens/c412138/job-output.txt
~~~

[chandan kumar (chkumar246)]

It can be reproduced easily
❯ virtualenv -p /usr/bin/python2.7 .venv
Running virtualenv with interpreter /usr/bin/python2.7
Already using interpreter /usr/bin/python2.7
New python executable in /tmp/.venv/bin/python2.7
Also creating executable in /tmp/.venv/bin/python
Installing setuptools, pip, wheel...
done.

/tmp took 3s
❯ source .venv/bin/activate

/tmp via 🐍 v2.7.17 (.venv)
❯ pip install http://mirror.mtl01.inap.opendev.org/pypifiles/packages/31/f2/8ced1e395a1b69042d764ca0014ee977a5cbaf8f152bb1de4550093a0ef3/python-heatclient-2.0.0.tar.gz
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting http://mirror.mtl01.inap.opendev.org/pypifiles/packages/31/f2/8ced1e395a1b69042d764ca0014ee977a5cbaf8f152bb1de4550093a0ef3/python-heatclient-2.0.0.tar.gz
  Downloading http://mirror.mtl01.inap.opendev.org/pypifiles/packages/31/f2/8ced1e395a1b69042d764ca0014ee977a5cbaf8f152bb1de4550093a0ef3/python-heatclient-2.0.0.tar.gz (172 kB)
     |████████████████████████████████| 172 kB 110 kB/s
ERROR: Package 'python-heatclient' requires a different Python: 2.7.17 not in '>=3.6'

Somehow python-heatclient-2.0.0.tar.gz is getting pulled on python-2.7 in ci

I need we need to update TQ requirements.txt
https://github.com/openstack/tripleo-quickstart/blob/master/requirements.txt#L12
with following line:
python-heatclient<2.0.0;python_version=='2.7' # Apache-2.0
python-heatclient;python_version>='3.6' # Apache-2.0

will fix the issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/715179

[Sandeep Yadav (sandeepyadav93)]

It seems more like a mirror issue

* In my local env I was not able to reproduce without mirror info http://pastebin.test.redhat.com/848616

* Added mirror info and then able to reproduce.

~~~
[root@centos-vm tripleo-quickstart]# cat /etc/pip.conf
[global]
timeout = 60
index-url = http://mirror.mtl01.inap.opendev.org/pypi/simple
trusted-host = mirror.mtl01.inap.opendev.org
extra-index-url = http://mirror.mtl01.inap.opendev.org/wheel/centos-7.7-x86_64
[root@centos-vm tripleo-quickstart]#
[root@centos-vm tripleo-quickstart]#
[root@centos-vm tripleo-quickstart]# virtualenv -p /usr/bin/python2.7 .venv2
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in /root/tripleo-quickstart/.venv2/bin/python2.7
Also creating executable in /root/tripleo-quickstart/.venv2/bin/python
Installing setuptools, pip, wheel...done.
[root@centos-vm tripleo-quickstart]# source .venv2/bin/activate
(.venv2) [root@centos-vm tripleo-quickstart]# pip install .
Processing /root/tripleo-quickstart
Collecting cmd2==0.8.5 (from tripleo-quickstart==2.1.2.dev275)
  Downloading http://mirror.mtl01.inap.opendev.org/pypifiles/packages/ee/02/faa35704449a68f18db39ee9a047e32c8b569e672daadfef3501259fe6bb/cmd2-0.8.5-py2.py3-none-any.whl (51kB)
    100% |████████████████████████████████| 61kB 1.3MB/s
Collecting ara<1.0 (from tripleo-quickstart==2.1.2.dev275)
  Downloading http://mirror.mtl01.inap.opendev.org/pypifiles/packages/a4/d5/019731cd1fb75ef819b86fda4f2e77a32eb1a0035c598285c202cb500129/ara-0.16.6-py2.py3-none-any.whl (164kB)
    100% |████████████████████████████████| 174kB 4.0MB/s
Collecting ansible<2.10,>=2.9 (from tripleo-quickstart==2.1.2.dev275)
  Downloading http://mirror.mtl01.inap.opendev.org/pypifiles/packages/ae/b7/c717363f767f7af33d90af9458d5f1e0960db9c2393a6c221c2ce97ad1aa/ansible-2.9.6.tar.gz (14.2MB)
    100% |████████████████████████████████| 14.2MB 29.7MB/s
Collecting jmespath (from tripleo-quickstart==2.1.2.dev275)
  Downloading http://mirror.mtl01.inap.opendev.org/pypifiles/packages/a3/43/1e939e1fcd87b827fe192d0c9fc25b48c5b3368902bfb913de7754b0dc03/jmespath-0.9.5-py2.py3-none-any.whl
Collecting netaddr>=0.7.18 (from tripleo-quickstart==2.1.2.dev275)
  Downloading http://mirror.mtl01.inap.opendev.org/pypifiles/packages/ba/97/ce14451a9fd7bdb5a397abf99b24a1a6bb7a1a440b019bebd2e9a0dbec74/netaddr-0.7.19-py2.py3-none-any.whl (1.6MB)
    100% |████████████████████████████████| 1.6MB 27.7MB/s
Collecting os-client-config (from tripleo-quickstart==2.1.2.dev275)
  Downloading http://mirror.mtl01.inap.opendev.org/pypifiles/packages/1b/bf/66b7e355f9de07644f8778f8d418fb6ed66e65ca8ad39b85a39358f6b955/os_client_config-2.0.0-py2.py3-none-any.whl
Collecting pbr>=1.6 (from tripleo-quickstart==2.1.2.dev275)
  Downloading http://mirror.mtl01.inap.opendev.org/pypifiles/packages/7a/db/a968fd7beb9fe06901c1841cb25c9ccb666ca1b9a19b114d1bbedf1126fc/pbr-5.4.4-py2.py3-none-any.whl (110kB)
    100% |████████████████████████████████| 112kB 4.2MB/s
Requirement already satisfied: setuptools>=11.3 in ./.venv2/lib/python2.7/site-packages (from tripleo-quickstart==2.1.2.dev275)
Collecting shade>=1.8.0 (from tripleo-quickstart==2.1.2.dev275)
  Downloading http://mi...

[wes hayutin (weshayutin)]

@Sandeep @Chandan
Two thoughts here.. please take a look at https://review.opendev.org/#/c/715287/4/requirements.txt

1. I think we can remove some of the requirements in tq/requirements.txt as shade and the openstackclients are no longer used. We can also use bindep for the openstackclients w/ rpms if needed.

2. As things start to fall apart in the py27 world it would be a good idea to utilize the upper-constraints file in stein or train for the chain of python deps we incur from the following file [a]. Note that we inject tripleo-upgrade at run time in the same way. So to test this locally add "git+https://opendev.org/openstack/tripleo-upgrade/#egg=tripleo-upgrade" to quickstart-extras-requirements.txt locally.

FYI.. I am testing this locally w/ the following.
./quickstart.sh -b -n -w /var/tmp/wes

[a] https://opendev.org/openstack/tripleo-quickstart/src/branch/master/quickstart-extras-requirements.txt

[wes hayutin (weshayutin)]

diff list https://www.diffchecker.com/Mzy5attc

[Cédric Jeanneret (cjeanner)]

Apparently python-heatclient isn't the only one:

2020-03-27 07:24:16.066652 | primary | ERROR: Package 'cliff' requires a different Python: 2.7.5 not in '>=3.6'

[yatin (yatinkarel)]

Actually issue is not with projects, it's with infra mirrors so likely multiple pip source packages are affected.

I checked some mirrors and issue can be seen clearly:-
pip install --index-url=http://mirror.sjc1.vexxhost.openstack.org/pypi/simple --trusted-host=mirror.sjc1.vexxhost.openstack.org --extra-index-url=//mirror.sjc1.vexxhost.openstack.org/wheel/centos-7.7-x86_64 cliff

With pypi cdn this will pass and with some other infra mirror it may fail. Infra mirrros are not in complete sync with pypi sources mainly data-requires-python attribute is missing which causing issues, see [1]
Example:- pip install --index-url=http://mirror.mtl01.inap.opendev.org/pypi/simple --trusted-host=mirror.mtl01.inap.opendev.org --extra-index-url=http://mirror.mtl01.inap.opendev.org/wheel/centos-7-x86_64 cliff
or pip install cliff

[1] view-source:http://mirror.mtl01.inap.opendev.org/pypi/simple/cliff/ vs view-source:https://pypi.org/simple/cliff/
copying issue part here as it can change with mirror sync:-

<a href="/pypifiles/packages/90/e4/624f02aa2fcbf6efcd9d6bf90f92836a2ae46bc4376a824e317d10506fc8/cliff-3.0.0-py3-none-any.whl#sha256=c34ade07f7226b8f15af775413f6f305717fde577009887e1c56fa8ac5a8c9ed">cliff-3.0.0-py3-none-any.whl</a><br/>
<a href="/pypifiles/packages/17/7f/0704826112e089d4e7d188818b825b7e16810749228cf92be12558d71c4c/cliff-3.0.0.tar.gz#sha256=89e86767ba575c6b9f55a0b9d39918c5e665367744d754475a93ffac15377ec1">cliff-3.0.0.tar.gz</a><br/>

vs

<a href="https://files.pythonhosted.org/packages/90/e4/624f02aa2fcbf6efcd9d6bf90f92836a2ae46bc4376a824e317d10506fc8/cliff-3.0.0-py3-none-any.whl#sha256=c34ade07f7226b8f15af775413f6f305717fde577009887e1c56fa8ac5a8c9ed" data-requires-python="&gt;=3.6">cliff-3.0.0-py3-none-any.whl</a><br/>
<a href="https://files.pythonhosted.org/packages/17/7f/0704826112e089d4e7d188818b825b7e16810749228cf92be12558d71c4c/cliff-3.0.0.tar.gz#sha256=89e86767ba575c6b9f55a0b9d39918c5e665367744d754475a93ffac15377ec1" data-requires-python="&gt;=3.6">cliff-3.0.0.tar.gz</a><br/>

[yatin (yatinkarel)]

<< Actually issue is not with projects, it's with infra mirrors so likely multiple pip source packages are affected.
All mirrors seems fine now(data-requires-python contents synched), jobs are getting green now.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715287
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=a5860c8f9e0a23d25d9fa9d4abac9683fdc81ed4
Submitter: Zuul
Branch: master

commit a5860c8f9e0a23d25d9fa9d4abac9683fdc81ed4
Author: Wes Hayutin <email address hidden>
Date: Thu Mar 26 12:09:56 2020 -0600

    removing old requirements, pin other reqs for 2.7

    pin:
    futurist
    clif
    oslo.x

    removing requirements that are breaking old
    branches and are no longer used directly.
    openstacksdk requires futurist but only
    on centos8 py3

    Closes-Bug: #1869161
    Depends-On: https://review.opendev.org/#/c/715324/
    Change-Id: I224b950bf33d365c83ecac68c2b9a0092bb969bb

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chandan Kumar (raukadah) (<email address hidden>) on branch: master
Review: https://review.opendev.org/715179
Reason: no longer needed