Centos 7 train jobs failing with pymod2pkg requiring python 3.6

[Arx Cruz]

https://1c854b43b98bfdb525a8-06ff0b77c4b7bc18f9f8408eaae15058.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/tripleo-quickstart-extras/master/tripleo-ci-centos-7-containers-multinode-stein/ab1f655/job-output.txt

2020-11-24 07:51:07.768583 | primary | stdout: Looking in indexes: https://mirror.bhs1.ovh.opendev.org/pypi/simple, https://mirror.bhs1.ovh.opendev.org/wheel/centos-7.8-x86_64
2020-11-24 07:51:07.769322 | primary | Collecting rdopkg
2020-11-24 07:51:07.770054 | primary | Downloading https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/67/b2/ad9379800e858323509337bf706cd97ffae4039f3e422f2adee56c7d21b0/rdopkg-1.3.0.tar.gz (139 kB)
2020-11-24 07:51:07.770358 | primary | Collecting blessings
2020-11-24 07:51:07.770583 | primary | Downloading https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/8d/b1/a3fe6fd8a012e6d019bafd671c2fee0597ea97ff2e76c25aadfa4545fc32/blessings-1.7-py2-none-any.whl (26 kB)
2020-11-24 07:51:07.770948 | primary | Collecting distroinfo>=0.3.0
2020-11-24 07:51:07.771209 | primary | Downloading https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/88/e5/e3f6a502251476966273a2065befb38d37687a58c50a3afd9dd92895f4a1/distroinfo-0.3.2-py2-none-any.whl (18 kB)
2020-11-24 07:51:07.771291 | primary | Collecting munch
2020-11-24 07:51:07.771566 | primary | Downloading https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/cc/ab/85d8da5c9a45e072301beb37ad7f833cd344e04c817d97e0cc75681d248f/munch-2.5.0-py2.py3-none-any.whl (10 kB)
2020-11-24 07:51:07.771899 | primary | Collecting pbr>=0.5.6
2020-11-24 07:51:07.772125 | primary | Downloading https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/fb/48/69046506f6ac61c1eaa9a0d42d22d54673b69e176d30ca98e3f61513e980/pbr-5.5.1-py2.py3-none-any.whl (106 kB)
2020-11-24 07:51:07.772196 | primary | Collecting PyYAML
2020-11-24 07:51:07.772403 | primary | Downloading https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz (269 kB)
2020-11-24 07:51:07.772456 | primary | Collecting pyOpenSSL>=16.2.0
2020-11-24 07:51:07.772675 | primary | Downloading https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/9e/de/f8342b68fa9e981d348039954657bdf681b2ab93de27443be51865ffa310/pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)
2020-11-24 07:51:07.772954 | primary | Collecting pymod2pkg
2020-11-24 07:51:07.773192 | primary | Downloading https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/8d/9b/975981cd55a0c5157696516fe15d33d969b67b7f30e3b5fc021f047addd5/pymod2pkg-0.26.0.tar.gz (17 kB)
2020-11-24 07:51:07.773225 | primary |
2020-11-24 07:51:07.774355 | primary | :stderr: DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
2020-11-24 07:51:07.774903 | primary | ERROR: Package 'pymod2pkg' requires a different Python: 2.7.5 not in '>=3.6'
2020-11-24 07:51:07.774950 | primary |
2020-11-24 07:51:07.774978 | primary |

[Javier Peña (jpena-c)]

Following advice from ykarel, we can reproduce this if we run the following command-line on a python2 venv:

$ pip install --index-url="https://mirror.bhs1.ovh.opendev.org/pypi/simple" --extra-index-url="https://mirror.bhs1.ovh.opendev.org/wheel/centos-7.8-x86_64" --trusted-host=mirror.bhs1.ovh.opendev.org rdopkg

However, when using a different mirror, it works as expected:

$ pip install --index-url="https://mirror.regionone.vexxhost.rdoproject.org/pypi/simple" --extra-index-url="https://mirror.regionone.vexxhost.rdoproject.org/wheel/centos-7.8-x86_64" --trusted-host=mirror.regionone.vexxhost.rdoproject.org rdopkg

[Javier Peña (jpena-c)]

I have tested the above reproduced, and it works as expected with the following mirrors:

- mirror01.bhs1.ovh.opendev.org
- mirror01.ca-ymq-1.vexxhost.opendev.org
- mirror01.dfw.rax.opendev.org
- mirror01.iad.rax.opendev.org
- mirror01.kna1.airship-citycloud.opendev.org
- mirror01.ord.rax.opendev.org
- mirror01.regionone.limestone.opendev.org
- mirror01.us-east.openedge.opendev.org
- mirror02.gra1.ovh.opendev.org
- mirror02.mtl01.inap.opendev.org
- mirror02.regionone.linaro-us.opendev.org

Note that the bhs1 OVH mirror is named "mirror01" instead of "mirror" as in the failing log output. Also, I see both hostnames point to the same IP, so it could be some different vhost configuration.

[Martin Schuppert (mschuppert)]

I hit this in https://review.opendev.org/c/openstack/tripleo-common/+/763721 / https://zuul.opendev.org/t/openstack/build/bc0d7cc1ab814e6cbcb96b838961a0e7

[Javier Peña (jpena-c)]

I have tried again today with mirror.bhs1.ovh.opendev.org as a pypi mirror, and now it seems to work as expected. Can you test if this is the case in the CI jobs as well?

[Rafael Folco (rafaelfolco)]

Using centos-7-containers-multinode-stein from the original report as baseline, we got 3 green runs in a row today:
https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode-stein

Closing. Reopen if needed.

[yatin (yatinkarel)]

Would be good to move away with python2 in venv as quickstart is already switched to python3 in venv to avoid similar issues in future.

[yatin (yatinkarel)]

In progress Patch:- https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/764232

[Marios Andreou (marios-b)]

I am moving the bug to fix-released because https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode-stein is very green

I know there is still a related patch at https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/764232 but it isn't about fixing the job but rather more like preventive action