tox-py36-ci-config job is broken

[Amol Kahat]

Job tox-py36-ci-config is broken

Actual Results:
2020-12-14 16:08:40.878604 | rdo-centos-8 | Looking in indexes: http://mirror.regionone.vexxhost-nodepool-sf.rdoproject.org/pypi/simple, http://mirror.regionone.vexxhost-nodepool-sf.rdoproject.org/wheel/centos-8.3-x86_64
2020-12-14 16:08:40.878623 | rdo-centos-8 | ERROR: Could not find a version that satisfies the requirement mock==2.0.0
2020-12-14 16:08:40.878634 | rdo-centos-8 | ERROR: No matching distribution found for mock==2.0.0

Job link:
https://review.rdoproject.org/zuul/builds?job_name=tox-py36-ci-config

[Amol Kahat (amolkahat)]

Not only mock[1] and other packages are also broken [2]

[1] https://logserver.rdoproject.org/14/28014/44/check/tox-py36-ci-config/1db957c/job-output.txt
[2] https://logserver.rdoproject.org/14/28014/44/check/tox-py36-ci-config/1db957c/job-output.txt

[Amol Kahat (amolkahat)]

Correct link:

[2] https://logserver.rdoproject.org/58/31358/1/check/tox-py36-ci-config/bb922f5/job-output.txt

[Marios Andreou (marios-b)]

@Amol yes agree... I have tried updating both of those with https://review.rdoproject.org/r/#/c/31358/3/test-requirements.txt and it gets further but then fails on some more requirements problems there:

        * https://logserver.rdoproject.org/58/31358/3/check/tox-py36-ci-config/eed7741/job-output.txt
        * 2020-12-15 15:13:28.263648 | rdo-centos-8 | ERROR: No matching distribution found for httplib2==0.11.3

We are setting that for rrcockpit in https://github.com/rdo-infra/ci-config/blob/c25934c03d15d317501bcf63ba2cdb737596f719/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf/requirements.txt#L1 * httplib2==0.11.3

This is a real mess... I am going to try unpinning those ^^^ and see how far it can get.

[Marios Andreou (marios-b)]

This is all to do with the new pip resolver [1]. After updating the various locked requirements [2] it get further but eventually times out at [5].

Looking through [5] the entire half hour run is taken by dependency discovery and resolution. For example this [6] is what it does to discover available packages for a single dependency dlrnapi_client (extracted from [5] to illustrate the issue).

There is a relevant warning printed by pip in [5]

 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking

I just bumped to 60 mins to see if it will even complete in that time but nope it takes the whole 1 hour for deps resolution [8].

The new resolver is known to take longer, however we are making it worse by having too many requirements there [7].

Perhaps one solution is to split those into separate jobs?

[1] https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020
[2] https://review.rdoproject.org/r/#/c/31358/9/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf/requirements.txt
[3] https://review.rdoproject.org/r/#/c/31358/9/test-requirements.txt
[4] https://review.rdoproject.org/r/#/c/31358/9/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf/requirements.txt
[5] https://logserver.rdoproject.org/58/31358/8/check/tox-py36-ci-config/6d836f0/tox/py36-1.log
[6] http://paste.openstack.org/raw/801115/
[7] https://github.com/rdo-infra/ci-config/blob/c25934c03d15d317501bcf63ba2cdb737596f719/tox.ini#L44-L48
[8] https://logserver.rdoproject.org/58/31358/9/check/tox-py36-ci-config/cd11408/tox/py36-1.log

[chandan kumar (chkumar246)]

https://review.rdoproject.org/r/#/c/28014/58/tox.ini@12 setting download = True in tox.ini will update the pip, setuptools and wheels in the tox venv.

If this does not work, let's fallback to legacy resolver: --use-deprecated=legacy-resolver

[chandan kumar (chkumar246)]

If all the above does not work, let's split py36 in three jobs.

[Marios Andreou (marios-b)]

fix for this was merged there:

https://review.rdoproject.org/r/#/c/31399/ Fix tox-py36-ci-config

[Marios Andreou (marios-b)]

moving the bug fix-released though sorin has said he is still working here there may be further updates

but the tox-py36-ci-config is no longer blocked after https://review.rdoproject.org/r/#/c/31399/ Fix

[Sorin Sbarnea (ssbarnea)]

While current patch avoided the endless install, a further patch from https://bugs.launchpad.net/tripleo/+bug/1908274/ should address the root cause and make the code work with any version of pip.

Extra bonus would be that we will no longer get breakages from updated dependencies, as this pins the dependencies that we use during testing and adds a simple command to update them.

[Sorin Sbarnea (ssbarnea)]

Oops, wrong paste url, patch is at https://review.rdoproject.org/r/31400