Zuul CI is giving false positive on role-addition and molecule consistently

[Sofer Athlan-Guyot]

Hi,

so since 2020-04-29T11:43:33, tripleo-ansible-centos-8-role-addition
fails all the time. If you take 721292,18 it was working and then
started to fail, on the same patchset.

Same is true for
tripleo-ansible-centos-8-molecule-tripleo_redhat_enforce. Since
2020-04-29T15:34:13 it fails with RETRY_LIMIT while the same patchset
was previously working.

Thanks,

[1] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ansible-centos-8-role-addition
[2] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ansible-centos-8-molecule-tripleo_redhat_enforce#

[Marios Andreou (marios-b)]

looks like it could be related to pep517 at last that's where the trace is coming from

12:55 < marios> zbr: well you can see here it is where the error is coming from
12:55 < marios> zbr: 2020-04-30 03:41:04.887579 | centos-8 | File
"/home/zuul/src/opendev.org/openstack/tripleo-ansible/.tox/role-addition/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 76, in _build_backend
12:55 < marios> 2020-04-30 03:41:04.887592 | centos-8 | AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'
12:55 < marios> https://f90bbe801532b1321206-1f2089e8a25dfa4acbcd47153d041690.ssl.cf2.rackcdn.com/721292/18/check/tripleo-ansible-centos-8-role-addition/f39b625/job-output.txt

can we consider pip extra_args --no-use-pep517 for workaround. This blocks the gate (e.g. there https://review.opendev.org/#/c/721292/ )

[Marios Andreou (marios-b)]

following the trace it looks like this fails in the tox role from zuul-jobs so we might not be able to workaround on our side

        * 2020-04-30 03:39:57.060134 | RUN START: [untrusted : opendev.org/zuul/zuul-jobs/playbooks/tox/run.yaml@master]
        * 2020-04-30 03:40:06.842149 | TASK [tox : Run tox without tests]
        * 2020-04-30 03:41:04.887481 | centos-8 | Getting requirements to build wheel: finished with status 'error'
          2020-04-30 03:41:04.887487 | centos-8 | ERROR: Command errored out with exit status 1:
          2020-04-30 03:41:04.887496 | centos-8 | command: /home/zuul/src/opendev.org/openstack/tripleo-ansible/.tox/role-addition/bin/python /home/zuul/src/opendev.org/openstack/tripleo-ansible/.tox/role-addition/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpopu60e02

which is there https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/tasks/siblings.yaml

13:15 < marios> zbr: akahat|ruck: panda|ruck: looks like it is coming from the invocation there
                https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/tasks/siblings.yaml ie zuul-jobs so we can't workaround
                it i think

[Sorin Sbarnea (ssbarnea)]

As I said, this has nothing to so with zuul-jobs code, is a side-effect of enabling system site packages inside tox.ini config, somethign that is known to cause such problems when system packages are in conflict with required dependencies. (there are various ways it can fail).

Take a look at https://review.opendev.org/#/c/724627/ as a proof.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/724643

[OpenStack Infra (hudson-openstack)]

Change abandoned by Marios Andreou (<email address hidden>) on branch: master
Review: https://review.opendev.org/724643
Reason: just posted as an example/reference for an irc conversation

[Marios Andreou (marios-b)]

16:22 < openstackgerrit> Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Disable the use of pep517
                         https://review.opendev.org/724647

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/724670

[wes hayutin (weshayutin)]

https://zuul.openstack.org/builds?job_name=tripleo-ansible-centos-8-role-addition

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/724811

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724670
Committed: https://git.openstack.org/cgit/openstack/tripleo-upgrade/commit/?id=31126cf89b321be5dbc971c467c5c6fb3e1253c7
Submitter: Zuul
Branch: master

commit 31126cf89b321be5dbc971c467c5c6fb3e1253c7
Author: Jesse Pretorius (odyssey4me) <email address hidden>
Date: Thu Apr 30 15:39:27 2020 +0100

    Avoid broken dogpile.cache

    Bug: https://github.com/sqlalchemy/dogpile.cache/issues/178

    Related-Bug: #1876073

    Change-Id: Ia3f7940e836d4bd90b73c7f82fc7203bfe76f2c5

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jesse Pretorius (odyssey4me) (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/724811
Reason: This is not requires as per Sorin's comment.