zuul jobs failing for some repos running ubuntu-jammy

[Al Bailey]

Brief Description
-----------------
This openstack-zuul-job submission https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/861116 merged over the weekend, causing all the openstack-tox-docs (linters, etc..) jobs to use jammy by default.

Not all of the STX repos are setup to properly run tox in that environment.

Some of the errors are:
c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
     15 | #include <ffi.h>
        | ^~~~~~~
  compilation terminated.

This is due to missing thirdparty components.

At the moment only distcloud is seeing this issue, but other repos may also need to be updated.

Severity
--------
Major

Steps to Reproduce
------------------
Submit a gerrit review in stx/distcloud

Expected Behavior
------------------
It should pass zuul

Actual Behavior
----------------
Failing openstack-tox-docs and openstack-tox-linters

Reproducibility
---------------
100%

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
Nov 21, 2022

Last Pass
---------
Nov 19, 2022

Timestamp/Logs
--------------
c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
     15 | #include <ffi.h>
        | ^~~~~~~
  compilation terminated.

Test Activity
-------------
Code submission

Workaround
----------
We can always turn off that zuul job, or override it with a different nodeset

A review is being uploaded to ensure bindep requirements are satisfied but that does not appear to resolve the issue
https://review.opendev.org/c/starlingx/distcloud/+/865148

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/distcloud/+/865148
Committed: https://opendev.org/starlingx/distcloud/commit/d8b1a08d1e73a71aa34b29a2d116fe85e0dd2fe8
Submitter: "Zuul (22348)"
Branch: master

commit d8b1a08d1e73a71aa34b29a2d116fe85e0dd2fe8
Author: Al Bailey <email address hidden>
Date: Mon Nov 21 15:35:42 2022 +0000

    Fix openstack-tox jobs for DC repo

    Bindep updates are required to get cffi to install on
    ubuntu-jammy.

    This update adds a bindep.txt to ensure Zuul will install
    the required third party entries

    The Zuul upper-constraints env variable declaration needed
    to be added to tox.ini otherwise an older constraints
    was being used which does not work with newer
    versions of python.

    Partial-Bug: #1997255

    Signed-off-by: Al Bailey <email address hidden>
    Change-Id: I7e4b032f9c91bfaab33fec49f23613004101f44b

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/gui/+/865211

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ha/+/865212

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/fault/+/865213

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/gui/+/865211
Committed: https://opendev.org/starlingx/gui/commit/fd9dfbfd9b34d6467f18b5487e06a1c4c78b09f4
Submitter: "Zuul (22348)"
Branch: master

commit fd9dfbfd9b34d6467f18b5487e06a1c4c78b09f4
Author: Al Bailey <email address hidden>
Date: Tue Nov 22 01:04:55 2022 +0000

    Fix openstack-tox jobs for gui repo

    The Zuul upper-constraints env variable declaration needed
    to be added to tox.ini otherwise an older constraints
    was being used which does not work with newer
    versions of python.

    Adding legacy-resolver to attempt to resolve dependency
    issues when importing different stx client modules.

    Partial-Bug: #1997255

    Signed-off-by: Al Bailey <email address hidden>
    Change-Id: I88b779138aa3b19aca57e8f03f3c17ec077e7d58

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/fault/+/865213
Committed: https://opendev.org/starlingx/fault/commit/cf658fba9882f8fc7fcffed998b82de17134f7ff
Submitter: "Zuul (22348)"
Branch: master

commit cf658fba9882f8fc7fcffed998b82de17134f7ff
Author: Al Bailey <email address hidden>
Date: Tue Nov 22 01:15:17 2022 +0000

    Fix openstack-tox jobs for fault repo

    The Zuul upper-constraints env variable declaration needed
    to be added to tox.ini otherwise an older constraints
    was being used which does not work with newer
    versions of python.

    Partial-Bug: #1997255

    Signed-off-by: Al Bailey <email address hidden>
    Change-Id: Ie912dc7ae3f9f1639311f0c1f5cf62070f44909d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ha/+/865212
Committed: https://opendev.org/starlingx/ha/commit/e18364ed2c4e2bc4afadb1b4a9c8797c6de87844
Submitter: "Zuul (22348)"
Branch: master

commit e18364ed2c4e2bc4afadb1b4a9c8797c6de87844
Author: Al Bailey <email address hidden>
Date: Tue Nov 22 01:12:53 2022 +0000

    Fix openstack-tox jobs for ha repo

    The Zuul upper-constraints env variable declaration needed
    to be added to tox.ini otherwise an older constraints
    was being used which does not work with newer
    versions of python.

    Needed to suppress bugbear error B009 also, due to newer
    version of flake8-bugbear.

    Partial-Bug: #1997255

    Signed-off-by: Al Bailey <email address hidden>
    Change-Id: I73f4fbbf51a09f1e4c1a633a7a8b62cf969ff33a

[Al Bailey (albailey1974)]

I am going to list this as fixed.
There were 4 repos affected:
distcloud
ha
fault
nfv

If others are detected, I can update them in their own bugs

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/update/+/865322

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/update/+/865322
Committed: https://opendev.org/starlingx/update/commit/4e0bfeab6c0159d3029a2ed1f2565decc634b6c6
Submitter: "Zuul (22348)"
Branch: master

commit 4e0bfeab6c0159d3029a2ed1f2565decc634b6c6
Author: Al Bailey <email address hidden>
Date: Tue Nov 22 18:40:35 2022 +0000

    Fix openstack-tox jobs for update repo

    The bindep entry for python2 rpm was causing Zuul
    setup on ubuntu-jammy nodes to report
    "No package matching 'python-rpm' is available"

    Partial-Bug: #1997255

    Signed-off-by: Al Bailey <email address hidden>
    Change-Id: Ib1abeee22ec0c186846e68242bb89c8ac2fce114

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/distcloud/+/867147
Committed: https://opendev.org/starlingx/distcloud/commit/ede5de8f332509d353a5d682b48f6848e076bd23
Submitter: "Zuul (22348)"
Branch: master

commit ede5de8f332509d353a5d682b48f6848e076bd23
Author: Al Bailey <email address hidden>
Date: Fri Dec 9 22:19:12 2022 +0000

    Fix failing zuul jobs due to tox showconfig

    As part of how we nest the tox.ini files in STX,
    the environments in the sub directories need to also
    be defined in the tox.ini in the base file.

    I suspect the tox_extra_args change here may have exposed
    the problem in some of our tox files.
    https://review.opendev.org/c/zuul/zuul-jobs/+/866935

    Partial-Bug: #1997255

    Signed-off-by: Al Bailey <email address hidden>
    Change-Id: Ia963897be5f844599d2f6145f472f5eac1df6805

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/distcloud-client/+/867665

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/distcloud-client/+/867665
Committed: https://opendev.org/starlingx/distcloud-client/commit/40459e1dc9775adc882620177bff326b464be605
Submitter: "Zuul (22348)"
Branch: master

commit 40459e1dc9775adc882620177bff326b464be605
Author: Al Bailey <email address hidden>
Date: Wed Dec 14 14:40:01 2022 +0000

    Fix failing zuul jobs due to tox showconfig

    As part of how we nest the tox.ini files in STX,
    the environments in the sub directories need to also
    be defined in the tox.ini in the base file.

    The issue in our tox files was exposed by this commit
    https://review.opendev.org/c/zuul/zuul-jobs/+/866935

    Partial-Bug: #1997255

    Signed-off-by: Al Bailey <email address hidden>
    Change-Id: I526066015b90dc0637968defbc3648b310a27d44

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/app-istio/+/868351

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/integ/+/868488

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/openstack-armada-app/+/868489

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/868491

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/update/+/868495

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/868491
Committed: https://opendev.org/starlingx/config/commit/94034b91f967fa81b628ac498134f886ebed0a68
Submitter: "Zuul (22348)"
Branch: master

commit 94034b91f967fa81b628ac498134f886ebed0a68
Author: Al Bailey <email address hidden>
Date: Fri Dec 23 14:46:01 2022 +0000

    Update tox.ini to work with tox 4

    This change will allow this repo to pass zuul now
    that this has merged:
    https://review.opendev.org/c/zuul/zuul-jobs/+/866943

    Tox 4 deprecated whitelist_externals.
    Replace whitelist_externals with allowlist_externals

    skipsdist and usedevelop are incompatible in tox 4.
    Setting tox < 4 in .zuul.yaml rather than trying to
    figure out why the sysinv component is not properly
    installing in tox.

    Partial-Bug: #1997255

    Signed-off-by: Al Bailey <email address hidden>
    Change-Id: If8e339d20e57e0830ea4ce27c098bacfef2f2ef8

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/app-istio/+/868351
Committed: https://opendev.org/starlingx/app-istio/commit/f33032c1d05748e95b05bad882e22325bccabe14
Submitter: "Zuul (22348)"
Branch: master

commit f33032c1d05748e95b05bad882e22325bccabe14
Author: Al Bailey <email address hidden>
Date: Wed Dec 21 15:17:53 2022 +0000

    Fix zuul pylint setup failures

    The pylint zuul target will now be setup the same as the
    py39 target.

    The pip legacy resolver is still needed because not all
    of the other STX dependencies being installed by tox
    are aligned properly for python 3.9

    Enforce the nodeset for pylint to be the same as py39

    Tox 4 was enabled by:
    https://review.opendev.org/c/zuul/zuul-jobs/+/866943

    Tox 4 deprecated whitelist_externals.
    Replace whitelist_externals with allowlist_externals

    Partial-Bug: #1997255
    Partial-Bug: #2000399
    Signed-off-by: Al Bailey <email address hidden>
    Change-Id: I7d7ca4531600bd863a045d3daeb0be82fec34377