vault py39 tox job does not run under zuul

[Michel Thebeau [WIND]]

Brief Description
-----------------
On my local machine I can run py69 tox, to run helm units tests. but when i push a code review the job fails.

Severity
--------
Minor

Steps to Reproduce
------------------
Assuming py39 is enabled in vault-armada-app .zuul.yaml and python3-k8sapp-vault/k8sapp_vault/tox.ini: push a misc change for the python code to review.opendev.org

Expected Behavior
------------------
py69 job log should show successful result

Actual Behavior
----------------
As shown in the following example, the job fails with logs including the following text:

"k8sapp-vault-tox-py39 https://zuul.opendev.org/t/openstack/build/1d3c483b0c1d41c0ad51b84ab187f202 : FAILURE in 6m 46s" - https://review.opendev.org/c/starlingx/vault-armada-app/+/899630

"testtools.matchers._impl.MismatchError: 'server' not in {}" - https://zuul.opendev.org/t/openstack/build/b28089d0de5347bbb1f536f4a59da04c/console

Reproducibility
---------------
When attempting to enable the py39 job per review 899630: 100%. Otherwise the job will remain disable until this issue is resolved.

System Configuration
--------------------
build and code-review

Branch/Pull Time/Commit
-----------------------
master branch

Last Pass
---------
June 30 2023: https://review.opendev.org/c/starlingx/vault-armada-app/+/882348

Timestamp/Logs
--------------
per previous sections

Test Activity
-------------
code review

Workaround
----------
"works on my machine"

[Michel Thebeau [WIND] (mthebeau)]

The successful log on my local machine includes:

py39 installed: alabaster==0.7.8,alembic==1.4.3,amqp==5.0.3,amqplib==1.0.2,attrs==20.3.0,Babel==2.8.0,bcrypt==3.1.7,beautifulsoup4==4.9.3,boto3==1.28.73,botocore==1.31.73,cachetools==4.2.1,certifi==2020.6.20,cffi==1.14.5,-e git+https://opendev.org/starlingx/update.git@e6744eb9f5014b83e6de73bb0be59148b6cc2f2f#egg=cgcs_patch&subdirectory=../../../../cgcs-root/stx/update/sw-patch/cgcs-patch,chardet==4.0.0,cliff==3.4.0,cmd2==0.8.5,coverage==7.3.2,cryptography==3.3.2,debtcollector==2.2.0,decorator==4.4.2,discover==0.4.0,dnspython==1.16.0,docker==4.1.0,docutils==0.16,dogpile.cache==1.0.2,eventlet==0.26.1,exceptiongroup==1.1.3,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,flake8==3.9.2,-e git+https://opendev.org/starlingx/fault.git@1fb9ffdc545070e8aa52420e718f7b9ef78a5af9#egg=fm_api&subdirectory=../../../../cgcs-root/stx/fault/fm-api/source,-e git+https://opendev.org/starlingx/fault.git@1fb9ffdc545070e8aa52420e718f7b9ef78a5af9#egg=fmclient&subdirectory=../../../../cgcs-root/stx/fault/python-fmclient/fmclient,future==0.18.2,futurist==2.3.0,gevent==23.9.1,google-auth==2.23.3,greenlet==3.0.1,hacking==2.0.0,idna==2.10,imagesize==1.2.0,iniconfig==2.0.0,ipaddress==1.0.23,iso8601==0.1.13,jeepney==0.8.0,Jinja2==2.11.3,jmespath==0.10.0,jsonpatch==1.25,jsonpointer==2.0,jsonschema==3.2.0,keyring==22.0.1,keystoneauth1==4.2.1,keystonemiddleware==9.1.0,kombu==5.0.2,kubernetes==22.6.0,linecache2==1.0.0,logutils==0.3.5,lxml==4.6.3,Mako==1.1.3,MarkupSafe==1.1.1,mccabe==0.6.1,migrate==0.3.8,mock==5.1.0,monotonic==1.5,msgpack==1.0.0,netaddr==0.7.19,netifaces==0.10.9,numpy==1.19.5,oauthlib==3.1.0,os-service-types==1.7.0,oslo.cache==2.6.1,oslo.concurrency==4.3.1,oslo.config==8.3.3,oslo.context==3.1.1,oslo.db==8.4.0,oslo.i18n==5.0.1,oslo.log==4.4.0,oslo.messaging==12.5.2,oslo.middleware==4.1.1,oslo.policy==3.5.0,oslo.rootwrap==7.1.0,oslo.serialization==4.0.2,oslo.service==2.4.0,oslo.utils==4.6.0,oslosphinx==4.18.0,oslotest==4.5.0,packaging==20.9,paramiko==2.7.2,passlib==1.7.4,Paste==3.5.0,PasteDeploy==2.1.1,pbr==5.5.0,pecan==1.3.3,pluggy==1.3.0,prettytable==0.7.2,psutil==5.9.6,psycopg2-binary==2.9.9,pyasn1==0.4.8,pyasn1-modules==0.3.0,pycadf==3.1.1,pycodestyle==2.7.0,pycparser==2.21,pycryptodomex==3.9.7,pyflakes==2.3.1,pyghmi==1.5.64,Pygments==2.7.1,pyinotify==0.9.6,PyNaCl==1.4.0,pyOpenSSL==20.0.1,pyparsing==2.4.7,pyperclip==1.8.0,pyrsistent==0.15.5,pytest==7.4.3,python-barbicanclient==5.0.1,-e git+https://opendev.org/starlingx/utilities.git@b327116a1f4191f7b2aa0f8927200ff294eab642#egg=python_cephclient&subdirectory=../../../../cgcs-root/stx/utilities/ceph/python-cephclient/python-cephclient,python-cinderclient==7.2.0,python-dateutil==2.8.1,python-editor==1.0.3,python-glanceclient==3.2.2,python-keystoneclient==4.1.1,python-ldap==3.2.0,python-mimeparse==1.6.0,python-subunit==1.4.3,pytz==2021.1,pyudev==0.22.0,PyYAML==6.0,pyzmq==20.0.0,repoze.lru==0.7,requests==2.25.1,requests-mock==1.11.0,requests-oauthlib==1.3.1,retrying==1.3.4,rfc3986==1.4.0,Routes==2.5.1,rsa==4.9,ruamel.yaml==0.18.3,ruamel.yaml.clib==0.2.8,s3transfer==0.7.0,SecretStorage==3.3.1,simplegeneric==0.8.1,simplejson==3.19.2,six==1.16.0,snowballstemmer==2.1.0,...

[Michel Thebeau [WIND] (mthebeau)]

This abandoned review demonstrates that I'm breaking the py39 job somehow:
https://review.opendev.org/c/starlingx/vault-armada-app/+/899876

I've updated my review, which was originally intended to enable flake8 and bandit, to omit changes to zuul and tox. Those jobs py39 and pylint are working without my attempts.
https://review.opendev.org/c/starlingx/vault-armada-app/+/899630

[Michel Thebeau [WIND] (mthebeau)]

Joshua worked to align and correct tox jobs across repos. Following commit for vault resolves this bug report.

fd1d13a0083cba2172f77bc0fde83b61c4d49d27
Author: Reed, Joshua <email address hidden>
Date: Mon Dec 18 06:56:04 2023 -0700

    Update app Zuul Check Jobs.

    Modify code to conform to flake8 and pylint.

    Jobs are now flake8, pylint, py39 and metadata.

    Test Plan
    PASS - All zuul jobs pass as expected.

    Story: 2010929
    Task: 49283

    Change-Id: I3e3f5191a2dac94e35b75bccdd563dc108f187bf
    Signed-off-by: Reed, Joshua <email address hidden>