[stable branches] devstack-tobiko-neutron job Fails with InvocationError('could not find executable python', None)

[yatin]

It started failing[1] since the job switched to ubuntu-jammy[2].

Fails as below:-
2023-09-13 16:46:18.124882 | TASK [tobiko-tox : run sanity test cases before creating resources]
2023-09-13 16:46:19.463567 | controller | neutron_sanity create: /home/zuul/src/opendev.org/x/tobiko/.tox/py3
2023-09-13 16:46:20.518574 | controller | neutron_sanity installdeps: -c/home/zuul/src/opendev.org/x/tobiko/upper-constraints.txt, -r/home/zuul/src/opendev.org/x/tobiko/requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/test-requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/extra-requirements.txt
2023-09-13 16:46:20.519390 | controller | ERROR: could not install deps [-c/home/zuul/src/opendev.org/x/tobiko/upper-constraints.txt, -r/home/zuul/src/opendev.org/x/tobiko/requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/test-requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/extra-requirements.txt]; v = InvocationError('could not find executable python', None)
2023-09-13 16:46:20.520263 | controller | ___________________________________ summary ____________________________________
2023-09-13 16:46:20.555843 | controller | ERROR: neutron_sanity: could not install deps [-c/home/zuul/src/opendev.org/x/tobiko/upper-constraints.txt, -r/home/zuul/src/opendev.org/x/tobiko/requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/test-requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/extra-requirements.txt]; v = InvocationError('could not find executable python', None)
2023-09-13 16:46:21.141713 | controller | ERROR
2023-09-13 16:46:21.142024 | controller | {
2023-09-13 16:46:21.142117 | controller | "delta": "0:00:01.484351",
2023-09-13 16:46:21.142197 | controller | "end": "2023-09-13 16:46:20.556249",
2023-09-13 16:46:21.142276 | controller | "failed_when_result": true,
2023-09-13 16:46:21.142353 | controller | "msg": "non-zero return code",
2023-09-13 16:46:21.142688 | controller | "rc": 1,
2023-09-13 16:46:21.142770 | controller | "start": "2023-09-13 16:46:19.071898"
2023-09-13 16:46:21.142879 | controller | }
2023-09-13 16:46:21.142972 | controller | ERROR: Ignoring Errors

Example failures zed/stable2023.1:-
- https://zuul.opendev.org/t/openstack/build/591dae67122444daa35195f7458ffafe
- https://zuul.opendev.org/t/openstack/build/5838bf0704b247dc8f1eb12367b1d33e
- https://zuul.opendev.org/t/openstack/build/8d2e22ff171944b0b549c12e1aaac476

Wallaby/Xena/Yoga builds started failing with:-
++ functions:write_devstack_version:852 : git log '--format=%H %s %ci' -1
+ ./stack.sh:main:230 : SUPPORTED_DISTROS='bullseye|focal|f35|opensuse-15.2|opensuse-tumbleweed|rhel8|rhel9|openEuler-20.03'
+ ./stack.sh:main:232 : [[ ! jammy =~ bullseye|focal|f35|opensuse-15.2|opensuse-tumbleweed|rhel8|rhel9|openEuler-20.03 ]]
+ ./stack.sh:main:233 : echo 'WARNING: this script has not been tested on jammy'

Example:-
- https://zuul.opendev.org/t/openstack/build/0bd0421e30804b7aa9b6ea032d271be7
- https://zuul.opendev.org/t/openstack/build/8e06dfc0ccd940f3ab71edc0ec93466c
- https://zuul.opendev.org/t/openstack/build/899634e90ee94e0294985747075fb26c

Even before these jobs were broken but there tests used to fail not the test setup, that can be handled once the current issues are cleared.

[1] https://zuul.opendev.org/t/openstack/builds?job_name=devstack-tobiko-neutron&branch=stable%2F2023.1
[2] https://review.opendev.org/c/x/devstack-plugin-tobiko/+/893662?usp=search

[Eduardo Olivares (eolivare)]

Sharing some findings, but I'm still far from the solution.

Regarding the first issue with the 2023.1 branch
1) 2023.1 tobiko job using ubuntu-focal nodes
PASS - tox3.28 - python3.8
https://1d99560c221207a33e89-14a854fbbf5a230846aba3c6a65bcd80.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/stable/2023.1/devstack-tobiko-neutron/d1de823/job-output.txt
2023-09-13 02:32:25.704782 | TASK [tobiko-ensure-tox : show Tox facts]
2023-09-13 02:32:25.765288 | {'tox_executable': '/usr/bin/python3 -m tox', 'tox_version': '3.28.0 imported from /home/zuul/.local/lib/python3.8/site-packages/tox/__init__.py'}
...
2023-09-13 02:33:41.677560 | controller | neutron_sanity run-test: commands[0] | /home/zuul/src/opendev.org/x/tobiko/.tox/py3/bin/python /home/zuul/src/opendev.org/x/tobiko/tools/run_tests.py /home/zuul/src/opendev.org/x/tobiko/tobiko/tests/sanity/neutron

2) 2023.1 tobiko job using ubuntu-jammy nodes
FAIL - tox3.28 - python3.10
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_583/894857/1/experimental/devstack-tobiko-neutron/5838bf0/job-output.txt
2023-09-13 16:46:17.132141 | TASK [tobiko-ensure-tox : show Tox facts]
2023-09-13 16:46:17.185958 | {'tox_executable': '/usr/bin/python3 -m tox', 'tox_version': '3.28.0 imported from /usr/local/lib/python3.10/dist-packages/tox/__init__.py'}
...
2023-09-13 16:46:20.555843 | controller | ERROR: neutron_sanity: could not install deps [-c/home/zuul/src/opendev.org/x/tobiko/upper-constraints.txt, -r/home/zuul/src/opendev.org/x/tobiko/requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/test-requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/extra-requirements.txt]; v = InvocationError('could not find executable python', None)

3) master tobiko job using ubuntu-jammy nodes
PASS - tox3.28 - python3.10
https://83fd172b39bb4bd8571a-4a42abeb7e7779ef25c6242290fc9b63.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/devstack-tobiko-neutron/f7ae7d8/job-output.txt
2023-09-15 02:34:39.846829 | TASK [tobiko-ensure-tox : show Tox facts]
2023-09-15 02:34:39.954407 | {'tox_executable': '/usr/bin/python3 -m tox', 'tox_version': '3.28.0 imported from /home/zuul/.local/lib/python3.10/site-packages/tox/__init__.py'}
...
2023-09-15 02:35:29.405645 | controller | neutron_sanity run-test: commands[0] | /home/zuul/src/opendev.org/x/tobiko/.tox/py3/bin/python /home/zuul/src/opendev.org/x/tobiko/tools/run_tests.py /home/zuul/src/opendev.org/x/tobiko/tobiko/tests/sanity/neutron

Comparing 2 and 3, it looks to me like python packages (at least, tox) are installed in different paths and that may be related to the issue:
'3.28.0 imported from /usr/local/lib/python3.10/dist-packages/tox/__init__.py'
versus
'3.28.0 imported from /home/zuul/.local/lib/python3.10/site-packages/tox/__init__.py'

The error raised [1] happens when tox tries to find the path to python:
https://github.com/tox-dev/tox/blob/3.28.0/src/tox/venv.py#L676

Regarding the second issue [2], ubuntu jammy is not a supported version:
https://github.com/openstack/devstack/blob/stable/wallaby/stack.sh#L230
SUPPORTED_DISTROS="bionic|focal|f31|f32|opensuse-15.2|opensuse-tumbleweed|rhel8|rhel...

[Eduardo Olivares (eolivare)]

Yatin proposed:
https://review.opendev.org/c/x/devstack-plugin-tobiko/+/895582

Apparently, this is not a neutron bug, but a problem in tobiko's jobs.

[Brian Haley (brian-haley)]

As this was not a neutron bug and the tobiko patch has merged will close this bug.