Devstack stack fails running Py36 on Ubuntu20

[michael-mcaleer]

Stacking DevStack on Master targeting py36 in local.conf instead of the distro version py38 (ubuntu 20.04.1) results in failures from Keystone:

INFO keystone.cmd.bootstrap [None req-4d0eef02-0e81-4148-b71c-97155e48e821 None None] Created region RegionOne
INFO keystone.cmd.bootstrap [None req-4d0eef02-0e81-4148-b71c-97155e48e821 None None] Created public endpoint http://10.10.10.10/identity
INFO keystone.cmd.bootstrap [None req-4d0eef02-0e81-4148-b71c-97155e48e821 None None] Created admin endpoint http://10.10.10.10/identity
+./stack.sh:main:1082 create_keystone_accounts
+lib/keystone:create_keystone_accounts:314 local admin_project
++lib/keystone:create_keystone_accounts:315 oscwrap project show admin -f value -c id
Failed to discover available identity versions when contacting http://10.10.10.10/identity. Attempting to parse version from URL.
Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Internal Server Error (HTTP 500)
++functions-common:oscwrap:2350 return 1
+lib/keystone:create_keystone_accounts:315 admin_project=
+lib/keystone:create_keystone_accounts:1 exit_trap
+./stack.sh:exit_trap:489 local r=1
++./stack.sh:exit_trap:490 jobs -p
+./stack.sh:exit_trap:490 jobs=
+./stack.sh:exit_trap:493 [[ -n '' ]]
+./stack.sh:exit_trap:499 '[' -f /tmp/tmp.XkcNcmnhKL ']'
+./stack.sh:exit_trap:500 rm /tmp/tmp.XkcNcmnhKL
+./stack.sh:exit_trap:504 kill_spinner
+./stack.sh:kill_spinner:399 '[' '!' -z '' ']'
+./stack.sh:exit_trap:506 [[ 1 -ne 0 ]]
+./stack.sh:exit_trap:507 echo 'Error on exit'
Error on exit
+./stack.sh:exit_trap:509 type -p generate-subunit
+./stack.sh:exit_trap:510 generate-subunit 1598956624 2545 fail
+./stack.sh:exit_trap:512 [[ -z /opt/stack/logs ]]
+./stack.sh:exit_trap:515 /usr/local/bin/python3.6 /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs
+./stack.sh:exit_trap:524 exit 1

Running the oscwrap command in isolation:

stack@test:~/devstack$ oscwrap project show admin -f value -c id
Failed to discover available identity versions when contacting http://10.10.10.10/identity. Attempting to parse version from URL.
Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Internal Server Error (HTTP 500)
-bash: $OSCWRAP_TIMER_FILE: ambiguous redirect

If we have a look at the Keystone logs we can see the following log message:

Sep 01 12:35:57 test <email address hidden>[80544]: Python version: 3.8.2 (default, Jul 16 2020, 14:00:26) [GCC 9.3.0]

Which results in a ModuleNotFoundError exception later because we have specified keystone to be installed in Python36:

Sep 01 12:35:57 test <email address hidden>[80546]: Traceback (most recent call last):
Sep 01 12:35:57 test <email address hidden>[80546]: File "/usr/local/bin/keystone-wsgi-public", line 6, in <module>
Sep 01 12:35:57 test <email address hidden>[80546]: from keystone.server.wsgi import initialize_public_application
Sep 01 12:35:57 test <email address hidden>[80546]: ModuleNotFoundError: No module named 'keystone'
Sep 01 12:35:57 test <email address hidden>[80546]: unable to load app 0 (mountpoint='') (callable not found or import error)

Checking plugin INI configuration from /etc/keystone/keystone-uwsgi-public.ini we can see reference to Python 3:

    plugins = http,python3

And if we go looking at the uwsgi plugins python3 is just a symlink to python3.8:

stack@test:~/.local/bin$ sudo ls -ls /usr/lib/uwsgi/plugins/ | grep python
196 -rw-r--r-- 1 root root 199896 Apr 11 12:15 python38_plugin.so
  0 lrwxrwxrwx 1 root root 38 Sep 1 11:46 python3_plugin.so -> /etc/alternatives/uwsgi-plugin-python3

stack@test:~/.local/bin$ sudo ls -ls /etc/alternatives/uwsgi-plugin-python3 | grep python
0 lrwxrwxrwx 1 root root 41 Sep 1 11:46 /etc/alternatives/uwsgi-plugin-python3 -> /usr/lib/uwsgi/plugins/python38_plugin.so

For additional versions of Python to work on Ubuntu 20 there needs to be allowances for versions other than Python 3.8. It is possible to do this with uwsgi but not with the build included with DevStack:

PYTHON=python3.4 ./uwsgi --build-plugin "plugins/python python34"
PYTHON=python2.7 ./uwsgi --build-plugin "plugins/python python27"
PYTHON=python2.6 ./uwsgi --build-plugin "plugins/python python26"

https://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html#bonus-multiple-python-versions-for-the-same-uwsgi-binary

[Dr. Jens Harbott (j-harbott)]

Why do you want to install an outdated python version? And how do you do it? As far as I can see, Focal at best has python3.7. The upstream tests for py36 are being run on Bionic, would that be an option for you?

[michael-mcaleer (mmcaleer)]

It is listed as a supported verison here for Victoria: https://wiki.openstack.org/wiki/Python3#Current_Status

CI is run against it for our submissions: https://review.opendev.org/#/c/741447/

Consistent OS versions across our CI systems reduces the maintenance effort. It also reduces maintenance effort on development systems and testing systems.

Python 3.6 is built from source and installed using 'make altinstall' so it does not overwrite the system bundled 3.8 version.

For previous releases on Ubuntu 18 there was no problem running Python 3.7 when the OS bundled version was Python 3.6.

[Dr. Jens Harbott (j-harbott)]

I don't think that this is a scenario that we can commit to support. If you can create a patch that fixes your scenario and doesn't break anything else, we can consider merging it, but I doubt anyone else would actively work on fixing this.

[michael-mcaleer (mmcaleer)]

That's unfortunate, its a pretty glaring bug in a supported Python version on the latest Ubuntu release. I can appreciate bugs going unfixed in older releases but we are right up to date here on the OS and OpenStack distro.

Without domain specific knowledge it would take me forever to build a fix for this, but all the background work is done so unless im interpreting this issue incorrectly it shouldn't be a big ask of someone that knows the code.