stack.sh fails on Rocky 9 and Centos 9 because of global virtualenv

[Tore Anderson]

Current Devstack Git master (08b434e5) appears to be broken on Rocky 9.

Steps taken to reproduce:

1) Creating a new instance from https://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2 (downloaded today) in a production OpenStack environment

2) As the default 'rocky' user, issuing the following commands in the newly created instace:

[rocky@devstack ~]$ sudo dnf -y upgrade; sudo dnf -y install git-core; sudo reboot
[rocky@devstack ~]$ git clone https://opendev.org/openstack/devstack
[rocky@devstack ~]$ cd devstack/
[rocky@devstack devstack]$ cp samples/local.conf .
[rocky@devstack devstack]$ ./stack.sh |& tee ../stack.sh.log

This produces a ton of output, which I will attach as a separate file.

I assume the interesting parts is found in the last few lines:

INFO keystone.cmd.bootstrap [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] Created project admin
WARNING keystone.common.password_hashing [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] Truncating password to algorithm specific maximum length 72 characters.: keystone.exception.UserNotFound: Could not find user: admin.
DEBUG passlib.handlers.bcrypt [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] detected 'bcrypt' backend, version '4.0.1' {{(pid=53715) _load_backend_mixin /opt/stack/data/venv/lib64/python3.9/site-packages/passlib/handlers/bcrypt.py:625}}
DEBUG passlib.handlers.bcrypt [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=53715) _finalize_backend_mixin /opt/stack/data/venv/lib64/python3.9/site-packages/passlib/handlers/bcrypt.py:406}}
INFO keystone.cmd.bootstrap [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] Created user admin
INFO keystone.cmd.bootstrap [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] Created role reader
INFO keystone.cmd.bootstrap [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] Created role member
INFO keystone.cmd.bootstrap [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] Created implied role where d556020f89b74ceb9686990a053dde96 implies 412d79cda6b4479aa7b2b47eb075145e
INFO keystone.cmd.bootstrap [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] Created role admin
INFO keystone.cmd.bootstrap [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] Created implied role where 59e9e8efa61d4dada0f7d4a90bb44e97 implies d556020f89b74ceb9686990a053dde96
INFO keystone.cmd.bootstrap [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] Created role service
INFO keystone.cmd.bootstrap [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] Granted role admin on project admin to user admin.
INFO keystone.cmd.bootstrap [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] Granted role admin on the system to user admin.
WARNING py.warnings [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] /opt/stack/data/venv/lib64/python3.9/site-packages/pycadf/identifier.py:71: UserWarning: Invalid uuid: RegionOne. To ensure interoperability, identifiers should be a valid uuid.
  warnings.warn(('Invalid uuid: %s. To ensure interoperability, '

INFO keystone.cmd.bootstrap [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] Created region RegionOne
INFO keystone.cmd.bootstrap [None req-86789409-bda8-44a2-b873-ba4c2be8d784 None None] Created public endpoint http://192.168.0.108/identity
+ lib/keystone:bootstrap_keystone:615 : '[' False == True ']'
+ ./stack.sh:main:1144 : create_keystone_accounts
+ lib/keystone:create_keystone_accounts:314 : local admin_project
++ lib/keystone:create_keystone_accounts:315 : oscwrap project show admin -f value -c id
Failed to discover available identity versions when contacting http://192.168.0.108/identity. Attempting to parse version from URL.
Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Internal Server Error (HTTP 500)
++ functions-common:oscwrap:2434 : return 1
+ lib/keystone:create_keystone_accounts:315 : admin_project=
+ lib/keystone:create_keystone_accounts:1 : exit_trap
+ ./stack.sh:exit_trap:551 : local r=1
++ ./stack.sh:exit_trap:552 : jobs -p
+ ./stack.sh:exit_trap:552 : jobs=49870
+ ./stack.sh:exit_trap:555 : [[ -n 49870 ]]
+ ./stack.sh:exit_trap:555 : [[ -n /opt/stack/logs/stack.sh.log.2023-08-17-095733 ]]
+ ./stack.sh:exit_trap:555 : [[ True == \T\r\u\e ]]
+ ./stack.sh:exit_trap:556 : echo 'exit_trap: cleaning up child processes'
exit_trap: cleaning up child processes
+ ./stack.sh:exit_trap:557 : kill 49870
+ ./stack.sh:exit_trap:561 : '[' -f /tmp/tmp.tBByjFGzx5 ']'
+ ./stack.sh:exit_trap:562 : rm /tmp/tmp.tBByjFGzx5
+ ./stack.sh:exit_trap:566 : kill_spinner
+ ./stack.sh:kill_spinner:461 : '[' '!' -z '' ']'
+ ./stack.sh:exit_trap:568 : [[ 1 -ne 0 ]]
+ ./stack.sh:exit_trap:569 : echo 'Error on exit'
Error on exit
+ ./stack.sh:exit_trap:571 : type -p generate-subunit
+ ./stack.sh:exit_trap:572 : generate-subunit 1692266231 731 fail
+ ./stack.sh:exit_trap:574 : [[ -z /opt/stack/logs ]]
+ ./stack.sh:exit_trap:577 : /opt/stack/data/venv/bin/python3 /home/rocky/devstack/tools/worlddump.py -d /opt/stack/logs
ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
+ ./stack.sh:exit_trap:586 : exit 1

I am more than happy to provide direct login to the VM in question to any Devstack developer willing to take a look, just send me an SSH pubkey and I will add it to the 'rocky' user's authorized_keys file.

[Tore Anderson (toreanderson)]

The stable/2023.1 branch works fine. Bisecting...

[Dr. Jens Harbott (j-harbott)]

You'll need to add "GLOBAL_VENV=false" into your local.conf on Rocky currently

[Tore Anderson (toreanderson)]

Hi Jens,

You're right, with GLOBAL_VENV=false master works fine.

For what it is worth (and I guess you know this already), git-bisect tells me a40f9cb91fbedddec89f0ffd6c7dd4b3828a232e is the first bad commit.

[Martin Kopec (mkopec)]

The failures were caused by changing default behavior of Devstack and installing everything in global virtualenv:
https://review.opendev.org/c/openstack/devstack/+/558930

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/devstack/+/892395

[Michael Johnson (johnsom)]

FYI, I think this section is building an invalid path on CentOS 9 Stream:
https://github.com/openstack/devstack/blob/master/lib/keystone#L42

In our jobs, if you look at the screen-keystone.txt:

Aug 23 15:12:23.972481 np0035030837 <email address hidden>[46510]: failed to open python file /opt/stack/data/venv/bin/keystone-wsgi-public
Aug 23 15:12:23.972481 np0035030837 <email address hidden>[46510]: unable to load app 0 (mountpoint='') (callable not found or import error)
Aug 23 15:12:23.972481 np0035030837 <email address hidden>[46510]: *** no app loaded. going in full dynamic mode ***

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/892395
Committed: https://opendev.org/openstack/devstack/commit/7cd3a8eebe1830f94c02bb6ec010c0365f6ab6f1
Submitter: "Zuul (22348)"
Branch: master

commit 7cd3a8eebe1830f94c02bb6ec010c0365f6ab6f1
Author: Martin Kopec <email address hidden>
Date: Tue Aug 22 20:40:20 2023 +0200

    Set GLOBAL_VENV to false for centos and rocky

    As a temporary workaround, let's set the GLOBAL_VENV to false
    specifically for centos 9 stream and rocky distros where we
    encountered issues after changing the default value
    of GLOBAL_VENV to True in Devstack:
    https://review.opendev.org/c/openstack/devstack/+/558930

    Related-Bug: #2031639
    Change-Id: I708b5a81c32b0bd650dcd63a51e16346863a6fc0

[Martin Kopec (mkopec)]

The failures are very similar to this LP where the root cause was that a python version wasn't supported by one or more services:

https://bugs.launchpad.net/devstack/+bug/1893776

[Dr. Jens Harbott (j-harbott)]

@jokke please keep this bug specific to Rocky/centos. if you see other issues, please create a new bug report

[Martin Kopec (mkopec)]

additional info:
https://<email address hidden>/thread/HVFN5RBSHRTM3B2UUKPAWKH6H6AT6CYR/