[dalmatian] skip level grenade jobs fails at keystone-manage db_sync

[yatin]

Fails like:-
2024-05-20 03:18:02.465 | + /opt/stack/new/grenade/projects/10_keystone/upgrade.sh:main:61 : /usr/local/bin/keystone-manage db_sync
2024-05-20 03:18:05.597 | INFO dbcounter [-] Registered counter for database keystone
2024-05-20 03:18:05.613 | CRITICAL keystone [-] Unhandled error: NotImplementedError
2024-05-20 03:18:05.613 | ERROR keystone Traceback (most recent call last):
2024-05-20 03:18:05.613 | ERROR keystone  File "/usr/local/bin/keystone-manage", line 10, in <module>
2024-05-20 03:18:05.614 | ERROR keystone  sys.exit(main())
2024-05-20 03:18:05.614 | ERROR keystone  File "/opt/stack/new/keystone/keystone/cmd/manage.py", line 40, in main
2024-05-20 03:18:05.614 | ERROR keystone  cli.main(argv=sys.argv, developer_config_file=developer_config)
2024-05-20 03:18:05.614 | ERROR keystone  File "/opt/stack/new/keystone/keystone/cmd/cli.py", line 1463, in main
2024-05-20 03:18:05.614 | ERROR keystone  CONF.command.cmd_class.main()
2024-05-20 03:18:05.614 | ERROR keystone  File "/opt/stack/new/keystone/keystone/cmd/cli.py", line 413, in main
2024-05-20 03:18:05.614 | ERROR keystone  upgrades.offline_sync_database_to_version(
2024-05-20 03:18:05.614 | ERROR keystone  File "/opt/stack/new/keystone/keystone/common/sql/upgrades.py", line 304, in offline_sync_database_to_version
2024-05-20 03:18:05.614 | ERROR keystone  _db_sync(engine=engine)
2024-05-20 03:18:05.614 | ERROR keystone  File "/opt/stack/new/keystone/keystone/common/sql/upgrades.py", line 195, in _db_sync
2024-05-20 03:18:05.614 | ERROR keystone  with sql.session_for_write() as session:
2024-05-20 03:18:05.614 | ERROR keystone  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
2024-05-20 03:18:05.614 | ERROR keystone  return next(self.gen)
2024-05-20 03:18:05.614 | ERROR keystone  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope
2024-05-20 03:18:05.614 | ERROR keystone  with current._produce_block(
2024-05-20 03:18:05.614 | ERROR keystone  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
2024-05-20 03:18:05.614 | ERROR keystone  return next(self.gen)
2024-05-20 03:18:05.614 | ERROR keystone  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 641, in _session
2024-05-20 03:18:05.614 | ERROR keystone  self.session = self.factory._create_session(
2024-05-20 03:18:05.614 | ERROR keystone  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 404, in _create_session
2024-05-20 03:18:05.614 | ERROR keystone  self._start()
2024-05-20 03:18:05.614 | ERROR keystone  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 493, in _start
2024-05-20 03:18:05.614 | ERROR keystone  self._setup_for_connection(
2024-05-20 03:18:05.614 | ERROR keystone  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 519, in _setup_for_connection
2024-05-20 03:18:05.614 | ERROR keystone  engine = engines.create_engine(
2024-05-20 03:18:05.614 | ERROR keystone  File "/usr/local/lib/python3.10/dist-packages/debtcollector/renames.py", line 41, in decorator
2024-05-20 03:18:05.614 | ERROR keystone  return wrapped(*args, **kwargs)
2024-05-20 03:18:05.614 | ERROR keystone  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py", line 194, in create_engine
2024-05-20 03:18:05.614 | ERROR keystone  engine = sqlalchemy.create_engine(url, **engine_args)
2024-05-20 03:18:05.615 | ERROR keystone  File "<string>", line 2, in create_engine
2024-05-20 03:18:05.615 | ERROR keystone  File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/util/deprecations.py", line 281, in warned
2024-05-20 03:18:05.615 | ERROR keystone  return fn(*args, **kwargs) # type: ignore[no-any-return]
2024-05-20 03:18:05.615 | ERROR keystone  File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/create.py", line 548, in create_engine
2024-05-20 03:18:05.615 | ERROR keystone  u, plugins, kwargs = u._instantiate_plugins(kwargs)
2024-05-20 03:18:05.615 | ERROR keystone  File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/url.py", line 739, in _instantiate_plugins
2024-05-20 03:18:05.615 | ERROR keystone  new_u = plugin.update_url(u)
2024-05-20 03:18:05.615 | ERROR keystone  File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/interfaces.py", line 2870, in update_url
2024-05-20 03:18:05.615 | ERROR keystone  raise NotImplementedError()
2024-05-20 03:18:05.615 | ERROR keystone NotImplementedError
2024-05-20 03:18:05.615 | ERROR keystone 
2024-05-20 03:18:06.079 | + /opt/stack/new/grenade/projects/10_keystone/upgrade.sh:main:61 : die 61 'DB sync error'
2024-05-20 03:18:06.082 | + functions-common:die:285 : local exitcode=1
2024-05-20 03:18:06.085 | [Call Trace]
2024-05-20 03:18:06.085 | /opt/stack/new/grenade/projects/10_keystone/upgrade.sh:61:die
2024-05-20 03:18:06.089 | [ERROR] /opt/stack/new/grenade/projects/10_keystone/upgrade.sh:61 DB sync error
2024-05-20 03:18:07.092 | 1 die /opt/stack/new/devstack/functions-common
2024-05-20 03:18:07.092 | 61 main /opt/stack/new/grenade/projects/10_keystone/upgrade.sh
2024-05-20 03:18:07.092 | Exit code: 1
2024-05-20 03:18:09.512 | + inc/upgrade:upgrade_service:65 : die 65 'Failure in /opt/stack/new/grenade/projects/10_keystone/upgrade.sh'
2024-05-20 03:18:09.514 | + /opt/stack/new/devstack/functions-common:die:285 : local exitcode=1
2024-05-20 03:18:09.517 | [Call Trace]
2024-05-20 03:18:09.517 | ./grenade.sh:347:upgrade_service
2024-05-20 03:18:09.517 | /opt/stack/new/grenade/inc/upgrade:65:die
2024-05-20 03:18:09.521 | [ERROR] /opt/stack/new/grenade/inc/upgrade:65 Failure in /opt/stack/new/grenade/projects/10_keystone/upgrade.sh
2024-05-20 03:18:10.523 | 1 die /opt/stack/new/devstack/functions-common
2024-05-20 03:18:10.523 | 65 upgrade_service /opt/stack/new/grenade/inc/upgrade
2024-05-20 03:18:10.523 | 347 main ./grenade.sh
2024-05-20 03:18:10.523 | Exit code: 1

Example failure:- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ae/periodic/opendev.org/openstack/neutron/master/neutron-ovn-grenade-multinode-skip-level/3aee59a/controller/logs/grenade.sh_log.txt
Example builds:- https://zuul.openstack.org/builds?job_name=neutron-ovn-grenade-multinode-skip-level&job_name=neutron-ovs-grenade-multinode-skip-level&project=openstack%2Fneutron&branch=master&skip=0

Timing wise could be related to sqlalchemy update to 2.0
Also jobs upgrading from 2023.1 , this also needs to be fixed to upgrade from 2023.2

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/920075

[yatin (yatinkarel)]

Jobs green with switch upgrade from to 2023.2:-
neutron-ovs-grenade-multinode-skip-level https://zuul.opendev.org/t/openstack/build/d9370992e39e4fc4a2f848c84676379f : SUCCESS in 43m 52s
neutron-ovn-grenade-multinode-skip-level https://zuul.opendev.org/t/openstack/build/7c2cc5cba49742e394293d90eef8b9c5 : SUCCESS in 54m 38s

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/920075
Committed: https://opendev.org/openstack/neutron/commit/2beb20f92b4b306af9e640e977a5166f376bf098
Submitter: "Zuul (22348)"
Branch: master

commit 2beb20f92b4b306af9e640e977a5166f376bf098
Author: yatinkarel <email address hidden>
Date: Tue May 21 11:49:37 2024 +0530

    Update grenade_from_branch for skip level jobs

    Skip level jobs were wrongly upgrading from stable/2023.1.
    As master is currently 2024.2, upgrade from branch should
    be 2023.2, this patch fixes it.

    Related-Bug: #2066221
    Change-Id: I568ebb93bc009951ca113a09ee750982f580b40d