There is (probably) some issue with configuration of the functional tests job neutron-functional-with-sqlalchemy-master which runs in periodic queue. It is failing every day with same error:
2023-04-11 03:57:17.753314 | controller | neutron.tests.functional.db.test_migrations.TestWalkMigrationsPostgreSQL.test_walk_versions
2023-04-11 03:57:17.753325 | controller | -------------------------------------------------------------------------------------------
2023-04-11 03:57:17.753336 | controller |
2023-04-11 03:57:17.753346 | controller | Captured traceback:
2023-04-11 03:57:17.753357 | controller | ~~~~~~~~~~~~~~~~~~~
2023-04-11 03:57:17.753368 | controller | Traceback (most recent call last):
2023-04-11 03:57:17.753379 | controller |
2023-04-11 03:57:17.753390 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
2023-04-11 03:57:17.753401 | controller | self._dbapi_connection = engine.raw_connection()
2023-04-11 03:57:17.753412 | controller |
2023-04-11 03:57:17.753429 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3275, in raw_connection
2023-04-11 03:57:17.753441 | controller | return self.pool.connect()
2023-04-11 03:57:17.753452 | controller |
2023-04-11 03:57:17.753463 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 455, in connect
2023-04-11 03:57:17.753474 | controller | return _ConnectionFairy._checkout(self)
2023-04-11 03:57:17.753492 | controller |
2023-04-11 03:57:17.753504 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1271, in _checkout
2023-04-11 03:57:17.753514 | controller | fairy = _ConnectionRecord.checkout(pool)
2023-04-11 03:57:17.753525 | controller |
2023-04-11 03:57:17.753536 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 719, in checkout
2023-04-11 03:57:17.753548 | controller | rec = pool._do_get()
2023-04-11 03:57:17.753558 | controller |
2023-04-11 03:57:17.753570 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
2023-04-11 03:57:17.753581 | controller | with util.safe_reraise():
2023-04-11 03:57:17.753591 | controller |
2023-04-11 03:57:17.753602 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
2023-04-11 03:57:17.753613 | controller | raise exc_value.with_traceback(exc_tb)
2023-04-11 03:57:17.753624 | controller |
2023-04-11 03:57:17.753635 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 166, in _do_get
2023-04-11 03:57:17.753645 | controller | return self._create_connection()
2023-04-11 03:57:17.753656 | controller |
2023-04-11 03:57:17.753667 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 396, in _create_connection
2023-04-11 03:57:17.753678 | controller | return _ConnectionRecord(self)
2023-04-11 03:57:17.753689 | controller |
2023-04-11 03:57:17.753700 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 681, in __init__
2023-04-11 03:57:17.753711 | controller | self.__connect()
2023-04-11 03:57:17.753722 | controller |
2023-04-11 03:57:17.753733 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 905, in __connect
2023-04-11 03:57:17.753743 | controller | with util.safe_reraise():
2023-04-11 03:57:17.753754 | controller |
2023-04-11 03:57:17.753769 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
2023-04-11 03:57:17.753781 | controller | raise exc_value.with_traceback(exc_tb)
2023-04-11 03:57:17.753792 | controller |
2023-04-11 03:57:17.753803 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 901, in __connect
2023-04-11 03:57:17.753814 | controller | self.dbapi_connection = connection = pool._invoke_creator(self)
2023-04-11 03:57:17.753825 | controller |
2023-04-11 03:57:17.753836 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 636, in connect
2023-04-11 03:57:17.753847 | controller | return dialect.connect(*cargs, **cparams)
2023-04-11 03:57:17.753858 | controller |
2023-04-11 03:57:17.753868 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 579, in connect
2023-04-11 03:57:17.753879 | controller | return self.loaded_dbapi.connect(*cargs, **cparams)
2023-04-11 03:57:17.753890 | controller |
2023-04-11 03:57:17.753901 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
2023-04-11 03:57:17.753912 | controller | conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2023-04-11 03:57:17.753923 | controller |
2023-04-11 03:57:17.753934 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/support/psycopg2_patcher.py", line 46, in eventlet_wait_callback
2023-04-11 03:57:17.753945 | controller | state = conn.poll()
2023-04-11 03:57:17.753962 | controller |
2023-04-11 03:57:17.753974 | controller | psycopg2.OperationalError: FATAL: password authentication failed for user "openstack_citest"
2023-04-11 03:57:17.753985 | controller | connection to server at "localhost" (::1), port 5432 failed: FATAL: password authentication failed for user "openstack_citest"
2023-04-11 03:57:17.753997 | controller |
2023-04-11 03:57:17.754007 | controller |
2023-04-11 03:57:17.754018 | controller |
2023-04-11 03:57:17.754029 | controller | The above exception was the direct cause of the following exception:
2023-04-11 03:57:17.754040 | controller |
2023-04-11 03:57:17.754051 | controller |
2023-04-11 03:57:17.754062 | controller | Traceback (most recent call last):
2023-04-11 03:57:17.754073 | controller |
2023-04-11 03:57:17.754084 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
2023-04-11 03:57:17.754095 | controller | return f(self, *args, **kwargs)
2023-04-11 03:57:17.754105 | controller |
2023-04-11 03:57:17.754116 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/db/test_migrations.py", line 576, in test_walk_versions
2023-04-11 03:57:17.754128 | controller | migration.do_alembic_command(config, 'upgrade', dest)
2023-04-11 03:57:17.754138 | controller |
2023-04-11 03:57:17.754149 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/migration/cli.py", line 90, in do_alembic_command
2023-04-11 03:57:17.754160 | controller | getattr(alembic_command, cmd)(config, *args, **kwargs)
2023-04-11 03:57:17.754171 | controller |
2023-04-11 03:57:17.754188 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/alembic/command.py", line 378, in upgrade
2023-04-11 03:57:17.754200 | controller | script.run_env()
2023-04-11 03:57:17.754211 | controller |
2023-04-11 03:57:17.754222 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
2023-04-11 03:57:17.754233 | controller | util.load_python_file(self.dir, "env.py")
2023-04-11 03:57:17.754244 | controller |
2023-04-11 03:57:17.754255 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
2023-04-11 03:57:17.754266 | controller | module = load_module_py(module_id, path)
2023-04-11 03:57:17.754277 | controller |
2023-04-11 03:57:17.754288 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
2023-04-11 03:57:17.754299 | controller | spec.loader.exec_module(module) # type: ignore
2023-04-11 03:57:17.754310 | controller |
2023-04-11 03:57:17.754321 | controller | File "<frozen importlib._bootstrap_external>", line 883, in exec_module
2023-04-11 03:57:17.754332 | controller |
2023-04-11 03:57:17.754343 | controller | File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2023-04-11 03:57:17.754354 | controller |
2023-04-11 03:57:17.754365 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/migration/alembic_migrations/env.py", line 120, in <module>
2023-04-11 03:57:17.754376 | controller | run_migrations_online()
2023-04-11 03:57:17.754387 | controller |
2023-04-11 03:57:17.754398 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/migration/alembic_migrations/env.py", line 106, in run_migrations_online
2023-04-11 03:57:17.754409 | controller | with DBConnection(neutron_config.database.connection, connection) as conn:
2023-04-11 03:57:17.754420 | controller |
2023-04-11 03:57:17.754431 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/migration/connection.py", line 32, in __enter__
2023-04-11 03:57:17.754442 | controller | self.engine = session.create_engine(self.connection_url)
2023-04-11 03:57:17.754453 | controller |
2023-04-11 03:57:17.754464 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/debtcollector/removals.py", line 260, in wrapper
2023-04-11 03:57:17.754475 | controller | return f(*args, **kwargs)
2023-04-11 03:57:17.754485 | controller |
2023-04-11 03:57:17.754496 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/debtcollector/renames.py", line 41, in decorator
2023-04-11 03:57:17.754517 | controller | return wrapped(*args, **kwargs)
2023-04-11 03:57:17.754529 | controller |
2023-04-11 03:57:17.754540 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 257, in create_engine
2023-04-11 03:57:17.754551 | controller | test_conn = _test_connection(engine, max_retries, retry_interval)
2023-04-11 03:57:17.754562 | controller |
2023-04-11 03:57:17.754573 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 453, in _test_connection
2023-04-11 03:57:17.754584 | controller | return engine.connect()
2023-04-11 03:57:17.754595 | controller |
2023-04-11 03:57:17.754606 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3251, in connect
2023-04-11 03:57:17.754617 | controller | return self._connection_cls(self)
2023-04-11 03:57:17.754628 | controller |
2023-04-11 03:57:17.754639 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 147, in __init__
2023-04-11 03:57:17.754650 | controller | Connection._handle_dbapi_exception_noconnection(
2023-04-11 03:57:17.754660 | controller |
2023-04-11 03:57:17.754671 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2410, in _handle_dbapi_exception_noconnection
2023-04-11 03:57:17.754682 | controller | raise newraise.with_traceback(exc_info[2]) from e
2023-04-11 03:57:17.754693 | controller |
2023-04-11 03:57:17.754704 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
2023-04-11 03:57:17.754715 | controller | self._dbapi_connection = engine.raw_connection()
2023-04-11 03:57:17.754726 | controller |
2023-04-11 03:57:17.754737 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3275, in raw_connection
2023-04-11 03:57:17.754748 | controller | return self.pool.connect()
2023-04-11 03:57:17.754764 | controller |
2023-04-11 03:57:17.754776 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 455, in connect
2023-04-11 03:57:17.754787 | controller | return _ConnectionFairy._checkout(self)
2023-04-11 03:57:17.754798 | controller |
2023-04-11 03:57:17.754809 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1271, in _checkout
2023-04-11 03:57:17.754820 | controller | fairy = _ConnectionRecord.checkout(pool)
2023-04-11 03:57:17.754831 | controller |
2023-04-11 03:57:17.754842 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 719, in checkout
2023-04-11 03:57:17.754853 | controller | rec = pool._do_get()
2023-04-11 03:57:17.754864 | controller |
2023-04-11 03:57:17.754881 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
2023-04-11 03:57:17.754893 | controller | with util.safe_reraise():
2023-04-11 03:57:17.754905 | controller |
2023-04-11 03:57:17.754916 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
2023-04-11 03:57:17.754926 | controller | raise exc_value.with_traceback(exc_tb)
2023-04-11 03:57:17.754938 | controller |
2023-04-11 03:57:17.754949 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 166, in _do_get
2023-04-11 03:57:17.754960 | controller | return self._create_connection()
2023-04-11 03:57:17.754971 | controller |
2023-04-11 03:57:17.754982 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 396, in _create_connection
2023-04-11 03:57:17.754999 | controller | return _ConnectionRecord(self)
2023-04-11 03:57:17.755011 | controller |
2023-04-11 03:57:17.755022 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 681, in __init__
2023-04-11 03:57:17.755033 | controller | self.__connect()
2023-04-11 03:57:17.755044 | controller |
2023-04-11 03:57:17.755055 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 905, in __connect
2023-04-11 03:57:17.755066 | controller | with util.safe_reraise():
2023-04-11 03:57:17.755076 | controller |
2023-04-11 03:57:17.755091 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
2023-04-11 03:57:17.755103 | controller | raise exc_value.with_traceback(exc_tb)
2023-04-11 03:57:17.755114 | controller |
2023-04-11 03:57:17.755125 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 901, in __connect
2023-04-11 03:57:17.755136 | controller | self.dbapi_connection = connection = pool._invoke_creator(self)
2023-04-11 03:57:17.755147 | controller |
2023-04-11 03:57:17.755158 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 636, in connect
2023-04-11 03:57:17.755169 | controller | return dialect.connect(*cargs, **cparams)
2023-04-11 03:57:17.755180 | controller |
2023-04-11 03:57:17.755191 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 579, in connect
2023-04-11 03:57:17.755202 | controller | return self.loaded_dbapi.connect(*cargs, **cparams)
2023-04-11 03:57:17.755213 | controller |
2023-04-11 03:57:17.755224 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
2023-04-11 03:57:17.755235 | controller | conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2023-04-11 03:57:17.755246 | controller |
2023-04-11 03:57:17.755257 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/support/psycopg2_patcher.py", line 46, in eventlet_wait_callback
2023-04-11 03:57:17.755268 | controller | state = conn.poll()
2023-04-11 03:57:17.755279 | controller |
2023-04-11 03:57:17.755290 | controller | sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: password authentication failed for user "openstack_citest"
2023-04-11 03:57:17.755301 | controller | connection to server at "localhost" (::1), port 5432 failed: FATAL: password authentication failed for user "openstack_citest"
2023-04-11 03:57:17.755311 | controller |
2023-04-11 03:57:17.755322 | controller | (Background on this error at: https://sqlalche.me/e/20/e3q8)
Today's execution [1] (April 20th) didn't fail. But I don't see any new patch in sqlalchemy, Neutron, neutron-lib or oslo.db that could change the previous behaviour. I'm still investigating.
[1]https:/