Description
===========
During an upgrade from xena to yoga, I followed upgrade steps using openstack-ansible to upgrade nova. During this time i was met with the following log message:
alembic.runtime.migration [-] Running upgrade d67eeaabee36 -> b30f573d3377, Remove unused build_requests columns
And the python exception:
/openstack/venvs/nova-25.2.1.dev3/bin/nova-manage api_db sync
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
An error has occurred:
Traceback (most recent call last):
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
self.dialect.do_execute(
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/cursors.py", line 148, in execute
result = self._query(query)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/cursors.py", line 310, in _query
conn.query(q)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 548, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 775, in _read_query_result
result.read()
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 1156, in read
first_packet = self.connection._read_packet()
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 725, in _read_packet
packet.raise_for_error()
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/protocol.py", line 221, in raise_for_error
err.raise_mysql_exception(self._data)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.OperationalError: (1091, "Can't DROP COLUMN `vm_state`; check that it exists")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/cmd/manage.py", line 3402, in main
ret = fn(*fn_args, **fn_kwargs)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/cmd/manage.py", line 612, in sync
return migration.db_sync(version, database='api')
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/db/migration.py", line 154, in db_sync
_upgrade_alembic(engine, config, version)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/db/migration.py", line 109, in _upgrade_alembic
alembic_api.upgrade(config, version or 'head')
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/command.py", line 320, in upgrade
script.run_env()
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/script/base.py", line 563, in run_env
util.load_python_file(self.dir, "env.py")
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
module = load_module_py(module_id, path)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 848, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/db/api/migrations/env.py", line 117, in <module>
run_migrations_online()
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/db/api/migrations/env.py", line 111, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
self.get_context().run_migrations(**kw)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
step.migration_fn(**kw)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/db/api/migrations/versions/b30f573d3377_remove_unused_build_requests_columns.py", line 45, in upgrade
batch_op.drop_column('request_spec_id')
File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
next(self.gen)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/operations/base.py", line 374, in batch_alter_table
impl.flush()
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/operations/batch.py", line 108, in flush
fn(*arg, **kw)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/ddl/impl.py", line 329, in drop_column
self._exec(base.DropColumn(table_name, column, schema=schema))
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/ddl/impl.py", line 193, in _exec
return conn.execute(construct, multiparams)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1289, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
return connection._execute_ddl(
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1381, in _execute_ddl
ret = self._execute_context(
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
self._handle_dbapi_exception(
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2024, in _handle_dbapi_exception
util.raise_(newraise, with_traceback=exc_info[2], from_=e)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
self.dialect.do_execute(
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/cursors.py", line 148, in execute
result = self._query(query)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/cursors.py", line 310, in _query
conn.query(q)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 548, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 775, in _read_query_result
result.read()
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 1156, in read
first_packet = self.connection._read_packet()
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 725, in _read_packet
packet.raise_for_error()
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/protocol.py", line 221, in raise_for_error
err.raise_mysql_exception(self._data)
File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
raise errorclass(errno, errval)
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1091, "Can't DROP COLUMN `vm_state`; check that it exists")
[SQL: ALTER TABLE build_requests DROP COLUMN vm_state]
Steps to reproduce
==================
I was able to reproduce this bug by dropping the entire nova database, creating a nova database and running /openstack/venvs/nova-25.2.1.dev3/bin/nova-manage api_db sync
Expected result
===============
I expect nova-manage to test if build_requests is present before attempting to alter/drop it.
Actual result
=============
The python module reports the exception mentioned above.
Environment
===============
Nova: 25.0.2
Openstack: Yoga
Mariadb: 10.6.11
Hypervisor: libvert+kvm
Storage Type: ceph
Network: Neutron
so we are chating about htis on irc right now and we belive the issues is that while the
"nova" table was dropped that is the nova cell1 db
https:/ /github. com/openstack/ openstack- ansible- os_nova/ blob/stable/ yoga/defaults/ main.yml# L93 /github. com/openstack/ openstack- ansible- os_nova/ blob/abac462dc2 18b08656b56c57e 5a401aa639a0f0b /tasks/ main.yml# L69-L80
https:/
not the nova api db which is named nova_api
https:/ /github. com/openstack/ openstack- ansible- os_nova/ blob/stable/ yoga/defaults/ main.yml# L93