"DBNonExistentTable: (sqlite3.OperationalError) no such table: services" when starting nova-metadata under uwsgi

[Matt Riedemann]

Jens Harbott reported this in devstack:

https://review.openstack.org/#/c/635519/

He's running the n-api-meta service in the subnode per the multinode guide:

https://docs.openstack.org/devstack/latest/guides/multinode-lab.html

However with current devstack, which doesn't configure nova.conf with database access on the subnode, n-api-meta fails with this:

http://paste.openstack.org/show/744683/

Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: CRITICAL nova [None req-538e6b11-f91b-48d8-9c7a-d177dff7739b None None] Unhandled error: DBNonExistentTable: (sqlite3.OperationalError) no such t
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova Traceback (most recent call last):
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/bin/nova-metadata-wsgi", line 52, in <module>
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova application = init_application()
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/api/metadata/wsgi.py", line 20, in init_application
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return wsgi_app.init_application(NAME)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 82, in init_application
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova _setup_service(CONF.host, name)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 49, in _setup_service
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova ctxt, host, binary)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova result = fn(cls, context, *args, **kwargs)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/objects/service.py", line 334, in get_by_host_and_binary
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova host, binary)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/db/api.py", line 127, in service_get_by_host_and_binary
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return IMPL.service_get_by_host_and_binary(context, host, binary)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 242, in wrapped
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return f(context, *args, **kwargs)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 500, in service_get_by_host_and_binary
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova filter_by(binary=binary).\
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2979, in first
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova ret = list(self[0:1])
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2771, in __getitem__
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return list(res)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3081, in __iter__
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return self._execute_and_instances(context)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova result = conn.execute(querycontext.statement, self._params)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 980, in execute
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return meth(self, multiparams, params)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return connection._execute_clauseelement(self, multiparams, params)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova distilled_params,
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova e, statement, parameters, cursor, context
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova util.raise_from_cause(newraise, exc_info)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova reraise(type(exception), exception, tb=exc_tb, cause=cause)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova cursor, statement, parameters, context
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 536, in do_execute
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova cursor.execute(statement, parameters)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova DBNonExistentTable: (sqlite3.OperationalError) no such table: services [SQL: u'SELECT services.created_at AS services_created_at, serv
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: unable to load app 0 (mountpoint='') (callable not found or import error)

I believe this is due to the nova-metadata service not configuring the conductor indirection API when run in uwsgi mode like it is when run with eventlet:

https://github.com/openstack/nova/blob/3cfcd117ce7ddb5a8d94396191cde559db73fb86/nova/cmd/api_metadata.py#L45

It also appears that we don't have any multinode CI jobs that run with the n-api-meta service configured so we don't hit this in the gate.

The workaround which most deployments probably use is configuring nova.conf with database access and run n-api-meta with that, even from subnodes.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/635577

[David O Neill (dmzoneill)]

Hi,

I seem to hitting the same bug.

https://paste.ubuntu.com/p/nJkPWKBwbH/

https://paste.ubuntu.com/p/9g2ZT22bJ5/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.opendev.org/635577
Reason: I'm not actively working on this.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.opendev.org/635577