"DBNonExistentTable: (sqlite3.OperationalError) no such table: services" when starting nova-metadata under uwsgi

Bug #1815082 reported by Matt Riedemann on 2019-02-07
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Unassigned
Pike
Medium
Unassigned
Queens
Medium
Unassigned
Rocky
Medium
Unassigned

Bug Description

Jens Harbott reported this in devstack:

https://review.openstack.org/#/c/635519/

He's running the n-api-meta service in the subnode per the multinode guide:

https://docs.openstack.org/devstack/latest/guides/multinode-lab.html

However with current devstack, which doesn't configure nova.conf with database access on the subnode, n-api-meta fails with this:

http://paste.openstack.org/show/744683/

Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: CRITICAL nova [None req-538e6b11-f91b-48d8-9c7a-d177dff7739b None None] Unhandled error: DBNonExistentTable: (sqlite3.OperationalError) no such t
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova Traceback (most recent call last):
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/bin/nova-metadata-wsgi", line 52, in <module>
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova application = init_application()
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/api/metadata/wsgi.py", line 20, in init_application
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return wsgi_app.init_application(NAME)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 82, in init_application
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova _setup_service(CONF.host, name)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 49, in _setup_service
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova ctxt, host, binary)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova result = fn(cls, context, *args, **kwargs)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/objects/service.py", line 334, in get_by_host_and_binary
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova host, binary)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/db/api.py", line 127, in service_get_by_host_and_binary
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return IMPL.service_get_by_host_and_binary(context, host, binary)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 242, in wrapped
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return f(context, *args, **kwargs)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 500, in service_get_by_host_and_binary
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova filter_by(binary=binary).\
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2979, in first
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova ret = list(self[0:1])
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2771, in __getitem__
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return list(res)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3081, in __iter__
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return self._execute_and_instances(context)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova result = conn.execute(querycontext.statement, self._params)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 980, in execute
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return meth(self, multiparams, params)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova return connection._execute_clauseelement(self, multiparams, params)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova distilled_params,
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova e, statement, parameters, cursor, context
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova util.raise_from_cause(newraise, exc_info)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova reraise(type(exception), exception, tb=exc_tb, cause=cause)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova cursor, statement, parameters, context
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 536, in do_execute
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova cursor.execute(statement, parameters)
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova DBNonExistentTable: (sqlite3.OperationalError) no such table: services [SQL: u'SELECT services.created_at AS services_created_at, serv
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: ERROR nova
Feb 07 13:33:32 jh-devstack01a <email address hidden>[22651]: unable to load app 0 (mountpoint='') (callable not found or import error)

I believe this is due to the nova-metadata service not configuring the conductor indirection API when run in uwsgi mode like it is when run with eventlet:

https://github.com/openstack/nova/blob/3cfcd117ce7ddb5a8d94396191cde559db73fb86/nova/cmd/api_metadata.py#L45

It also appears that we don't have any multinode CI jobs that run with the n-api-meta service configured so we don't hit this in the gate.

The workaround which most deployments probably use is configuring nova.conf with database access and run n-api-meta with that, even from subnodes.

Fix proposed to branch: master
Review: https://review.openstack.org/635577

Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
status: New → In Progress

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.opendev.org/635577
Reason: I'm not actively working on this.

Matt Riedemann (mriedem) on 2019-10-22
Changed in nova:
assignee: Matt Riedemann (mriedem) → nobody
status: In Progress → Confirmed

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.opendev.org/635577

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers