OpenStack Nova Cloud Controller Charm

unable to find services table, does not exist

Bug #1818865 reported by Alexander Balderson on 2019-03-06

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Nova Cloud Controller Charm	Expired	Undecided	Unassigned

Bug Description

Juju status reports "Services not running that should be: apache2, nova-spiceproxy, haproxy, nova-scheduler, memcached, nova-conductor"
the nova-consoleauth.log however shows that the unit never had a database to connect to.

2019-03-06 05:18:01.427 792538 ERROR oslo_service.service Traceback (most recent call last):
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service context)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 508, in do_execute
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service cursor.execute(statement, parameters)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 165, in execute
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service result = self._query(query)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 321, in _query
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service conn.query(q)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 860, in query
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1061, in _read_query_result
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service result.read()
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1349, in read
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service first_packet = self.connection._read_packet()
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1018, in _read_packet
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service packet.check_error()
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 384, in check_error
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service err.raise_mysql_exception(self._data)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service raise errorclass(errno, errval)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service pymysql.err.ProgrammingError: (1146, "Table 'nova.services' doesn't exist")
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service The above exception was the direct cause of the following exception:
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service Traceback (most recent call last):
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 794, in run_service
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service service.start()
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/service.py", line 166, in start
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service ctxt, self.host, self.binary)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service result = fn(cls, context, *args, **kwargs)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/objects/service.py", line 329, in get_by_host_and_binary
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service host, binary)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/db/api.py", line 127, in service_get_by_host_and_binary
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service return IMPL.service_get_by_host_and_binary(context, host, binary)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service return f(context, *args, **kwargs)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 513, in service_get_by_host_and_binary
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service filter_by(binary=binary).\
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 2835, in first
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service ret = list(self[0:1])
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 2627, in __getitem__
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service return list(res)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 2935, in __iter__
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service return self._execute_and_instances(context)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 2958, in _execute_and_instances
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service result = conn.execute(querycontext.statement, self._params)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 948, in execute
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service return meth(self, multiparams, params)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service return connection._execute_clauseelement(self, multiparams, params)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service compiled_sql, distilled_params
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service context)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service util.raise_from_cause(newraise, exc_info)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 186, in reraise
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service raise value.with_traceback(tb)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service context)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 508, in do_execute
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service cursor.execute(statement, parameters)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 165, in execute
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service result = self._query(query)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 321, in _query
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service conn.query(q)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 860, in query
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1061, in _read_query_result
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service result.read()
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1349, in read
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service first_packet = self.connection._read_packet()
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1018, in _read_packet
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service packet.check_error()
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 384, in check_error
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service err.raise_mysql_exception(self._data)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service raise errorclass(errno, errval)
2019-03-06 05:18:01.427 792538 ERROR oslo_service.service sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, "Table 'nova.services' doesn't exist") [SQL: 'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.uuid AS services_uuid, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason, services.last_seen_up AS services_last_seen_up, services.forced_down AS services_forced_down, services.version AS services_version \nFROM services \nWHERE services.deleted = %(deleted_1)s AND services.host = %(host_1)s AND services.`binary` = %(binary_1)s \n LIMIT %(param_1)s'] [parameters: {'deleted_1': 0, 'host_1': 'juju-0cc286-3-lxd-5', 'binary_1': 'nova-consoleauth', 'param_1': 1}] (Background on this error at: http://sqlalche.me/e/f405)

Full logs will be attached

Revision history for this message

Alexander Balderson (asbalderson) wrote on 2019-03-06:

juju-crashdump-openstack-2019-03-06-05.17.01.tar.gz Edit (249.1 MiB, application/x-tar)

Revision history for this message

Chris MacNaughton (chris.macnaughton) wrote on 2019-05-14:

I would expect to see this early in a deployment (before migrations are run). Can you attach the juju bundle that showed this error, or a new reproducer?

Changed in charm-nova-cloud-controller:
status:	New → Incomplete

Revision history for this message

Alexander Balderson (asbalderson) wrote on 2019-05-15:

bundle.yaml Edit (47.1 KiB, text/plain)

attached is the bundle. It's deploying Rocky Bionic.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2019-07-15:

[Expired for OpenStack nova-cloud-controller charm because there has been no activity for 60 days.]

Changed in charm-nova-cloud-controller:
status:	Incomplete → Expired

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.