kolla

consoleauth is broken

Bug #1473062 reported by Steven Dake on 2015-07-09

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	kolla	Invalid	Critical	Sam Yaple

Bug Description

    compiled_sql, distilled_params
2015-07-09 12:42:08.174 1 INFO nova.service [-] Starting consoleauth node (version 12.0.0-dev45.el7.centos)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 155, in _handle_dbapi_exception
    e, statement, parameters, cursor, context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
    util.raise_from_cause(newraise, exc_info)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (OperationalError) (1054, "Unknown column 'services.last_seen_up' in 'field list'") 'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason, services.last_seen_up AS services_last_seen_up \nFROM services \nWHERE services.deleted = %s AND services.host = %s AND services.`binary` = %s \n LIMIT %s' (0, 'bigiron.broked.org', 'nova-consoleauth', 1)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup [req-076e9937-809f-43ff-88ae-1d1604b2bce9 - - - - -] (OperationalError) (1054, "Unknown column 'services.last_
seen_up' in 'field list'") 'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.
deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report
_count AS services_report_count, services.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason, services.last_seen_up AS services_last_seen_up \n
FROM services \nWHERE services.deleted = %s AND services.host = %s AND services.`binary` = %s \n LIMIT %s' (0, 'bigiron.broked.org', 'nova-consoleauth', 1)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup Traceback (most recent call last):
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 146, in wait
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup x.wait()
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 48, in wait
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup return self.thread.wait()
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup return self._exit_event.wait()
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup return hubs.get_hub().switch()
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup return self.greenlet.switch()
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup result = function(*args, **kwargs)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/service.py", line 495, in run_service
20
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup result = function(*args, **kwargs)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/service.py", line 495, in run_service
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup service.start()
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/service.py", line 162, in start
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup ctxt, self.host, self.binary)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 71, in wrapper
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup result = fn(cls, context, *args, **kwargs)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 149, in get_by_host_and_binary
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup host, binary)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 110, in service_get_by_host_and_binary
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup return IMPL.service_get_by_host_and_binary(context, host, binary)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 494, in service_get_by_host_and_bina
ry
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup filter_by(binary=binary).\
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2341, in first
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup ret = list(self[0:1])
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2208, in __getitem__
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup return list(res)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2412, in __iter__
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup return self._execute_and_instances(context)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2427, in _execute_and_instances
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup result = conn.execute(querycontext.statement, self._params)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup return meth(self, multiparams, params)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup return connection._execute_clauseelement(self, multiparams, params)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup compiled_sql, distilled_params
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup context)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 155, in _handle_dbap
i_exception
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup e, statement, parameters, cursor, context)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup util.raise_from_cause(newraise, exc_info)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup reraise(type(exception), exception, tb=exc_tb)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup context)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup cursor.execute(statement, parameters)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup self.errorhandler(self, exc, value)
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup raise errorclass, errorvalue
2015-07-09 12:42:08.436 1 ERROR nova.openstack.common.threadgroup OperationalError: (OperationalError) (1054, "Unknown column 'services.last_seen_up' in 'field list'") 'SELECT
services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, servic
es.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, se
rvices.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason, services.last_seen_up AS services_last_seen_up \nFROM services \nWHERE services.dele
ted = %s AND services.host = %s AND services.`binary` = %s \n LIMIT %s' (0, 'bigiron.broked.org', 'nova-consoleauth', 1)

Steven Dake (sdake) on 2015-07-09

Changed in kolla:
status:	New → Triaged
importance:	Undecided → Critical
milestone:	none → liberty-2

Revision history for this message

Harm Weites (harmw) wrote on 2015-07-31:

looks like it cannot connect with SQL. This kind of thing would normally happen when the Nova container responsible for setting up the database has not finished, though the consoleauth container should just restart if thats that case

Sam Yaple (s8m) on 2015-07-31

Changed in kolla:
assignee:	nobody → Sam Yaple (s8m)

Revision history for this message

Sam Yaple (s8m) wrote on 2015-07-31:

nova_consoleauth seems to be functional. This may have been fixed already

Changed in kolla:
milestone:	liberty-2 → none
status:	Triaged → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.