nova-conductor/cinder-volume starting when database isn't ready and didn't have all the data

Bug #1334005 reported by Aleksandr Shaposhnikov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
New
Medium
Fuel Library (Deprecated)

Bug Description

During deployment of Ubuntu+HA there is a lot of errors in nova logs about starting nova-conductor and cinder-volume services. It looks like nova-conductor/cinder didn't ready for start because there is no corresponding records in DB to work with.
Here is traceback from nova logs for nova-conductor (cinder logs looks the same way).

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 117, in wait
    x.wait()
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 49, in wait
    return self.thread.wait()
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
    return self._exit_event.wait()
  File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
    return hubs.get_hub().switch()
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
    return self.greenlet.switch()
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 480, in run_service
    service.start()
  File "/usr/lib/python2.7/dist-packages/nova/service.py", line 168, in start
    self.host, self.binary)
  File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 193, in service_get_by_args
    binary=binary)
  File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 966, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 441, in service_get_all_by
    result = self.db.service_get_by_args(context, host, binary)
  File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 134, in service_get_by_args
    return IMPL.service_get_by_args(context, host, binary)
  File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 501, in service_get_by_args
    filter_by(binary=binary).\
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2278, in first
    ret = list(self[0:1])
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2145, in __getitem__
    return list(res)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2349, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2364, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 662, in execute
    params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
    exc_info
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 195, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 324, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
ProgrammingError: (ProgrammingError) (1146, "Table 'nova.services' doesn't exist") 'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason \nFROM services \nWHERE services.deleted = %s AND services.host = %s AND services.`binary` = %s \n LIMIT %s' (0, 'node-25', 'nova-conductor', 1)

description: updated
summary: - nova-conductor starting when database isn't ready and didn't have all
- the data
+ nova-conductor/cinder-volume starting when database isn't ready and
+ didn't have all the data
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
importance: Undecided → Medium
milestone: none → 5.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.