--------------------------Enviornment------------------------------------------------------------
Controller - IP A 10.1.1.100
Compute - IP A 10.1.1.110
Release - Openstack Stein
--------------------------------Error -----------------------------------------------------------
root@overcloud-controller etc]# systemctl status openstack-nova-api.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service
● openstack-nova-api.service - OpenStack Nova API Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-11-09 15:44:21 +07; 58min ago
Main PID: 1213 (nova-api)
CGroup: /system.slice/openstack-nova-api.service
├─ 1213 /usr/bin/python2 /usr/bin/nova-api
├─17185 /usr/bin/python2 /usr/bin/nova-api
└─17186 /usr/bin/python2 /usr/bin/nova-api
Nov 09 15:42:51 overcloud-controller systemd[1]: Starting OpenStack Nova API Server...
Nov 09 15:44:11 overcloud-controller nova-api[1213]: /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: PkgResourcesDepreca...ately.
Nov 09 15:44:11 overcloud-controller nova-api[1213]: return pkg_resources.EntryPoint.parse("x=" + s).load(False)
Nov 09 15:44:21 overcloud-controller systemd[1]: Started OpenStack Nova API Server.
● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-scheduler.service; enabled; vendor preset: disabled)
Active: activating (start) since Sat 2019-11-09 16:42:25 +07; 15s ago
Main PID: 17141 (nova-scheduler)
CGroup: /system.slice/openstack-nova-scheduler.service
└─17141 /usr/bin/python2 /usr/bin/nova-scheduler
Nov 09 16:42:25 overcloud-controller systemd[1]: openstack-nova-scheduler.service holdoff time over, scheduling restart.
Nov 09 16:42:25 overcloud-controller systemd[1]: Stopped OpenStack Nova Scheduler Server.
Nov 09 16:42:25 overcloud-controller systemd[1]: Starting OpenStack Nova Scheduler Server...
● openstack-nova-conductor.service - OpenStack Nova Conductor Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-conductor.service; enabled; vendor preset: disabled)
Active: activating (start) since Sat 2019-11-09 16:42:31 +07; 9s ago
Main PID: 17156 (nova-conductor)
CGroup: /system.slice/openstack-nova-conductor.service
└─17156 /usr/bin/python2 /usr/bin/nova-conductor
Nov 09 16:42:31 overcloud-controller systemd[1]: openstack-nova-conductor.service holdoff time over, scheduling restart.
Nov 09 16:42:31 overcloud-controller systemd[1]: Stopped OpenStack Nova Conductor Server.
Nov 09 16:42:31 overcloud-controller systemd[1]: Starting OpenStack Nova Conductor Server...
● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-novncproxy.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-11-09 15:42:51 +07; 59min ago
Main PID: 1210 (nova-novncproxy)
CGroup: /system.slice/openstack-nova-novncproxy.service
└─1210 /usr/bin/python2 /usr/bin/nova-novncproxy --web /usr/share/novnc/
Nov 09 15:42:51 overcloud-controller systemd[1]: Started OpenStack Nova NoVNC Proxy Server.
Hint: Some lines were ellipsized, use -l to show in full.
-----------------------------------Expected Result ----------------------------------------
root@overcloud-controller etc]# systemctl status openstack-nova-api.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service
● openstack-nova-api.service - OpenStack Nova API Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-11-09 15:44:21 +07; 58min ago
Main PID: 1213 (nova-api)
CGroup: /system.slice/openstack-nova-api.service
├─ 1213 /usr/bin/python2 /usr/bin/nova-api
├─17193 /usr/bin/python2 /usr/bin/nova-api
└─17194 /usr/bin/python2 /usr/bin/nova-api
Nov 09 15:42:51 overcloud-controller systemd[1]: Starting OpenStack Nova API Server...
Nov 09 15:44:11 overcloud-controller nova-api[1213]: /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: PkgResourcesDepreca...ately.
Nov 09 15:44:11 overcloud-controller nova-api[1213]: return pkg_resources.EntryPoint.parse("x=" + s).load(False)
Nov 09 15:44:21 overcloud-controller systemd[1]: Started OpenStack Nova API Server.
● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-scheduler.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-11-09 16:42:42 +07; 1s ago
Main PID: 17141 (nova-scheduler)
CGroup: /system.slice/openstack-nova-scheduler.service
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: first_packet = self.connection._read_packet()
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, ...packet
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: packet.check_error()
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in ..._error
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: err.raise_mysql_exception(self._data)
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise...eption
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: raise errorclass(errno, errval)
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova.comp...local_
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: Removing descriptor: 5
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: Exception AssertionError: AssertionError('do not call blocking functions f...gnored
● openstack-nova-conductor.service - OpenStack Nova Conductor Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-conductor.service; enabled; vendor preset: disabled)
Active: activating (start) since Sat 2019-11-09 16:42:31 +07; 11s ago
Main PID: 17156 (nova-conductor)
CGroup: /system.slice/openstack-nova-conductor.service
└─17156 /usr/bin/python2 /usr/bin/nova-conductor
Nov 09 16:42:31 overcloud-controller systemd[1]: openstack-nova-conductor.service holdoff time over, scheduling restart.
Nov 09 16:42:31 overcloud-controller systemd[1]: Stopped OpenStack Nova Conductor Server.
Nov 09 16:42:31 overcloud-controller systemd[1]: Starting OpenStack Nova Conductor Server...
● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-novncproxy.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-11-09 15:42:51 +07; 59min ago
Main PID: 1210 (nova-novncproxy)
CGroup: /system.slice/openstack-nova-novncproxy.service
└─1210 /usr/bin/python2 /usr/bin/nova-novncproxy --web /usr/share/novnc/
Nov 09 15:42:51 overcloud-controller systemd[1]: Started OpenStack Nova NoVNC Proxy Server.
Hint: Some lines were ellipsized, use -l to show in full.
-----------------------------------NOVA SCHEDULAR LOGS-------------------------------------
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service Traceback (most recent call last):
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 796, in run_service
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service service.start()
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 166, in start
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ctxt, self.host, self.binary)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result = fn(cls, context, *args, **kwargs)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 334, in get_by_host_and_binary
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service host, binary)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 127, in service_get_by_host_and_binary
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return IMPL.service_get_by_host_and_binary(context, host, binary)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 251, in wrapped
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return f(context, *args, **kwargs)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 508, in service_get_by_host_and_binary
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service filter_by(binary=binary).\
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2979, in first
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ret = list(self[0:1])
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2771, in __getitem__
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return list(res)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3081, in __iter__
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return self._execute_and_instances(context)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result = conn.execute(querycontext.statement, self._params)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 980, in execute
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return meth(self, multiparams, params)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return connection._execute_clauseelement(self, multiparams, params)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service distilled_params,
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service e, statement, parameters, cursor, context
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service util.raise_from_cause(newraise, exc_info)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service cursor, statement, parameters, context
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service cursor.execute(statement, parameters)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 170, in execute
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result = self._query(query)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 328, in _query
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service conn.query(q)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 516, in query
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 727, in _read_query_result
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result.read()
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1066, in read
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service first_packet = self.connection._read_packet()
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service packet.check_error()
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service err.raise_mysql_exception(self._data)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service raise errorclass(errno, errval)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.uuid AS services_uuid, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason, services.last_seen_up AS services_last_seen_up, services.forced_down AS services_forced_down, services.version AS services_version \nFROM services \nWHERE services.deleted = %(deleted_1)s AND services.host = %(host_1)s AND services.`binary` = %(binary_1)s \n LIMIT %(param_1)s'] [parameters: {u'host_1': 'overcloud-controller', u'param_1': 1, u'deleted_1': 0, u'binary_1': 'nova-scheduler'}] (Background on this error at: http://sqlalche.me/e/f405)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service
--------------------------------------------Steps Taken--------------------------------------
Restart Rabbit mq
Created NOVA DB Again, New privileges provided.
Restarted nova services
This doesn't look like a nova bug. There are obviously some setup issues like the "services" table doesn't exist when the service starts. Did you run "nova-manage db sync" on the cell1 database?
The automatic restarts look like a systemd setup for your services. Where are you getting your packages?