Installing OpenStack liberty on Centos 7.1 i have error with starting nova service (on each compute node) - after that installation is broken but OpenStack is operational and i can create and manage created VMs.
Chef console output:
==> chef_server: 192.168.121.198 * service[nova-compute] action restart
==> chef_server: 192.168.121.198
==> chef_server: 192.168.121.198 ================================================================================
==> chef_server: 192.168.121.198 Error executing action `restart` on resource 'service[nova-compute]'
==> chef_server: 192.168.121.198 ================================================================================
==> chef_server: 192.168.121.198
==> chef_server: 192.168.121.198 Mixlib::ShellOut::CommandTimeout
==> chef_server: 192.168.121.198 --------------------------------
==> chef_server: 192.168.121.198 Command timed out after 600s:
==> chef_server: 192.168.121.198 Command exceeded allowed execution time, process terminated
==> chef_server: 192.168.121.198 ---- Begin output of /bin/systemctl --system restart openstack-nova-compute ----
==> chef_server: 192.168.121.198 STDOUT:
==> chef_server: 192.168.121.198 STDERR:
==> chef_server: 192.168.121.198 ---- End output of /bin/systemctl --system restart openstack-nova-compute ----
==> chef_server: 192.168.121.198 Ran /bin/systemctl --system restart openstack-nova-compute returned
==> chef_server: 192.168.121.198
==> chef_server: 192.168.121.198 Resource Declaration:
==> chef_server: 192.168.121.198 ---------------------
==> chef_server: 192.168.121.198 # In /var/chef/cache/cookbooks/openstack-compute/recipes/compute.rb
==> chef_server: 192.168.121.198
==> chef_server: 192.168.121.198 118: service 'nova-compute' do
==> chef_server: 192.168.121.198 119: service_name platform_options['compute_compute_service']
==> chef_server: 192.168.121.198 120: supports status: true, restart: true
==> chef_server: 192.168.121.198 121: subscribes :restart, resources('template[/etc/nova/nova.conf]')
==> chef_server: 192.168.121.198 122: subscribes :restart, resources('file[docker.filter]')
==> chef_server: 192.168.121.198 123:
==> chef_server: 192.168.121.198 124: action [:enable, :start]
==> chef_server: 192.168.121.198 125: end
==> chef_server: 192.168.121.198
==> chef_server: 192.168.121.198 Compiled Resource:
==> chef_server: 192.168.121.198 ------------------
==> chef_server: 192.168.121.198 # Declared in /var/chef/cache/cookbooks/openstack-compute/recipes/compute.rb:118:in `from_file'
==> chef_server: 192.168.121.198
==> chef_server: 192.168.121.198 service("nova-compute") do
==> chef_server: 192.168.121.198 action [:enable, :start]
==> chef_server: 192.168.121.198 updated true
==> chef_server: 192.168.121.198 supports {:status=>true, :restart=>true}
==> chef_server: 192.168.121.198 retries 0
==> chef_server: 192.168.121.198 retry_delay 2
==> chef_server: 192.168.121.198 default_guard_interpreter :default
==> chef_server: 192.168.121.198 service_name "openstack-nova-compute"
==> chef_server: 192.168.121.198 enabled true
==> chef_server: 192.168.121.198 pattern "nova-compute"
==> chef_server: 192.168.121.198 declared_type :service
==> chef_server: 192.168.121.198 cookbook_name "openstack-compute"
==> chef_server: 192.168.121.198 recipe_name "compute"
==> chef_server: 192.168.121.198 end
==> chef_server: 192.168.121.198
==> chef_server: 192.168.121.198 Platform:
==> chef_server: 192.168.121.198 ---------
==> chef_server: 192.168.121.198 x86_64-linux
==> chef_server: 192.168.121.198
Nova-compute.log:
016-05-05 11:15:18.340 19217 WARNING oslo_config.cfg [req-ae9205f3-5736-4353-9a15-a7c3bc6f27b8 - - - - -] Option "admin_auth_url" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future.
2016-05-05 11:15:18.340 19217 WARNING oslo_config.cfg [req-ae9205f3-5736-4353-9a15-a7c3bc6f27b8 - - - - -] Option "admin_password" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future.
2016-05-05 11:15:18.341 19217 WARNING oslo_config.cfg [req-ae9205f3-5736-4353-9a15-a7c3bc6f27b8 - - - - -] Option "admin_tenant_name" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future.
2016-05-05 11:15:18.341 19217 WARNING oslo_config.cfg [req-ae9205f3-5736-4353-9a15-a7c3bc6f27b8 - - - - -] Option "admin_username" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future.
2016-05-05 11:15:18.341 19217 WARNING oslo_config.cfg [req-ae9205f3-5736-4353-9a15-a7c3bc6f27b8 - - - - -] Option "auth_strategy" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future.
2016-05-05 11:15:18.342 19217 WARNING oslo_config.cfg [req-ae9205f3-5736-4353-9a15-a7c3bc6f27b8 - - - - -] Option "vnc_enabled" from group "DEFAULT" is deprecated. Use option "enabled" from group "vnc".
2016-05-05 11:15:18.342 19217 WARNING oslo_config.cfg [req-ae9205f3-5736-4353-9a15-a7c3bc6f27b8 - - - - -] Option "vnc_keymap" from group "DEFAULT" is deprecated. Use option "keymap" from group "vnc".
2016-05-05 11:15:18.342 19217 WARNING oslo_config.cfg [req-ae9205f3-5736-4353-9a15-a7c3bc6f27b8 - - - - -] Option "novncproxy_base_url" from group "DEFAULT" is deprecated. Use option "novncproxy_base_url" from group "vnc".
2016-05-05 11:15:18.343 19217 WARNING oslo_config.cfg [req-ae9205f3-5736-4353-9a15-a7c3bc6f27b8 - - - - -] Option "vncserver_listen" from group "DEFAULT" is deprecated. Use option "vncserver_listen" from group "vnc".
2016-05-05 11:15:18.343 19217 WARNING oslo_config.cfg [req-ae9205f3-5736-4353-9a15-a7c3bc6f27b8 - - - - -] Option "vncserver_proxyclient_address" from group "DEFAULT" is deprecated. Use option "vncserver_proxyclient_address" from group "vnc".
2016-05-05 11:15:18.343 19217 WARNING oslo_config.cfg [req-ae9205f3-5736-4353-9a15-a7c3bc6f27b8 - - - - -] Option "xvpvncproxy_base_url" from group "DEFAULT" is deprecated. Use option "xvpvncproxy_base_url" from group "vnc".
2016-05-05 11:15:18.402 19217 WARNING nova.virt.libvirt.driver [req-0ad29c22-d967-45b5-be21-862b7ce0c209 - - - - -] Cannot update service status on host "compute02" since it is not registered.
2016-05-05 11:15:18.874 19217 ERROR nova.compute.manager [req-0ad29c22-d967-45b5-be21-862b7ce0c209 - - - - -] No compute node record for host compute02
2016-05-05 11:15:18.880 19217 WARNING nova.compute.monitors [req-0ad29c22-d967-45b5-be21-862b7ce0c209 - - - - -] Excluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors).
2016-05-05 11:15:18.941 19217 WARNING nova.compute.resource_tracker [req-0ad29c22-d967-45b5-be21-862b7ce0c209 - - - - -] No compute node record for compute02:compute02
2016-05-05 11:15:19.110 19217 WARNING oslo_config.cfg [req-0ad29c22-d967-45b5-be21-862b7ce0c209 - - - - -] Option "rpc_thread_pool_size" from group "DEFAULT" is deprecated. Use option "executor_thread_pool_size" from group "DEFAULT".
2016-05-05 11:15:27.459 19217 WARNING nova.compute.manager [req-808cccc3-1c2e-46e3-9534-7c732647f753 - - - - -] Bandwidth usage not supported by hypervisor.
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db [-] Unexpected error while reporting service status
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 88, in _report_state
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db service.service_ref.save()
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 197, in wrapper
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db ctxt, self, fn.__name__, args, kwargs)
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 242, in object_action
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db objmethod=objmethod, args=args, kwargs=kwargs)
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db retry=self.retry)
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db timeout=timeout, retry=retry)
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db retry=retry)
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db raise result
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db RemoteError: Remote error: DBConnectionError (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '192.168.3.60' (111)") [SQL: u'SELECT 1']
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 447, in _object_dispatch\n return getattr(target, method)(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 213, in wrapper\n return fn(self, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 250, in save\n db_service = db.service_update(self._context, self.id, updates)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 153, in service_update\n return IMPL.service_update(context, service_id, values)\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 146, in wrapper\n ectxt.value = e.inner_exc\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 136, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 525, in service_update\n service_ref = _service_get(context, service_id, session=session)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 431, in _service_get\n result = query.first()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2634, in first\n ret = list(self[0:1])\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__\n return list(res)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__\n return self._execute_and_instances(context)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances\n close_with_result=True)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session\n **kw)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 905, in connection\n execution_options=execution_options)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind\n engine, execution_options)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind\n conn = bind.contextual_connect()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect\n **kwargs)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 92, in __init__\n self.dispatch.engine_connect(self, self.__branch)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__\n fn(*args, **kw)\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 80, in _connect_ping_listener\n connection.scalar(select([1]))\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 844, in scalar\n return self.execute(object, *multiparams, **params).scalar()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute\n return meth(self, multiparams, params)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection\n return connection._execute_clauseelement(self, multiparams, params)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement\n compiled_sql, distilled_params\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context\n None, None)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception\n util.raise_from_cause(newraise, exc_info)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1071, in _execute_context\n conn = self._revalidate_connection()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 393, in _revalidate_connection\n self.__connection = self.engine.raw_connection(_connection=self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection\n self.pool.unique_connection, _connection)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2080, in _wrap_pool_connect\n util.reraise(*sys.exc_info())\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect\n return fn()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection\n return _ConnectionFairy._checkout(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 485, in checkout\n rec.checkin()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 482, in checkout\n dbapi_connection = rec.get_connection()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 563, in get_connection\n self.connection = self.__connect()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect\n connection = self.__pool._invoke_creator(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect\n return dialect.connect(*cargs, **cparams)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect\n return self.dbapi.connect(*cargs, **cparams)\n', u' File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect\n return Connection(*args, **kwargs)\n', u' File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__\n super(Connection, self).__init__(*args, **kwargs2)\n', u'DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can\'t connect to MySQL server on \'192.168.3.60\' (111)") [SQL: u\'SELECT 1\']\n'].
2016-05-05 11:18:24.151 19217 ERROR nova.servicegroup.drivers.db
2016-05-05 11:18:26.879 19217 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.168.3.60:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds.
2016-05-05 11:18:26.892 19217 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.168.3.60:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds.
2016-05-05 11:18:27.893 19217 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.168.3.60:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.
2016-05-05 11:18:27.901 19217 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.168.3.60:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.
2016-05-05 11:18:30.006 19217 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.168.3.60:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.
2016-05-05 11:18:30.006 19217 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.168.3.60:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.
2016-05-05 11:18:34.130 19217 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.168.3.60:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds.
Please check whether the issue still exists for you with the current cookbooks for Mitaka or Newton.