heat-engine:oslo_service.service CantStartEngineError: No sql_connection parameter is established

[Venkatesh Velpula]

build-id :6(4.1.0.0-6)
build type :continuous build
DISTRO. :Ubuntu 16.04.2 LTS
SKU :Newton

Setup
===================================================

controller+openstack: nodec19
compute Node : nodei16, nodei18
SMLIte Node : nodea35

====================================================
while trying to bring up a newton SMLite setup with Ubuntu 16.04 using 4.1 latest builds (6,9), we see lot of error messages related to oslo wrt heat-engine are observed in the syslogs .
snip of the syslogs is given below and attaching the log file as well.
We haven’t observed these error messages while bringing up the ocata setups either single or multimode

<snip of the sylog>

Oct 10 12:07:31 nodec19 systemd[1]: Started Heat Engine.
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.276 32142 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admin_tenant_name configuration options was deprecated in the Mitaka release in favor of an auth_plugin and its related options. This class may be removed in a future release.
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.277 32142 WARNING keystonemiddleware.auth_token [-] Configuring admin URI using auth fragments was deprecated in the Kilo release, and will be removed in the N release, use 'identity_uri\ instead.
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.277 32142 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admin endpoint
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.283 32142 INFO heat.api [-] Starting Heat REST API on 0.0.0.0:8004
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.284 32142 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.285 32142 INFO heat.common.wsgi [-] Starting 0 workers
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.289 32142 INFO heat.common.wsgi [-] Started child 32342
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.290 32342 INFO eventlet.wsgi.server [-] (32342) wsgi starting up on http://0.0.0.0:8004
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.292 32142 INFO heat.common.wsgi [-] Started child 32343
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.294 32343 INFO eventlet.wsgi.server [-] (32343) wsgi starting up on http://0.0.0.0:8004
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.296 32142 INFO heat.common.wsgi [-] Started child 32344
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.298 32344 INFO eventlet.wsgi.server [-] (32344) wsgi starting up on http://0.0.0.0:8004
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.300 32142 INFO heat.common.wsgi [-] Started child 32345
Oct 10 12:07:31 nodec19 heat-api[32142]: 2017-10-10 12:07:31.302 32345 INFO eventlet.wsgi.server [-] (32345) wsgi starting up on http://0.0.0.0:8004
Oct 10 12:07:31 nodec19 heat-api-cfn[32236]: 2017-10-10 12:07:31.671 32236 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admin_tenant_name configuration options was deprecated in the Mitaka release in favor of an auth_plugin and its related options. This class may be removed in a future release.
Oct 10 12:07:31 nodec19 heat-api-cfn[32236]: 2017-10-10 12:07:31.672 32236 WARNING keystonemiddleware.auth_token [-] Configuring admin URI using auth fragments was deprecated in the Kilo release, and will be removed in the N release, use 'identity_uri\ instead.
Oct 10 12:07:31 nodec19 heat-api-cfn[32236]: 2017-10-10 12:07:31.672 32236 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admin endpoint
Oct 10 12:07:31 nodec19 heat-api-cfn[32236]: 2017-10-10 12:07:31.677 32236 INFO heat.api.cfn [-] Starting Heat API on 0.0.0.0:8000

Oct 10 12:07:32 nodec19 heat-engine[32326]: 2017-10-10 12:07:32.969 32326 INFO oslo_service.service [-] Starting 4 workers
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.011 32381 INFO heat.engine.worker [-] Starting engine_worker (1.3) in engine 3c98ca18-ffee-4495-a95a-f0d8e416d7a3.
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.025 32382 INFO heat.engine.worker [-] Starting engine_worker (1.3) in engine 36c79d73-5102-45f3-a51a-454f287e687c.
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.035 32383 INFO heat.engine.worker [-] Starting engine_worker (1.3) in engine 68d3fd88-dba0-4a2f-84ee-fcce5f23f492.
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service [req-8c42209e-1511-4205-925e-0b61902d5c08 - - - - -] Error starting thread.
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service Traceback (most recent call last):
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 708, in run_service
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service service.start()
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 154, in wrapper
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service return f(*args, **kwargs)
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 395, in start
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service self.service_manage_cleanup()
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 154, in wrapper
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service return f(*args, **kwargs)
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 2342, in service_manage_cleanup
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service cnxt, self.host, self.binary, self.hostname)
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/heat/objects/service.py", line 91, in get_all_by_args
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service hostname))
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/heat/db/api.py", line 426, in service_get_all_by_args
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service return IMPL.service_get_all_by_args(context, host, binary, hostname)
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/heat/db/sqlalchemy/api.py", line 1162, in service_get_all_by_args
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service return (context.session.query(models.Service).
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/heat/common/context.py", line 144, in session
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service self._session = db_api.get_session()
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/heat/db/api.py", line 42, in get_session
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service return IMPL.get_session()
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/heat/db/sqlalchemy/api.py", line 75, in get_session
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service return get_facade().get_session()
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/heat/db/sqlalchemy/api.py", line 60, in get_facade
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service _facade = db_session.EngineFacade.from_config(CONF)
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1171, in from_config
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service expire_on_commit=expire_on_commit, _conf=conf)
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1092, in __init__
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service slave_connection=slave_connection)
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 363, in _start
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service engine_args, maker_args)
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 385, in _setup_for_connection
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service "No sql_connection parameter is established")
Oct 10 12:07:33 nodec19 heat-engine[32326]: 2017-10-10 12:07:33.059 32381 ERROR oslo_service.service CantStartEngineError: No sql_connection parameter is established

 </snip>

---------+---------------------+----------------+-------------------+
| id | status | ip_address | mac_address |
+---------+---------------------+----------------+-------------------+
| nodec19 | provision_completed | 10.204.217.4 | 00:25:90:C3:AF:AA |
| nodei18 | provision_completed | 10.204.217.130 | 00:25:90:E7:7E:FC |
| nodei16 | provision_completed | 10.204.217.128 | 00:25:90:E7:80:30 |
+---------+---------------------+----------------+-------------------+

root@nodec19:~# docker exec -it controller "contrail-status"
== Contrail Control ==
contrail-control: active
contrail-named: active
contrail-dns: active
contrail-control-nodemgr: active
== Contrail Config ==
contrail-api: active
contrail-schema: active
contrail-svc-monitor: active
contrail-device-manager: active
contrail-config-nodemgr: active
== Contrail Config Database==
contrail-database: active

== Contrail Web UI ==
contrail-webui: active
contrail-webui-middleware: active
== Contrail Support Services ==
zookeeper: active
rabbitmq-server: inactive (disabled on boot)
root@nodec19:~# docker exec -it analytics "contrail-status"
== Contrail Analytics ==
contrail-collector: active
contrail-analytics-api: active
contrail-query-engine: active
contrail-alarm-gen: active
contrail-snmp-collector: active
contrail-topology: active
contrail-analytics-nodemgr: active
root@nodec19:~# docker exec -it analyticsdb "contrail-status"
== Contrail Database ==
contrail-database: active

kafka: active
contrail-database-nodemgr: active

root@nodec19:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-heat 4.1.0.0-6 6
contrail-nova-networkapi 4.1.0.0-6 6
contrail-openstack 4.1.0.0-6 6
contrail-openstack-dashboard 4.1.0.0-6 6
contrail-setup 4.1.0.0-6 6
contrail-utils 4.1.0.0-6 6
neutron-plugin-contrail 4.1.0.0-6 6
nova-api 2:14.0.2-0ubuntu1~cloud0.1contrail 6
nova-common 2:14.0.2-0ubuntu1~cloud0.1contrail 6
nova-conductor 2:14.0.2-0ubuntu1~cloud0.1contrail 6
nova-console 2:14.0.2-0ubuntu1~cloud0.1contrail 6
nova-consoleauth 2:14.0.2-0ubuntu1~cloud0.1contrail 6
nova-novncproxy 2:14.0.2-0ubuntu1~cloud0.1contrail 6
nova-scheduler 2:14.0.2-0ubuntu1~cloud0.1contrail 6
python-contrail 4.1.0.0-6 6
python-neutronclient 1:6.0.0-0ubuntu1~cloud0.2contrail 6
python-nova 2:14.0.2-0ubuntu1~cloud0.1contrail 6

root@nodei16:~# contrail-status
== Contrail vRouter ==
contrail-vrouter-agent: active
contrail-vrouter-nodemgr: active