Config schema is in initializing state in 5.1.0

[musharani]

Broughtup the setup with 5.1.0 build in multi node setup. Schema is not coming to active state. It is showing as schema: initializing (Database:Cassandra[] connection down).

This issue is seen in master queens. Setup also available.

Work around: Schema is coming up after restarting the config_schema_1 docker .

[root@nodec57 ~]#
== Contrail config ==
svc-monitor: active
nodemgr: active
device-manager: backup
api: active
schema: initializing (Database:Cassandra[] connection down)

instances:
  nodei1:
    ip: 10.204.216.150
    provider: bms
    roles:
        openstack_compute: null
        vrouter:
            PHYSICAL_INTERFACE: eno2
  nodec8:
    ip: 10.204.216.65
    provider: bms
    roles:
        analytics: null
        analytics_database: null
        config: null
        config_database: null
        control: null
        openstack: null
        webui: null
  nodec57:
    ip: 10.204.216.153
    provider: bms
    roles:
        analytics: null
        analytics_database: null
        config: null
        config_database: null
        control: null
        openstack: null
        webui: null
  nodei3:
    ip: 10.204.217.115
    provider: bms
    roles:
        openstack_compute: null
        vrouter:
            PHYSICAL_INTERFACE: eno2
  nodei2:
    ip: 10.204.217.114
    provider: bms
    roles:
        openstack_compute: null
        vrouter:
            PHYSICAL_INTERFACE: eno2
  nodec7:
    ip: 10.204.216.64
    provider: bms
    roles:
        analytics: null
        analytics_database: null
        config: null
        config_database: null
        control: null
        openstack: null
        webui: null

Got the below error in config-api log. Log location://root/config_api

10/12/2018 05:10:44 PM [contrail-api] [INFO]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_NOTICE]
<pre>Traceback (most recent call last):
  File "/usr/bin/contrail-api", line 9, in &lt;module>
    load_entry_point('contrail-api-server==0.1dev', 'console_scripts', 'contrail-api')()
  File "/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 5115, in server_main
    main(args_str, VncApiServer(args_str))
  File "/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 2121, in __init__
    self._db_init_entries()
  File "/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 3485, in _db_init_entries
    autonomous_system=64512, config_version=CONFIG_VERSION))
  File "/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 3816, in create_singleton_entry
    cass_uuid = self._db_conn._object_db.fq_name_to_uuid(obj_type, fq_name)
  File "/usr/lib/python2.7/site-packages/cfgm_common/vnc_cassandra.py", line 521, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cfgm_common/vnc_cassandra.py", line 1446, in fq_name_to_uuid
    finish=fq_name_str + ';')
  File "/usr/lib/python2.7/site-packages/cfgm_common/vnc_cassandra.py", line 208, in get
    finish=finish)
  File "/usr/lib/python2.7/site-packages/cfgm_common/vnc_cassandra.py", line 229, in multiget
    column_count=column_count)
  File "/usr/lib/python2.7/site-packages/cfgm_common/vnc_cassandra.py", line 521, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pycassa/columnfamily.py", line 772, in multiget
    packed_keys[offset:offset + buffer_size], cp, sp, consistency)
  File "/usr/lib/python2.7/site-packages/pycassa/pool.py", line 577, in execute
    return getattr(conn, f)(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pycassa/pool.py", line 127, in new_f
    result = f(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pycassa/cassandra/Cassandra.py", line 669, in multiget_slice
    return self.recv_multiget_slice()
  File "/usr/lib/python2.7/site-packages/pycassa/cassandra/Cassandra.py", line 695, in recv_multiget_slice
    raise result.ire
InvalidRequestException: InvalidRequestException(why='unconfigured table obj_fq_name_table')
</pre>

[vimal (vappachan)]

This issue is also seen in queens-5.0-327 .

Logs are copied in /cs-shared/bugs/1797560

== Contrail config ==
svc-monitor: active
nodemgr: active
device-manager: backup
api: active
schema: initializing (Database:Cassandra[] connection down)

[Jeba Paulaiyan (jebap)]

This was debugged by Nagendra / Shivayogi and root cause identified. Workaround is to restart schema trfr . In fab based provisioning, this restart under this condition was part of provisioning.