SM:Multi Node HA : R3.1 build 4 : contrail-svc-monitor is in failed state with InvalidRequestException(why="Keyspace 'svc_monitor_keyspace' does not exist"

[sundarkh]

SM:Multi Node HA : R3.1 build 4 : contrail-svc-monitor is in failed state on cfgm node

SM : 10.204.217.155

Sm_files/ cluster5sanity_new_param.json
Sm_files/server5sanity_new_param.json

Config Nodes : [u'nodeg21', u'nodec58']
Control Nodes : [u'nodeg33', u'nodec58']
Compute Nodes : [u'nodec38', u'nodeg17']
Openstack Node : nodeg33
WebUI Node : nodeg21
Analytics Nodes : [u'nodeg21']
Physical Devices : [u"'mx2'"]

Seen in nodeg21,nodec58,nodeg33

nodeg21
--------

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager active
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor failed
ifmap active

Traceback in contrail-svc-monitor-stdout.log

07/21/2016 08:33:26 PM [contrail-svc-monitor]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_NOTICE]
<pre>Traceback (most recent call last):
  File "/usr/bin/contrail-svc-monitor", line 9, in &lt;module>
    load_entry_point('svc-monitor==0.1dev', 'console_scripts', 'contrail-svc-monitor')()
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 902, in server_main
    main()
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 896, in main
    run_svc_monitor, args)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/zkclient.py", line 344, in master_election
    self._election.run(func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/kazoo/recipe/election.py", line 48, in run
    func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 853, in run_svc_monitor
    monitor = SvcMonitor(args)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 91, in __init__
    self._cassandra = ServiceMonitorDB(self._args, self.logger)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/db.py", line 43, in __init__
    credential=cred)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 123, in __init__
    self._cassandra_init(server_list)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 429, in _cassandra_init
    self._cassandra_init_conn_pools()
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 505, in _cassandra_init_conn_pools
    max_retries=30, timeout=5, credentials=self._credential)
  File "/usr/lib/python2.7/dist-packages/pycassa/pool.py", line 382, in __init__
    self.fill()
  File "/usr/lib/python2.7/dist-packages/pycassa/pool.py", line 442, in fill
    conn = self._create_connection()
  File "/usr/lib/python2.7/dist-packages/pycassa/pool.py", line 424, in _create_connection
    wrapper = self._get_new_wrapper(server)
  File "/usr/lib/python2.7/dist-packages/pycassa/pool.py", line 453, in _get_new_wrapper
    transport_factory=self.transport_factory)
  File "/usr/lib/python2.7/dist-packages/pycassa/pool.py", line 56, in __init__
    Connection.__init__(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pycassa/connection.py", line 57, in __init__
    self.set_keyspace(keyspace)
  File "/usr/lib/python2.7/dist-packages/pycassa/connection.py", line 61, in set_keyspace
    Cassandra.Client.set_keyspace(self, keyspace)
  File "/usr/lib/python2.7/dist-packages/pycassa/cassandra/Cassandra.py", line 499, in set_keyspace
    self.recv_set_keyspace()
  File "/usr/lib/python2.7/dist-packages/pycassa/cassandra/Cassandra.py", line 520, in recv_set_keyspace

   raise result.ire
InvalidRequestException: InvalidRequestException(why="Keyspace 'svc_monitor_keyspace' does not exist")

svc-monitor.log
-----------------

07/21/2016 08:33:34 PM [contrail-svc-monitor]: Starting Introspect on HTTP Port 8088
07/21/2016 08:33:34 PM [contrail-svc-monitor]: Cannot write http_port 8088 to /tmp/contrail-svc-monitor.2931.http_port
07/21/2016 08:33:34 PM [contrail-svc-monitor]: SANDESH: [DROP: NoSession] __default__ [SYS_WARN]: SvcMonitorLog: Warning! InvalidRequestException(why='Cannot add already existing table "service_instance_table" to keyspace "svc_monitor_keyspace"')
07/21/2016 08:33:34 PM [contrail-svc-monitor]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodeg21 process_status = [ << module_id = contrail-svc-monitor instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.204.217.61:2181, 10.204.217.98:2181, 10.204.217.73:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.204.217.61:9160, 10.204.217.98:9160, 10.204.217.73:9160, ] status = Initializing description = >>, << type = Discovery name = Collector server_addrs = [ 10.204.217.176:5998, ] status = Up description = Subscribe Response >>, ] description = Collector, Database:Cassandra[] connection down >>, ] >>
07/21/2016 08:33:34 PM [contrail-svc-monitor]: SANDESH: [DROP: NoSession] __default__ [SYS_WARN]: SvcMonitorLog: Warning! InvalidRequestException(why='Cannot add already existing table "pool_table" to keyspace "svc_monitor_keyspace"')

This is seen newly from R3.1 build 4 along with already existing bug https://bugs.launchpad.net/juniperopenstack/+bug/1603337 SM:3.1 : contrail-discovery failed in one of the config node

[sundarkh (sundar-kh)]

Issue not seen in mitaka build 6

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager backup
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor backup
ifmap active

[Abhay Joshi (abhayj)]

Sundar could not verify kilo & liberty with build 6 as that build was broken for kilo, liberty using SM. He is going to test with kilo & liberty with build 7 and update. If all are working, bug can be closed.

[sundarkh (sundar-kh)]

Issue not seen with build 12. Kilo/Liberty/mitaka