Vcenter-only: svc-monitor keep flapping Active state due to InvalidRequestException

[Sarath]

svc-monitor "active" state keep flapping due to exception and seen below related on svc-monitor logs,

>> svc-monitor logs

07/11/2016 12:11:15 PM [contrail-svc-monitor]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = oblocknode02 process_status = [ << module_id = contrail-svc-monitor instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 172.16.80.2:2181, 172.16.80.13:2181, 172.16.80.4:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 172.16.80.2:9160, 172.16.80.13:9160, 172.16.80.4:9160, ] status = Initializing description = >>, << type = Discovery name = Collector server_addrs = [ 172.16.80.2:5998, ] status = Initializing description = Subscribe >>, ] description = Collector, Database:Cassandra[], Discovery:Collector[Subscribe] connection down >>, ] >>
07/11/2016 12:11:15 PM [contrail-svc-monitor]: Starting Introspect on HTTP Port 8088
07/11/2016 12:11:15 PM [contrail-svc-monitor]: Cannot write http_port 8088 to /tmp/contrail-svc-monitor.1896.http_port
07/11/2016 12:11:15 PM [contrail-svc-monitor]: SANDESH: [DROP: NoSession] __default__ [SYS_WARN]: SvcMonitorLog: Warning! InvalidRequestException(why='Cannot add already existing table "service_instance_table" to keyspace "svc_monitor_keyspace"')
07/11/2016 12:11:15 PM [contrail-svc-monitor]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = oblocknode02 process_status = [ << module_id = contrail-svc-monitor instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 172.16.80.2:2181, 172.16.80.13:2181, 172.16.80.4:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 172.16.80.2:9160, 172.16.80.13:9160, 172.16.80.4:9160, ] status = Initializing description = >>, << type = Discovery name = Collector server_addrs = [ 172.16.80.2:5998, ] status = Up description = Subscribe Response >>, ] description = Collector, Database:Cassandra[] connection down >>, ] >>
07/11/2016 12:11:15 PM [contrail-svc-monitor]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = oblocknode02 process_status = [ << module_id = contrail-svc-monitor instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 172.16.80.2:2181, 172.16.80.13:2181, 172.16.80.4:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ 172.16.80.13:8086, ] status = Initializing description = Idle to Connect on EvCollectorChange >>, << type = Database name = Cassandra server_addrs = [ 172.16.80.2:9160, 172.16.80.13:9160, 172.16.80.4:9160, ] status = Initializing description = >>, << type = Discovery name = Collector server_addrs = [ 172.16.80.2:5998, ] status = Up description = Subscribe Response >>, ] description = Collector, Database:Cassandra[] connection down >>, ] >>
07/11/2016 12:11:15 PM [contrail-svc-monitor]: __default__ [SYS_WARN]: SvcMonitorLog: Warning! InvalidRequestException(why='Cannot add already existing table "pool_table" to keyspace "svc_monitor_keyspace"')
07/11/2016 12:11:16 PM [contrail-svc-monitor]: __default__ [SYS_NOTICE]: SvcMonitorLog: Cassandra connection ESTABLISHED
07/11/2016 12:11:16 PM [contrail-svc-monitor]: __default__ [SYS_NOTICE]: SvcMonitorLog: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@172.16.80.2:5672/', 'pyamqp://guest:guest@172.16.80.13:5672/', 'pyamqp://guest:guest@172.16.80.4:5672/']
07/11/2016 12:11:16 PM [contrail-svc-monitor]: __default__ [SYS_NOTICE]: SvcMonitorLog: RabbitMQ connection down
07/11/2016 12:11:16 PM [contrail-svc-monitor]: __default__ [SYS_NOTICE]: SvcMonitorLog: RabbitMQ connection ESTABLISHED <Connection: amqp://guest:**@172.16.80.2:5672// at 0x7f79bc8a8f90>
^C

root@oblocknode02:/var/log/contrail#
root@oblocknode02:/var/log/contrail#
root@oblocknode02:/var/log/contrail# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager backup
contrail-discovery:0 active
contrail-schema backup
contrail-svc-monitor backup
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
contrail-database: active

== Contrail Supervisor Database ==
supervisor-database: active
contrail-database-nodemgr active
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

root@oblocknode02:/var/log/contrail#

root@oblocknode02:/var/log/contrail#
root@oblocknode02:/var/log/contrail# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.1.0.0-2740 2740
contrail-config 3.1.0.0-2740 2740
contrail-config-vmware 3.1.0.0-2740 2740
contrail-control 3.1.0.0-2740 2740
contrail-database-common 3.1.0.0-2740 2740
contrail-dns 3.1.0.0-2740 2740
contrail-docs 3.1.0.0-2740 2740
contrail-f5 3.1.0.0-2740 2740
contrail-fabric-utils 3.1.0.0-2740 2740
contrail-install-packages 3.1.0.0-2740~vcenter 2740
contrail-install-vcenter-plugin 3.1.0.0-07092016 2740
contrail-lib 3.1.0.0-2740 2740
contrail-nodemgr 3.1.0.0-2740 2740
contrail-openstack-analytics 3.1.0.0-2740 2740
contrail-openstack-control 3.1.0.0-2740 2740
contrail-openstack-database 3.1.0.0-2740 2740
contrail-openstack-webui 3.1.0.0-2740 2740
contrail-setup 3.1.0.0-2740 2740
contrail-utils 3.1.0.0-2740 2740
contrail-vmware-config 3.1.0.0-2740 2740
contrail-vmware-utils 3.1.0.0-2740 2740
contrail-web-controller 3.1.0.0-2740 2740
contrail-web-core 3.1.0.0-2740 2740
ifmap-python-client 0.1-2 2740
ifmap-server 0.3.2-1contrail2 2740
python-contrail 3.1.0.0-2740 2740
root@oblocknode02:/var/log/contrail#

[Sarath (nsarath)]

-bash-4.1$ pwd
/cs-shared/bugs/1602011
-bash-4.1$
-bash-4.1$ ls -l
total 1654692
-rwxrwxrwx 1 nsarath test 648929280 Jul 11 14:37 Ctrl-A-log.tar*
-rwxrwxrwx 1 nsarath test 504043520 Jul 11 14:38 Ctrl-B-log.tar*
-rwxrwxrwx 1 nsarath test 515758080 Jul 11 14:38 Ctrl-C-log.tar*
-rwxrwxrwx 1 nsarath test 3829760 Jul 11 14:38 Vrtr-0-log.tar*
-rwxrwxrwx 1 nsarath test 3870720 Jul 11 14:38 Vrtr-1-log.tar*
-rwxrwxrwx 1 nsarath test 3758080 Jul 11 14:39 Vrtr-3-log.tar*
-rwxrwxrwx 1 nsarath test 3778560 Jul 11 14:39 Vrtr-5-log.tar*
-rwxrwxrwx 1 nsarath test 3768320 Jul 11 14:39 Vrtr-7-log.tar*
-bash-4.1$

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21903
Submitter: Rudra Rugge (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21904
Submitter: Rudra Rugge (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21904
Committed: http://github.org/Juniper/contrail-controller/commit/d5b0966bad81207fc2c23a67b733e1625d075826
Submitter: Zuul
Branch: R3.1

commit d5b0966bad81207fc2c23a67b733e1625d075826
Author: Rudra Rugge <email address hidden>
Date: Tue Jul 12 14:39:08 2016 -0700

Handle auth token failure exception

Handle auth token failure exception

Change-Id: Ia028fce235c11032f42a9d2cf36338b99b8604ed
Closes-Bug: #1602011

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21903
Committed: http://github.org/Juniper/contrail-controller/commit/51a903a834a2d36d4605eb5b3051226b2d0021ac
Submitter: Zuul
Branch: master

commit 51a903a834a2d36d4605eb5b3051226b2d0021ac
Author: Rudra Rugge <email address hidden>
Date: Tue Jul 12 14:39:08 2016 -0700

Handle auth token failure exception

Handle auth token failure exception

Change-Id: Ia028fce235c11032f42a9d2cf36338b99b8604ed
Closes-Bug: #1602011