Vcenter: after reboot of controllers, vcenter-plugin contrail-status failed

[Sarath]

This is Vcenter-only-mode. With 3 controllers setup, when I rebooted all of them i see on one of controller vcenter-plugin status shows failed.
This looks CLI-only issue because confirmed the operation works good when moving VN etc seeing the traces.

root@oblocknode02:~# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager backup
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor failed
contrail-vcenter-plugin failed
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
contrail-database: active
supervisor-database: active
contrail-database-nodemgr active
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

>>> above is cli-only issue as below shows correctly,
root@oblocknode02:~#
root@oblocknode02:~# service contrail-vcenter-plugin status
contrail-vcenter-plugin start/running, process 1923
root@oblocknode02:~#

>>> it is cli-only issue because when active controller failed, i see this controller takes-over and responding to VN changes

root@oblocknode02:/var/log/contrail# ^C
root@oblocknode02:/var/log/contrail# tail -f contrail-vcenter-plugin.log
[2016-03-07 16:51:59,937] [INFO ] [Thread-3:VCenterNotify@317] [+++++++++++++Received vcenter update of type enter+++++++++++++]
[2016-03-07 16:51:59,937] [INFO ] [Thread-3:VCenterNotify@431] [+++++++++++++Update Processing Complete +++++++++++++++++++++]
[2016-03-07 16:51:59,937] [INFO ] [Thread-3:VCenterNotify@317] [+++++++++++++Received vcenter update of type enter+++++++++++++]
[2016-03-07 16:51:59,937] [INFO ] [Thread-3:VCenterNotify@431] [+++++++++++++Update Processing Complete +++++++++++++++++++++]
[2016-03-07 16:51:59,937] [INFO ] [Thread-3:VCenterNotify@317] [+++++++++++++Received vcenter update of type enter+++++++++++++]
[2016-03-07 16:51:59,937] [INFO ] [Thread-3:VCenterNotify@431] [+++++++++++++Update Processing Complete +++++++++++++++++++++]
[2016-03-07 16:51:59,937] [INFO ] [Thread-3:VCenterNotify@317] [+++++++++++++Received vcenter update of type enter+++++++++++++]
[2016-03-07 16:51:59,937] [INFO ] [Thread-3:VCenterNotify@431] [+++++++++++++Update Processing Complete +++++++++++++++++++++]
[2016-03-07 16:51:59,937] [INFO ] [Thread-3:VCenterNotify@317] [+++++++++++++Received vcenter update of type enter+++++++

root@oblocknode02:~#
root@oblocknode02:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.0.0.0-2723 2723
contrail-config 3.0.0.0-2723 2723
contrail-control 3.0.0.0-2723 2723
contrail-dns 3.0.0.0-2723 2723
contrail-docs 3.0.0.0-2723 2723
contrail-f5 3.0.0.0-2723 2723
contrail-fabric-utils 3.0.0.0-2723 2723
contrail-install-packages 3.0.0.0-2723~vcenter 2723
contrail-install-vcenter-plugin 3.0.0.0-03012016 2723
contrail-lib 3.0.0.0-2723 2723
contrail-nodemgr 3.0.0.0-2723 2723
contrail-openstack-analytics 3.0.0.0-2723 2723
contrail-openstack-control 3.0.0.0-2723 2723
contrail-openstack-database 3.0.0.0-2723 2723
contrail-openstack-webui 3.0.0.0-2723 2723
contrail-setup 3.0.0.0-2723 2723
contrail-utils 3.0.0.0-2723 2723
contrail-vmware-config 3.0.0.0-2723 2723
contrail-vmware-utils 3.0.0.0-2723 2723
contrail-web-controller 3.0.0.0-2723 2723
contrail-web-core 3.0.0.0-2723 2723
ifmap-python-client 0.1-2 2723
ifmap-server 0.3.2-1contrail2 2723

python-contrail 3.0.0.0-2723 2723
root@oblocknode02:~#
root@oblocknode02:~#
root@oblocknode02:~#

[Sarath (nsarath)]

-bash-4.1$
-bash-4.1$ pwd
/cs-shared/bugs/1554289
-bash-4.1$
-bash-4.1$ ls -l
total 1959420
-rwxrwxrwx 1 nsarath test 1061703680 Mar 7 17:49 Ctrl-A-log.tar*
-rwxrwxrwx 1 nsarath test 426106880 Mar 7 17:49 Ctrl-B-log.tar*
-rwxrwxrwx 1 nsarath test 444057600 Mar 7 17:49 Ctrl-C-log.tar*
-rwxrwxrwx 1 nsarath test 9297920 Mar 7 17:48 Vrtr-0-log.tar*
-rwxrwxrwx 1 nsarath test 10209280 Mar 7 17:48 Vrtr-1-log.tar*
-rwxrwxrwx 1 nsarath test 9267200 Mar 7 17:48 Vrtr-3-log.tar*
-rwxrwxrwx 1 nsarath test 9584640 Mar 7 17:48 Vrtr-4-log.tar*
-rwxrwxrwx 1 nsarath test 9512960 Mar 7 17:48 Vrtr-5-log.tar*
-rwxrwxrwx 1 nsarath test 9390080 Mar 7 17:48 Vrtr-7-log.tar*
-rwxrwxrwx 1 nsarath test 9390080 Mar 7 17:48 Vrtr-8-log.tar*
-bash-4.1$

[Sachchidanand Vaidya (vaidyasd)]

IS this still an issue. We have moved contrail-vcenter-plugin out of config status and as independent service in 3.0

[Sachchidanand Vaidya (vaidyasd)]

IS this still an issue. We have moved contrail-vcenter-plugin out of config status and as independent service.

[Sarath (nsarath)]

Sure Sachin. Let me verify with latest and update

[Sarath (nsarath)]

Unreproducible during 3.1 testing.