R3.0.3 build61 - neutron server fails to restart on centos7.1 juno during setup_vnc_config

[Sudheendra Rao]

Neutron server fails to restart during setup_vnc_config on R3.0.3 build61 centos7.1 juno on contrail-HA setup.

=====
2016-09-20 09:17:27:965576: [root@10.204.217.6] out: + service neutron-server restart
2016-09-20 09:17:27:968989: [root@10.204.217.6] out: Redirecting to /bin/systemctl restart neutron-server.service
2016-09-20 09:17:27:984528: [root@10.204.217.6] out: Job for neutron-server.service failed. See 'systemctl status neutron-server.service' and 'journalctl -xn' for details.
2016-09-20 09:18:57:993523: [root@10.204.217.6] out:
2016-09-20 09:18:58:001143: [root@10.204.217.6] out: Fatal error: local() encountered an error (return code 1) while executing 'sudo quantum-server-setup.sh'
2016-09-20 09:18:58:001319: [root@10.204.217.6] out:
2016-09-20 09:18:58:001408: [root@10.204.217.6] out: Aborting.
2016-09-20 09:18:58:001489: [root@10.204.217.6] out:
2016-09-20 09:18:58:009179:

2016-09-20 09:18:58:023232: Fatal error: sudo() received nonzero return code 1 while executing!
2016-09-20 09:18:58:023232:
2016-09-20 09:18:58:023232: Requested: setup-vnc-config --self_ip 192.168.10.4 --cfgm_index 1 --collector_ip 192.168.10.4 --cassandra_ip_list 192.168.10.4 192.168.10.5 192.168.10.6 --zookeeper_ip_list 192.168.10.4 192.168.10.5 192.168.10.6 --control_ip_list 192.168.10.4 192.168.10.6 --quantum_port 9697 --nworkers 1 --service_token 5a3df3b7b4cf3e14b5ef --amqp_ip_list 192.168.10.4 192.168.10.5 192.168.10.6 --amqp_port 5672 --orchestrator openstack --seed_list 192.168.10.4,192.168.10.5 --data_dir /var/lib/cassandra/data --aaa_mode cloud-admin --keystone_ip 192.168.10.4 --keystone_version v2.0 --keystone_admin_passwd contrail123 --keystone_service_tenant_name service --neutron_password contrail123 --keystone_auth_protocol http --keystone_auth_port 35357 --keystone_insecure False --region_name RegionOne --contrail_internal_vip 192.168.10.7
2016-09-20 09:18:58:023232: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "cd /opt/contrail/bin && setup-vnc-config --self_ip 192.168.10.4 --cfgm_index 1 --collector_ip 192.168.10.4 --cassandra_ip_list 192.168.10.4 192.168.10.5 192.168.10.6 --zookeeper_ip_list 192.168.10.4 192.168.10.5 192.168.10.6 --control_ip_list 192.168.10.4 192.168.10.6 --quantum_port 9697 --nworkers 1 --service_token 5a3df3b7b4cf3e14b5ef --amqp_ip_list 192.168.10.4 192.168.10.5 192.168.10.6 --amqp_port 5672 --orchestrator openstack --seed_list 192.168.10.4,192.168.10.5 --data_dir /var/lib/cassandra/data --aaa_mode cloud-admin --keystone_ip 192.168.10.4 --keystone_version v2.0 --keystone_admin_passwd contrail123 --keystone_service_tenant_name service --neutron_password contrail123 --keystone_auth_protocol http --keystone_auth_port 35357 --keystone_insecure False --region_name RegionOne --contrail_internal_vip 192.168.10.7"
2016-09-20 09:18:58:023232:
2016-09-20 09:18:58:023309: Aborting.
====

Don't see much info in the neutron log:
2016-09-20 10:22:23.868 INFO [neutron.manager] Service L3_ROUTER_NAT is supported by the core plugin
2016-09-20 10:22:23.868 INFO [neutron.manager] Loading Plugin: neutron_plugin_contrail.plugins.opencontrail.loadbalancer.plugin.LoadBalancerPlugin
2016-09-20 10:22:53.5 WARNING [vnc_api.vnc_api] Exception: Service unavailable time out due to: Service Unavailable Timeout 503
2016-09-20 10:23:22.96 WARNING [vnc_api.vnc_api] Exception: Service unavailable time out due to: Service Unavailable Timeout 503
2016-09-20 10:23:51.193 WARNING [vnc_api.vnc_api] Exception: Service unavailable time out due to: Service Unavailable Timeout 503

logs in contrail-api: (seems like zookeeper connection is also down)

09/20/2016 09:17:33 AM [contrail-api]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodec21.englab.juniper.net process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Discovery name = Collector server_addrs = [ 192.168.10.7:5998, ] status = Down description = Subscribe - Status Code 503 >>, << type = IFMap name = IfMap server_addrs = [ 192.168.10.4:8443, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 192.168.10.4:2181, 192.168.10.5:2181, 192.168.10.6:2181, ] status = Down description = >>, ] description = Collector, Discovery:Collector[Subscribe - Status Code 503], Zookeeper:Zookeeper[] connection down >>, ] >>
09/20/2016 09:18:57 AM [contrail-api]: Connect error to contrail api 100 tries:
09/20/2016 09:20:38 AM [contrail-api]: Connect error to contrail api 200 tries:
09/20/2016 09:22:19 AM [contrail-api]: Connect error to contrail api 300 tries:

Setup:
env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6],
    'cfgm': [host1, host2, host3],
    'openstack': [host1],
    'webui': [host3],
    'control': [host1, host3],
    'compute': [host4, host5, host6],
    'collector': [host1, host2, host3],
    'database': [host1, host2, host3],
    'build': [host_build],
}
env.hostnames = {
    'all': ['nodec21.englab.juniper.net', 'nodec19.englab.juniper.net', 'nodec20.englab.juniper.net', 'nodec61.englab.juniper.net', 'nodec60.englab.juniper.net','nodec30.englab.juniper.net']
}
env.ha = {
    'contrail_internal_vip' : '192.168.10.7',
    'contrail_internal_virtual_router_id' : 200
}
ha_setup = True

[Ignatious Johnson Christopher (ijohnson-x)]

Need to back port changes to R3.0 from R3.0.2.x