2.21-104-Issue with discovery service soon after installation

[Sandeep Sridhar]

contrail-status report "Collector Connection Down" though contrail-collector is up.

This happened multiple times soon after installation. For demo purposes, we have installed all-in-one setup on a VM.

root@Ubuntu12DeviceManager:/var/log/contrail# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent initializing (Collector connection down)
contrail-vrouter-nodemgr active

== Contrail Control ==
supervisor-control: active
contrail-control initializing (Number of connections:4, Expected:5)
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==
supervisor-config: active
contrail-api:0 initializing (Collector connection down)
contrail-config-nodemgr active
contrail-device-manager backup
contrail-discovery:0 active
contrail-schema backup
contrail-svc-monitor initializing
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
supervisor-database: active
contrail-database active
contrail-database-nodemgr active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

contrail-collector.log has the following :

root@Ubuntu12DeviceManager:/var/log/contrail# tailf contrail-collector.log
2016-02-03 Wed 11:13:09:079.407 IST Ubuntu12DeviceManager [Thread 140442667783936, Pid 2128]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 127.0.0.1:8089::127.0.0.1:55493 < Read failed due to error 2 : End of file controller/src/io/tcp_session.cc 432
2016-02-03 Wed 11:13:23:973.192 IST Ubuntu12DeviceManager [Thread 140442638395136, Pid 2128]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 127.0.0.1:8089::127.0.0.1:55662 < Read failed due to error 2 : End of file controller/src/io/tcp_session.cc 432
2016-02-03 Wed 11:13:28:161.536 IST Ubuntu12DeviceManager [Thread 140442663585536, Pid 2128]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 127.0.0.1:8089::127.0.0.1:55722 < Read failed due to error 2 : End of file controller/src/io/tcp_session.cc 432
2016-02-03 Wed 11:13:30:445.115 IST Ubuntu12DeviceManager [Thread 140442642593536, Pid 2128]: Connection to DB Established/Re-Established
2016-02-03 Wed 11:15:48:608.565 IST Ubuntu12DeviceManager [Thread 140442646791936, Pid 2128]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 127.0.0.1:8089::127.0.0.1:59719 < Read failed due to error 2 : End of file controller/src/io/tcp_session.cc 432
2016-02-03 Wed 11:16:14:899.917 IST Ubuntu12DeviceManager [Thread 140442667783936, Pid 2128]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 127.0.0.1:8089::127.0.0.1:60756 < Read failed due to error 2 : End of file controller/src/io/tcp_session.cc 432
2016-02-03 Wed 11:19:01:203.968 IST Ubuntu12DeviceManager [Thread 140442646791936, Pid 2128]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 127.0.0.1:8089::127.0.0.1:39495 < Read failed due to error 2 : End of file controller/src/io/tcp_session.cc 432
2016-02-03 Wed 11:33:28:553.337 IST Ubuntu12DeviceManager [Thread 140442646791936, Pid 2128]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 127.0.0.1:8089::127.0.0.1:44810 < Read failed due to error 2 : End of file controller/src/io/tcp_session.cc 432
2016-02-03 Wed 12:03:48:053.135 IST Ubuntu12DeviceManager [Thread 140442667783936, Pid 2128]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 127.0.0.1:8089::127.0.0.1:50240 < Read failed due to error 2 : End of file controller/src/io/tcp_session.cc 432
2016-02-03 Wed 12:05:29:242.500 IST Ubuntu12DeviceManager [Thread 140442663585536, Pid 2128]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 127.0.0.1:8089::127.0.0.1:53368 < Read failed due to error 2 : End of file controller/src/io/tcp_session.cc 432

Access Details : 10.204.74.103 ( root/contrail admin/contrail123)

Greetings,
Sandeep.

[Sandeep Sridhar (ssandeep)]

Tried a different build on a physical server - single box installation. It looks like it has the same problem. I am not sure what is going wrong as I have done this in the past without any issues whatsoever.

Another server on which the same problem is seen - 10.204.74.160 ( root/contrail admin/contrail123 ) - 2.21.1 - Build 22

Please let us know as it is blocking us from working on other issues from customers.

Greetings,
Sandeep.

[Raj Reddy (rajreddy)]

The discovery showed all services down in this state.. The discovery logs didn't have much info..

Home | Publishers | Subscribers | Stats | Config | Cleanup |
Service Type Remote IP Service Id Oper State Admin State In Use Time since last Heartbeat
OpServer 10.204.74.103 Ubuntu12DeviceManager:OpServer up up 1 -1 day, 19:53:33
IfmapServer 10.204.74.103 Ubuntu12DeviceManager:IfmapServer up up 2 -1 day, 19:53:31
dns-server 10.204.74.103 Ubuntu12DeviceManager:dns-server up up 0 -1 day, 19:53:34
xmpp-server 10.204.74.103 Ubuntu12DeviceManager:xmpp-server up up 0 -1 day, 19:53:35
Collector 10.204.74.103 Ubuntu12DeviceManager:Collector up up 10 -1 day, 19:53:31
ApiServer 10.204.74.103 Ubuntu12DeviceManager:ApiServer up up 1 -1 day, 19:53:31