TOR Agent goes into initializing state for both TSNs

[Anoop Kumar Sahu]

There has been numerous instances where the TOR agent for a particular TOR goes into initializing state on both the TSNs.

Steps to reproduce:
a) RE switchover on a TOR with scaled VXLAN config (3K VNI, 100,000+ MACs)
b) The TA goes into initializing state for 10-15 minutes and recovers afterwards

root@ubuntu-bms-tsn:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-tor-agent-1 initializing (ToR:vdc-vcf-s1 connection
down)
contrail-tor-agent-2 active
contrail-tor-agent-3 initializing (ToR:vdc-vcf-s4 connection
down)
contrail-tor-agent-4 initializing (ToR:vdc-vcf-l1 connection
down)
contrail-tor-agent-5 initializing (ToR:vdc-vcf-l2 connection
down)
contrail-tor-agent-6 active
contrail-tor-agent-7 active
contrail-vrouter-agent active
contrail-vrouter-nodemgr active

root@ubuntu-tsn:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-tor-agent-1 initializing (ToR:vdc-vcf-s1 connection
down)
contrail-tor-agent-2 initializing (ToR:vdc-vcf-s2 connection
down)
contrail-tor-agent-3 active
contrail-tor-agent-4 active
contrail-tor-agent-5 active
contrail-tor-agent-6 initializing (ToR:vdc-core-sw2-re0
connection down)
contrail-tor-agent-7 initializing (ToR:vdc-vcf-l7 connection
down)
contrail-vrouter-agent active
contrail-vrouter-nodemgr active

[Hari Prasad Killi (haripk)]

Keep-alive time on tor-agent is configured as 10s. In scale configurations, we see that OVSDB server is busy when new connection is done and tor-agent doesn't receive any packet / keep alive response. On 10s expiry, connection is deleted and new connection is done. This happens repeatedly due to which connection is not setup.

Workaround is to increase the keepalive timeout on Contrail to a higher value (in /etc/contrail/contrail-tor-agent-<id>.conf) and restart tor-agent.