[4.0.0.0-2~mitaka] Tor Agent crash when configuring lif

[chhandak]

Tor Agent process crashing and going to failed state after creating lif.

Crash is observed with 4.0 Build 2 when using tor-agent and vrouter agent binary from Build 4 to avoid Bug 1689790

root@5b7s4:/usr/bin# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-tor-agent-1 failed
contrail-vrouter-agent active
contrail-vrouter-nodemgr active

========Run time service failures=============
/var/crashes/core.contrail-tor-ag.25809.5b7s4.1494657467
/var/crashes/core.contrail-tor-ag.28209.5b7s4.1494658892
/var/crashes/core.contrail-tor-ag.28234.5b7s4.1494658893
/var/crashes/core.contrail-tor-ag.25784.5b7s4.1494657466
/var/crashes/core.contrail-tor-ag.7161.5b7s4.1494657464
/var/crashes/core.contrail-tor-ag.25857.5b7s4.1494657472
/var/crashes/core.contrail-tor-ag.28272.5b7s4.1494658895
/var/crashes/core.contrail-tor-ag.28362.5b7s4.1494658899
/var/crashes/core.contrail-tor-ag.27878.5b7s4.1494658891
/var/crashes/core.contrail-tor-ag.25833.5b7s4.1494657469

Tor Defination in testbed file
---------------------------------
{host4:[{
                    'tor_ip':'172.18.90.1',
                    'tor_agent_id':'1',
                    'tor_type':'ovs',
                    'tor_ovs_port':'4321',
                    'tor_ovs_protocol':'pssl',
                    'tor_tsn_ip':'172.17.90.4',
                    'tor_tsn_name':'5b7s4',
                    'tor_name':'5b7-qfx2',
                    'tor_tunnel_ip':'34.34.34.34',
                    'tor_vendor_name':'Juniper',
                    'tor_product_name':'QFX5100',
                    'tor_agent_http_server_port': '1234',
                    'tor_agent_ovs_ka': '10000',
                       }

root@5b7s4:/usr/bin# ping 172.18.90.1
PING 172.18.90.1 (172.18.90.1) 56(84) bytes of data.
64 bytes from 172.18.90.1: icmp_seq=1 ttl=62 time=32.6 ms
64 bytes from 172.18.90.1: icmp_seq=2 ttl=62 time=4.98 ms
ç64 bytes from 172.18.90.1: icmp_seq=3 ttl=62 time=10.8 ms
^C
--- 172.18.90.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 4.980/16.130/32.602/11.887 ms
root@5b7s4:/usr/bin# ping 34.34.34.34
PING 34.34.34.34 (34.34.34.34) 56(84) bytes of data.
64 bytes from 34.34.34.34: icmp_seq=1 ttl=62 time=5.11 ms
64 bytes from 34.34.34.34: icmp_seq=2 ttl=62 time=10.8 ms

Backtrace:
------------
gdb) bt
#0 0x00007f6ef16c1404 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000082c4b4 in lock (this=0x20) at /usr/include/tbb/mutex.h:164
#2 acquire (mutex=..., this=<synthetic pointer>) at /usr/include/tbb/mutex.h:105
#3 scoped_lock (mutex=..., this=<synthetic pointer>) at /usr/include/tbb/mutex.h:91
#4 PortSubscribeTable::Get (this=0x0, vmi_uuid=..., vm_uuid=...) at controller/src/vnsw/agent/port_ipc/port_subscribe_table.cc:516
#5 0x0000000000aad80e in InterfaceTable::VmiProcessConfig (this=<optimized out>, node=0x7f6ed80167d0, req=..., u=...) at controller/src/vnsw/agent/oper/vm_interface_config.cc:1291
#6 0x00000000009e10b5 in ConfigManagerNodeList::Process (this=0x7f6ee40fa6c0, weight=weight@entry=57) at controller/src/vnsw/agent/oper/config_manager.cc:137
#7 0x00000000009dfaca in ConfigManager::Run (this=this@entry=0x223c760) at controller/src/vnsw/agent/oper/config_manager.cc:395
#8 0x00000000009dfbce in ConfigManager::TimerRun (this=0x223c760) at controller/src/vnsw/agent/oper/config_manager.cc:355
#9 0x000000000114c5c9 in operator() (this=<optimized out>) at /usr/include/boost/function/function_template.hpp:767
#10 Timer::TimerTask::Run (this=0x2245160) at controller/src/base/timer.cc:44
#11 0x00000000011442c7 in TaskImpl::execute (this=0x7f6eea0f3d40) at controller/src/base/task.cc:279
#12 0x00007f6ef14a3b3a in ?? () from /usr/lib/libtbb.so.2
#13 0x00007f6ef149f816 in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f6ef149ef4b in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f6ef149b0ff in ?? () from /usr/lib/libtbb.so.2
#16 0x00007f6ef149b2f9 in ?? () from /usr/lib/libtbb.so.2
#17 0x00007f6ef16bf184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#18 0x00007f6ef099837d in clone () from /lib/x86_64-linux-gnu/libc.so.6

#18 0x00007f6ef099837d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) quit
root@5b7s4:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-fabric-utils 4.0.0.0-2 2
contrail-install-packages 4.0.0.0-2~mitaka 2
contrail-lib 4.0.0.0-2 2
contrail-nodemgr 4.0.0.0-2 2
contrail-nova-vif 4.0.0.0-2 2
contrail-openstack-vrouter 4.0.0.0-2 2
contrail-setup 4.0.0.0-2 2
contrail-utils 4.0.0.0-2 2
^C
root@5b7s4:~# contrail-version | grep vrouter
contrail-openstack-vrouter 4.0.0.0-2 2
contrail-vrouter-agent 4.0.0.0-2 2
contrail-vrouter-common 4.0.0.0-2 2
contrail-vrouter-dkms 4.0.0.0-2 2
contrail-vrouter-init 4.0.0.0-2 2
contrail-vrouter-utils 4.0.0.0-2 2
python-contrail-vrouter-api 4.0.0.0-2 2
python-opencontrail-vrouter-netns 4.0.0.0-2 2

[chhandak (chhandak)]

Core copied to auto/cores/1690507

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31533
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31534
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31533
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31534
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31534
Committed: http://github.com/Juniper/contrail-controller/commit/689064750ca0c5b5111606cd33005f4e59bb5a3a
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 689064750ca0c5b5111606cd33005f4e59bb5a3a
Author: Hari Prasad Killi <email address hidden>
Date: Sat May 13 15:36:14 2017 +0530

Port IPC handler is not instantiated in case of TOR Agent.

Add check to see it is instantiated before use.

Change-Id: Ic0f655d2a18ca7f84ef55e6cd902bf487f012eea
closes-bug: #1690507

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31533
Committed: http://github.com/Juniper/contrail-controller/commit/6f216d93849bfeea9fd8286dd6ea29a80e0e593b
Submitter: Zuul (<email address hidden>)
Branch: master

commit 6f216d93849bfeea9fd8286dd6ea29a80e0e593b
Author: Hari Prasad Killi <email address hidden>
Date: Sat May 13 15:36:14 2017 +0530

Port IPC handler is not instantiated in case of TOR Agent.

Add check to see it is instantiated before use.

Change-Id: Ic0f655d2a18ca7f84ef55e6cd902bf487f012eea
closes-bug: #1690507