TSN: Agent crash @ __TBB_load_with_acquire

[chhandak]

Observed Following Agent crash on TSN node

Backtrace
-----------------
ImportError: No module named 'libstdcxx'
(gdb) bt
#0 load_with_acquire (location=@0x1f8: <error reading variable>) at /usr/include/tbb/tbb_machine.h:613
#1 __TBB_load_with_acquire<int> (location=@0x1f8: <error reading variable>) at /usr/include/tbb/tbb_machine.h:708
#2 operator tbb::internal::atomic_impl<unsigned int>::value_type (this=0x1f8) at /usr/include/tbb/atomic.h:306
#3 flow_export_msg_drops (this=0x0) at controller/src/vnsw/agent/vrouter/flow_stats/flow_stats_manager.h:133
#4 AgentStatsReq::HandleRequest (this=0x7f37f962d510) at controller/src/vnsw/agent/cmn/agent_stats.cc:69
#5 0x00000000010bfc7d in Sandesh::ProcessRecv (rsnh=0x7f37f962d510) at tools/sandesh/library/cpp/sandesh.cc:492
#6 0x00000000010cc63f in operator() (a0=0x7f37f962d510, this=0x7f3806ffab30)
    at /usr/include/boost/function/function_template.hpp:767
#7 RunQueue (this=0x7f37f85a9df0) at controller/src/base/queue_task.h:87
#8 QueueTaskRunner<SandeshRequest*, WorkQueue<SandeshRequest*> >::Run (this=0x7f37f85a9df0)
    at controller/src/base/queue_task.h:66
#9 0x0000000001186b3c in TaskImpl::execute (this=0x7f380cadfe40) at controller/src/base/task.cc:253
#10 0x00007f3813e8eb3a in ?? () from /usr/lib/libtbb.so.2
#11 0x00007f3813e8a816 in ?? () from /usr/lib/libtbb.so.2
#12 0x00007f3813e89f4b in ?? () from /usr/lib/libtbb.so.2
#13 0x00007f3813e860ff in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f3813e862f9 in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f38140aa182 in start_thread (arg=0x7f3806ffb700) at pthread_create.c:312
#16 0x00007f381338347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

root@5b7s4:~# contrail-version | grep vrouter
contrail-vrouter-3.13.0-40-generic 3.0.2.0-26 26
contrail-vrouter-agent 3.0.2.0-26 26
contrail-vrouter-common 3.0.2.0-26 26
contrail-vrouter-init 3.0.2.0-26 26
contrail-vrouter-utils 3.0.2.0-26 26
python-contrail-vrouter-api 3.0.2.0-26 26
python-opencontrail-vrouter-netns 3.0.2.0-26 26
root@5b7s4:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-tor-agent-1 active
contrail-tor-agent-2 active
contrail-vrouter-agent active
contrail-vrouter-nodemgr active

========Run time service failures=============
/var/crashes/core.contrail-vroute.1180.5b7s4.1461018723

[chhandak (chhandak)]

Core file copied in

/cs-shared/bugs/1571888
chhandak@ubuntu-build04:/cs-shared/bugs/1571888$ ls -lrt
total 633076
-rwxrwxrwx 1 chhandak epbg 645722112 Apr 18 23:50 core.contrail-vroute.1180.5b7s4.1461018723
chhandak@ubuntu-build04:/cs-shared/bugs/1571888$

[Ashok Singh (ashoksr)]

mainline commit https://github.com/Juniper/contrail-controller/commit/942c54320a3db8f63e61fd8870fbc3b9d0396f46
R3.0 commit http://github.org/Juniper/contrail-controller/commit/7ad9a9a1a4a17164168ba9a7f049a66acbd33f51