agent crash at FlowTable::DeleteVmFlowInfo during sanity run on build 108

[Vedamurthy Joshi]

R2.20 Ubuntu 12.04 Icehouse

Core is in /cs-shared/test_runs/nodea12/2015_11_10_20_22_13/nodei13_core.contrail-vroute.2772.nodei13.1447178803.gz

Build : 2.22-108
CoreLocation : /cs-shared/test_runs/nodea12/2015_11_10_20_22_13
cores : {'10.204.217.192': ['core.contrail-vroute.2772.nodei13.1447178803'], '10.204.217.127': ['core.contrail-vroute.21994.nodei15.1447183956', 'core.contrail-vroute.23177.nodei15.1447184180']}
LogsLocation : http://10.204.216.50/Docs/logs/2.22-108_2015_11_10_20_22_13/logs/
Report : http://10.204.216.50/Docs/logs/2.22-108_2015_11_10_20_22_13/junit-noframes.html
Topology :
Config Nodes : [u'nodea12', u'nodea15']
Control Nodes : [u'nodea15', u'nodea17']
Compute Nodes : [u'nodei13', u'nodei14', u'nodei15']
Openstack Node : nodea12
WebUI Node : nodea12
Analytics Nodes : [u'nodea12']

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal 6, Aborted.
#0 0x00007f4a014170d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#0 0x00007f4a014170d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4a0141a83b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f4a0140fd9e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f4a0140fe42 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000b4338e in boost::intrusive_ptr<FlowEntry>::~intrusive_ptr() ()
#5 0x0000000000b2f749 in FlowTable::DeleteVmFlowInfo(FlowEntry*, VmEntry const*) ()
#6 0x0000000000b35552 in FlowTable::DeleteFlowInfo(FlowEntry*) ()
#7 0x0000000000b3c765 in FlowTable::Allocate(FlowKey const&) ()
#8 0x0000000000b53a9b in PktFlowInfo::Add(PktInfo const*, PktControlInfo*, PktControlInfo*) ()
#9 0x0000000000b625f0 in FlowHandler::Run() ()
#10 0x0000000000b5f3c4 in Proto::ProcessProto(boost::shared_ptr<PktInfo>) ()
#11 0x0000000000b5ff62 in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, Proto, boost::shared_ptr<PktInfo> >, boost::_bi::list2<boost::_bi::value<Proto*>, boost::arg<1> > >, bool, boost::shared_ptr<PktInfo> >::invoke(boost::detail::function::function_buffer&, boost::shared_ptr<PktInfo>) ()
#12 0x0000000000b612d9 in QueueTaskRunner<boost::shared_ptr<PktInfo>, WorkQueue<boost::shared_ptr<PktInfo> > >::RunQueue() ()
#13 0x0000000001088ba5 in TaskImpl::execute() ()
#14 0x00007f4a01fcfe52 in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f4a01fcbc2d in ?? () from /usr/lib/libtbb.so.2
#16 0x00007f4a01fcb0db in ?? () from /usr/lib/libtbb.so.2
#17 0x00007f4a01fc8c1f in ?? () from /usr/lib/libtbb.so.2
#18 0x00007f4a01fc8e59 in ?? () from /usr/lib/libtbb.so.2
#19 0x00007f4a021e6e9a in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#20 0x00007f4a014d438d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#21 0x0000000000000000 in ?? ()

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/15177
Submitter: Naveen N (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/15177
Committed: http://github.org/Juniper/contrail-controller/commit/6f1e6ac50a08053929f0f0e0f85b4d208be46f9a
Submitter: Zuul
Branch: R2.20

commit 6f1e6ac50a08053929f0f0e0f85b4d208be46f9a
Author: Naveen N <email address hidden>
Date: Tue Nov 17 12:15:25 2015 +0530

* Enqueue request to delete short flow from flow stats collector
All flow table delete and add operation happens in FlowTable context
in case of short flow were getting deleted directly resulting in
flow table getting modified in multiple context. Correcting the same.
Closes-bug:#1515260

Change-Id: I8a69266d8f2f44c1036c809dd2284be0343c8f7c

[Vinod Nair (vinodnair)]

This is seem in Mainline Kilo Build 2713

Core : /cs-shared/bugs/1515260
Backtrace is a below

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f009ce3848c in std::string::assign(std::string const&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
    from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
(gdb) bt
#0 0x00007f009ce3848c in std::string::assign(std::string const&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1 0x0000000000c0e0b5 in operator= (__str=..., this=0x7f008aaf1e08) at /usr/include/c++/4.8/bits/basic_string.h:547
#2 FlowEntry::GetDestRouteInfo (this=this@entry=0x7f008aaf1d90, rt=0x7f00801741f0) at controller/src/vnsw/agent/pkt/flow_entry.cc:747
#3 0x0000000000c0e474 in FlowEntry::InitFwdFlow (this=this@entry=0x7f008aaf1d90, info=info@entry=0x7f007fffe6b0, pkt=pkt@entry=0x1ae26a0,
    ctrl=ctrl@entry=0x7f007fffe630, rev_ctrl=rev_ctrl@entry=0x7f007fffe670, rflow=rflow@entry=0x7f008aaf2230, agent=0x1ac2150)
    at controller/src/vnsw/agent/pkt/flow_entry.cc:518
#4 0x0000000000c37830 in PktFlowInfo::Add (this=this@entry=0x7f007fffe6b0, pkt=0x1ae26a0, in=in@entry=0x7f007fffe630, out=out@entry=0x7f007fffe670)
    at controller/src/vnsw/agent/pkt/pkt_flow_info.cc:1708
#5 0x0000000000c4575e in FlowHandler::Run (this=0x7f00780474b0) at controller/src/vnsw/agent/pkt/flow_handler.cc:131
#6 0x0000000000c41d28 in RunProtoHandler (handler=0x7f00780474b0, this=0x7f007fffe880) at controller/src/vnsw/agent/pkt/proto.cc:51
#7 Proto::ProcessProto (this=this@entry=0x7f0088eb6d20, msg_info=...) at controller/src/vnsw/agent/pkt/proto.cc:67
#8 0x0000000000c2ba72 in FlowProto::FlowEventHandler (this=0x7f0088eb6d20, req=..., table=0x7f008aaf1340) at controller/src/vnsw/agent/pkt/flow_proto.cc:263
#9 0x0000000000c30df8 in operator() (a0=<error reading variable: access outside bounds of object referenced via synthetic pointer>, this=0x7f007fffe900)
    at /usr/include/boost/function/function_template.hpp:767
#10 QueueTaskRunner<FlowEvent, WorkQueue<FlowEvent> >::RunQueue (this=0x1adc900) at controller/src/base/queue_task.h:87
#11 0x0000000001176eac in TaskImpl::execute (this=0x7f0095d2fb40) at controller/src/base/task.cc:253
#12 0x00007f009d0a1b3a in ?? () from /usr/lib/libtbb.so.2
#13 0x00007f009d09d816 in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f009d09cf4b in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f009d0990ff in ?? () from /usr/lib/libtbb.so.2
#16 0x00007f009d0992f9 in ?? () from /usr/lib/libtbb.so.2
#17 0x00007f009d2bd182 in start_thread (arg=0x7f007ffff700) at pthread_create.c:312
#18 0x00007f009c59647d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[Hari Prasad Killi (haripk)]

The new backtrace added above is same as https://bugs.launchpad.net/juniperopenstack/+bug/1545545, which is now fixed.