agent crash at FlowStatsCollector::FlowExport

[Vedamurthy Joshi]

R2.22 Build 111 Ubuntu 14.04 Juno

On nodek2, agent was thrashing. Took help from Praveen to run it via gdb , enable catch throw, and ran manually and we see the below backtrace.

[Switching to Thread 0x7fffd57f5700 (LWP 31519)]
Catchpoint 1 (exception thrown), 0x00007ffff5b7da30 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb)
(gdb)
(gdb) bt
#0 0x00007ffff5b7da30 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1 0x00007ffff5bcf5d7 in std::__throw_out_of_range(char const*) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x0000000000adb93a in at (__k=@0x7fffd0038c22: 14, this=<optimized out>) at /usr/include/c++/4.8/bits/stl_map.h:514
#3 FlowStatsCollector::FlowExport (this=this@entry=0x7fffbc001070, flow=flow@entry=0x7fffd0038a30, diff_bytes=diff_bytes@entry=162, diff_pkts=diff_pkts@entry=3)
    at controller/src/vnsw/agent/vrouter/flow_stats/flow_stats_collector.cc:435
#4 0x0000000000adbf84 in FlowStatsCollector::Run (this=0x7fffbc001070) at controller/src/vnsw/agent/vrouter/flow_stats/flow_stats_collector.cc:704
#5 0x0000000000adcf4a in StatsCollector::TimerExpiry (this=<optimized out>) at controller/src/vnsw/agent/uve/stats_collector.h:67
#6 0x0000000001021f59 in operator() (this=<optimized out>) at /usr/include/boost/function/function_template.hpp:767
#7 Timer::TimerTask::Run (this=0x16bacb0) at controller/src/base/timer.cc:42
#8 0x000000000101b950 in TaskImpl::execute (this=0x7fffeeacf340) at controller/src/base/task.cc:238
#9 0x00007ffff5e43b3a in ?? () from /usr/lib/libtbb.so.2
#10 0x00007ffff5e3f816 in ?? () from /usr/lib/libtbb.so.2
#11 0x00007ffff5e3ef4b in ?? () from /usr/lib/libtbb.so.2
#12 0x00007ffff5e3b0ff in ?? () from /usr/lib/libtbb.so.2
#13 0x00007ffff5e3b2f9 in ?? () from /usr/lib/libtbb.so.2
#14 0x00007ffff605f182 in start_thread (arg=0x7fffd57f5700) at pthread_create.c:312
#15 0x00007ffff533847d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

————

[Switching to Thread 0x7fff9e7f9700 (LWP 7468)]
Catchpoint 1 (exception thrown), 0x00007ffff5b7da30 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0 0x00007ffff5b7da30 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1 0x00007ffff5bcf5d7 in std::__throw_out_of_range(char const*) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x0000000000adb93a in at (__k=@0x7fff9800e162: 14, this=<optimized out>) at /usr/include/c++/4.8/bits/stl_map.h:514
#3 FlowStatsCollector::FlowExport (this=this@entry=0x7fff40000a80, flow=flow@entry=0x7fff9800df70, diff_bytes=0, diff_pkts=0)
    at controller/src/vnsw/agent/vrouter/flow_stats/flow_stats_collector.cc:435
#4 0x0000000000b298cc in FlowTable::SendFlowInternal (this=<optimized out>, fe=0x7fff9800df70, time=1449301884810462) at controller/src/vnsw/agent/pkt/flow_table.cc:1795
#5 0x0000000000b31753 in SendFlows (time=1449301884810462, rflow=0x7fff980385a0, flow=0x7fff9800df70, this=0x7fffe803e320)
    at controller/src/vnsw/agent/pkt/flow_table.cc:1805
#6 FlowTable::Delete (this=0x7fffe803e320, key=..., del_reverse_flow=true) at controller/src/vnsw/agent/pkt/flow_table.cc:1770
#7 0x0000000000b31ed9 in FlowTable::FlowDelete (this=<optimized out>, key=...) at controller/src/vnsw/agent/pkt/flow_table.cc:3675
#8 0x0000000000b3bb9f in operator() (a1=..., p=<optimized out>, this=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:165
#9 operator()<bool, boost::_mfi::mf1<bool, FlowTable, FlowKey>, boost::_bi::list1<FlowKey&> > (a=<synthetic pointer>, f=..., this=<optimized out>)
    at /usr/include/boost/bind/bind.hpp:303
#10 operator()<FlowKey> (a1=..., this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:32
#11 boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, FlowTable, FlowKey>, boost::_bi::list2<boost::_bi::value<FlowTable*>, boost::arg<1> > >, bool, FlowKey>::invoke (function_obj_ptr=..., a0=...) at /usr/include/boost/function/function_template.hpp:132
#12 0x0000000000b49659 in operator() (a0=<error reading variable: access outside bounds of object referenced via synthetic pointer>, this=0x7fff9e7f8a40)
    at /usr/include/boost/function/function_template.hpp:767
#13 QueueTaskRunner<FlowKey, WorkQueue<FlowKey> >::RunQueue (this=0x7fffa800bb50) at controller/src/base/queue_task.h:81
#14 0x000000000101b950 in TaskImpl::execute (this=0x7fffeea8bd40) at controller/src/base/task.cc:238
#15 0x00007ffff5e43b3a in ?? () from /usr/lib/libtbb.so.2
#16 0x00007ffff5e3f816 in ?? () from /usr/lib/libtbb.so.2
#17 0x00007ffff5e3ef4b in ?? () from /usr/lib/libtbb.so.2
#18 0x00007ffff5e3b0ff in ?? () from /usr/lib/libtbb.so.2
#19 0x00007ffff5e3b2f9 in ?? () from /usr/lib/libtbb.so.2
#20 0x00007ffff605f182 in start_thread (arg=0x7fff9e7f9700) at pthread_create.c:312
#21 0x00007ffff533847d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/15629
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/15629
Committed: http://github.org/Juniper/contrail-controller/commit/f30bb93d524961d11616226ae1813735c171fbd2
Submitter: Zuul
Branch: R2.20

commit f30bb93d524961d11616226ae1813735c171fbd2
Author: Ashok Singh <email address hidden>
Date: Sat Dec 5 00:50:25 2015 -0800

Fix agent crash at FlowStatsCollector::FlowExport

We were accessing invalid index from map. Provided accessor function
to retrive entries from map. The accessor function validates the index.
Closes-Bug: #1523059

Change-Id: Idd6a85b967aaab766ba54e84d13d6738613d7d58

[Ashok Singh (ashoksr)]

The mainline commit for this issue needs to committed only after https://github.com/Juniper/contrail-controller/commit/940e4c1252b4d9dea4a410a30069436cfda4ff87 goes in.

[Ashok Singh (ashoksr)]

Fix committed as part of https://github.com/Juniper/contrail-controller/commit/5e9a68085190956a467a12a5867d7d4ea6f3f984