Agent crash in tbb::strict_ppl::internal::micro_queue

[Vinod Nair]

WIth Kmod Vrouter , Agent crashed in tbb::strict_ppl::internal::micro_queue
This is with PB7 binary. The crash happened after a day or so

ImportError: No module named 'libstdcxx'
(gdb)
(gdb) bt
#0 0x00007fefd9741cc9 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fefd97450d8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fefda04c535 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007fefda04a6d6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007fefda04a703 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007fefda04b1bf in __cxa_pure_virtual () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x0000000000bc3cc9 in tbb::strict_ppl::internal::micro_queue<boost::shared_ptr<FlowExportReq> >::push (this=0x7fefd2f43e80, item=item@entry=0x7fefd0dbf5e0, k=0,
    base=...) at /usr/include/tbb/internal/_concurrent_queue_impl.h:237
#7 0x0000000000bc40d8 in internal_push (src=0x7fefd0dbf5e0, this=0x7fefb80151a8) at /usr/include/tbb/internal/_concurrent_queue_impl.h:478
#8 push (source=..., this=0x7fefb80151a8) at /usr/include/tbb/concurrent_queue.h:109
#9 EnqueueInternal (entry=..., this=0x7fefb80151a8) at controller/src/base/queue_task.h:540
#10 WorkQueue<boost::shared_ptr<FlowExportReq> >::Enqueue (this=this@entry=0x7fefb80151a8, entry=...) at controller/src/base/queue_task.h:304
#11 0x0000000000bbe242 in FlowStatsCollector::AddEvent (this=0x7fefb80150f0, flow=...) at controller/src/vnsw/agent/vrouter/flow_stats/flow_stats_collector.cc:523
#12 0x0000000000c2cba4 in FlowMgmtManager::AddFlow (this=0x7fefa4435a70, flow=...) at controller/src/vnsw/agent/pkt/flow_mgmt.cc:452
#13 0x0000000000c2cf39 in FlowMgmtManager::RequestHandler (this=0x7fefa4435a70, req=...) at controller/src/vnsw/agent/pkt/flow_mgmt.cc:340
#14 0x0000000000c2e76c in operator() (a1=..., p=<optimized out>, this=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:165
#15 operator()<bool, boost::_mfi::mf1<bool, FlowMgmtManager, boost::shared_ptr<FlowMgmtRequest> >, boost::_bi::list1<boost::shared_ptr<FlowMgmtRequest>&> > (
    a=<synthetic pointer>, f=..., this=<optimized out>) at /usr/include/boost/bind/bind.hpp:303
#16 operator()<boost::shared_ptr<FlowMgmtRequest> > (a1=..., this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:32
#17 boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, FlowMgmtManager, boost::shared_ptr<FlowMgmtRequest> >, boost::_bi::list2<boost::_bi::value<FlowMgmtManager*>, boost::arg<1> > >, bool, boost::shared_ptr<FlowMgmtRequest> >::invoke (function_obj_ptr=..., a0=...)
    at /usr/include/boost/function/function_template.hpp:132
#18 0x0000000000c34654 in operator() (a0=..., this=0x7fefd0dbfaa0) at /usr/include/boost/function/function_template.hpp:767
#19 QueueTaskRunner<boost::shared_ptr<FlowMgmtRequest>, WorkQueue<boost::shared_ptr<FlowMgmtRequest> > >::RunQueue (this=0x7fef7040d970)
    at controller/src/base/queue_task.h:87
#20 0x000000000119359c in TaskImpl::execute (this=0x7fefa9e73b40) at controller/src/base/task.cc:253
#21 0x00007fefda310b3a in ?? () from /usr/lib/libtbb.so.2
#22 0x00007fefda30c816 in ?? () from /usr/lib/libtbb.so.2
#23 0x00007fefda30bf4b in ?? () from /usr/lib/libtbb.so.2
#24 0x00007fefda3080ff in ?? () from /usr/lib/libtbb.so.2
#25 0x00007fefda3082f9 in ?? () from /usr/lib/libtbb.so.2
#26 0x00007fefda52c182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#27 0x00007fefd980547d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)

Core in /cs-shared/bugs/1552826
Build : pb7 binary

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18436
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18442
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18436
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18442
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/18436
Committed: http://github.org/Juniper/contrail-controller/commit/ba705e7c50fe44316a9b8acec9c46ab447393593
Submitter: Zuul
Branch: R3.0

commit ba705e7c50fe44316a9b8acec9c46ab447393593
Author: Ashok Singh <email address hidden>
Date: Tue Mar 15 01:59:51 2016 -0700

Re-initialize FlowStatsCollector pointer of FlowEntry.

During flow entry allocation from free_list, the fsc_ pointer of
FlowEntry was not being initialized. Because of state fsc_ pointer
agent was crashing.

Also add changes to return FlowStatsCollector which is not marked
for delete, during FlowStatsCollector pointer assignment to FlowEntry.
Also ensure that forward and reverse flows go to same FlowStatsCollector.
Closes-Bug: #1552826

Change-Id: I8950f3799944d67c3b8af9c7bfdb7a3ee1f01d50

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/18442
Committed: http://github.org/Juniper/contrail-controller/commit/f99c76fd1dc4256c1adea42d76eb2be94f005e2c
Submitter: Zuul
Branch: master

commit f99c76fd1dc4256c1adea42d76eb2be94f005e2c
Author: Ashok Singh <email address hidden>
Date: Tue Mar 15 01:59:51 2016 -0700

Re-initialize FlowStatsCollector pointer of FlowEntry.

During flow entry allocation from free_list, the fsc_ pointer of
FlowEntry was not being initialized. Because of state fsc_ pointer
agent was crashing.

Also add changes to return FlowStatsCollector which is not marked
for delete, during FlowStatsCollector pointer assignment to FlowEntry.
Also ensure that forward and reverse flows go to same FlowStatsCollector.
Closes-Bug: #1552826

(cherry picked from commit b9998834b92c8894f19194987fd7a30e9587108d)

Change-Id: Ia589fdf97cbd3efba06277e44ccaac062da41598