vrouter-agent crash at FlowEntry::SetRpfNH after agent restart during lbaas test

[Vedamurthy Joshi]

R2.20 Build 99 Centos 6.5 Icehouse

On this setup, a LBAAS sanity test test_lbmethod_round_robin was being run.
After all configuration in the middle of the lbaas test, i manually restarted vrouter-agent on nodei2 (nodei2 and nodei3 had the lbaas netns instances)
In about 2 mins after this, vrouter-agent crashed twice

Both cores will be in http://10.204.216.50/Docs/bugs/#

gdb /usr/bin/contrail-vrouter-agent core.contrail-vroute.15653.nodei2.1443002601

(gdb) bt
#0 0x00002b65661c0925 in raise () from /lib64/libc.so.6
#1 0x00002b65661c2105 in abort () from /lib64/libc.so.6
#2 0x00002b65661b9a4e in __assert_fail_base () from /lib64/libc.so.6
#3 0x00002b65661b9b10 in __assert_fail () from /lib64/libc.so.6
#4 0x000000000161538f in FlowEntry::SetRpfNH(FlowTable*, AgentRoute const*) ()
#5 0x000000000161b49b in FlowTable::ResyncRpfNH(RouteFlowKey const&, AgentRoute const*) ()
#6 0x00000000016197bb in InetRouteFlowUpdate::NhChange(AgentRoute*, NextHop const*, NextHop const*) ()
#7 0x0000000001618c5a in RouteFlowUpdate::Notify(DBTablePartBase*, DBEntryBase*) ()
#8 0x00000000016434e1 in void boost::_mfi::mf2<void, RouteFlowUpdate, DBTablePartBase*, DBEntryBase*>::call<InetRouteFlowUpdate*, DBTablePartBase*, DBEntryBase*>(InetRouteFlowUpdate*&, void const*, DBTablePartBase*&, DBEntryBase*&) const ()
#9 0x0000000001642581 in void boost::_mfi::mf2<void, RouteFlowUpdate, DBTablePartBase*, DBEntryBase*>::operator()<InetRouteFlowUpdate*>(InetRouteFlowUpdate*&, DBTablePartBase*, DBEntryBase*) const ()
#10 0x000000000164036a in void boost::_bi::list3<boost::_bi::value<InetRouteFlowUpdate*>, boost::arg<1>, boost::arg<2> >::operator()<boost::_mfi::mf2<void, RouteFlowUpdate, DBTablePartBase*, DBEntryBase*>, boost::_bi::list2<DBTablePartBase*&, DBEntryBase*&> >(boost::_bi::type<void>, boost::_mfi::mf2<void, RouteFlowUpdate, DBTablePartBase*, DBEntryBase*>&, boost::_bi::list2<DBTablePartBase*&, DBEntryBase*&>&, int) ()
#11 0x000000000163d376 in void boost::_bi::bind_t<void, boost::_mfi::mf2<void, RouteFlowUpdate, DBTablePartBase*, DBEntryBase*>, boost::_bi::list3<boost::_bi::value<InetRouteFlowUpdate*>, boost::arg<1>, boost::arg<2> > >::operator()<DBTablePartBase*, DBEntryBase*>(DBTablePartBase*&, DBEntryBase*&) ()
#12 0x000000000163a157 in boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, RouteFlowUpdate, DBTablePartBase*, DBEntryBase*>, boost::_bi::list3<boost::_bi::value<InetRouteFlowUpdate*>, boost::arg<1>, boost::arg<2> > >, void, DBTablePartBase*, DBEntryBase*>::invoke(boost::detail::function::function_buffer&, DBTablePartBase*, DBEntryBase*) ()
#13 0x0000000001b5f38a in boost::function2<void, DBTablePartBase*, DBEntryBase*>::operator()(DBTablePartBase*, DBEntryBase*) const ()
#14 0x0000000001b5e8c0 in DBTableBase::ListenerInfo::RunNotify(DBTablePartBase*, DBEntryBase*) ()
#15 0x0000000001b5d343 in DBTableBase::RunNotify(DBTablePartBase*, DBEntryBase*) ()
#16 0x0000000001b635d9 in DBTablePartBase::RunNotify() ()
#17 0x0000000001b5a150 in DBPartition::QueueRunner::Run() ()
#18 0x0000000001c953d2 in TaskImpl::execute() ()
#19 0x00002b65657ca18a in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x2b6618000900, parent=..., child=0x2b6620004240)
    at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/custom_scheduler.h:449
#20 0x00002b65657c1033 in tbb::internal::arena::process (this=0x3cc0780, s=...) at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/arena.cpp:99
#21 0x00002b65657bf906 in tbb::internal::market::process (this=0x3cbe380, j=...) at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/market.cpp:393
#22 0x00002b65657ba4bc in tbb::internal::rml::private_worker::run (this=0x3cbeb80) at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/private_server.cpp:263
#23 0x00002b65657ba362 in tbb::internal::rml::private_worker::thread_routine (arg=0x3cbeb80)
    at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/private_server.cpp:231
#24 0x00002b65655829d1 in start_thread () from /lib64/libpthread.so.0
#25 0x00002b6566276b5d in clone () from /lib64/libc.so.6
(gdb)

[Hari Prasad Killi (haripk)]

R2.20 review : https://review.opencontrail.org/#/c/14033/
R2.20 commit : https://github.com/Juniper/contrail-controller/commit/940e4c1252b4d9dea4a410a30069436cfda4ff87