Gateway_less_Fwd: vrouter agent crash observed @__assert_fail, AgentPath::ComputeNextHop, RouteKSyncEntry::GetActiveNextHop

[Chandra Sekhar Reddy Mallam]

vrouter agent crash observed @__assert_fail, AgentPath::ComputeNextHop, RouteKSyncEntry::GetActiveNextHop, RouteKSyncEntry::Sync

Build
------
R4.1.0.0 Build 23 Ubuntu 14.04 Mitaka

Topology
—————
Control/config/analytics node :nodei15
Compute nodes : nodek11, nodec23 and nodeb3

Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f6864da1c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007f6864da1c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6864da5028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f6864d9abf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f6864d9aca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000001ae15bd in AgentPath::ComputeNextHop (this=0x7f68240784a0, agent=0x5086e50) at controller/src/vnsw/agent/oper/agent_path.cc:85
#5 0x00000000020db9fd in RouteKSyncEntry::GetActiveNextHop (this=0x7f6824352230, route=0x7f68582db860) at controller/src/vnsw/agent/vrouter/ksync/route_ksync.cc:376
#6 0x00000000020dbb4f in RouteKSyncEntry::Sync (this=0x7f6824352230, e=0x7f68582db860) at controller/src/vnsw/agent/vrouter/ksync/route_ksync.cc:403
#7 0x0000000002222cd5 in KSyncDBObject::Notify (this=0x7f68500120d0, partition=0x7f6831a02d40, e=0x7f68582db860) at controller/src/ksync/ksync_object.cc:496
#8 0x000000000223123a in boost::_mfi::mf2<void, KSyncDBObject, DBTablePartBase*, DBEntryBase*>::operator() (this=0x7f685c5fca68, p=0x7f68500120d0, a1=0x7f6831a02d40, a2=0x7f68582db860)
    at /usr/include/boost/bind/mem_fn_template.hpp:280
#9 0x000000000222ff6a in boost::_bi::list3<boost::_bi::value<KSyncDBObject*>, boost::arg<1>, boost::arg<2> >::operator()<boost::_mfi::mf2<void, KSyncDBObject, DBTablePartBase*, DBEntryBase*>, boost::_bi::list2<DBTablePartBase*&, DBEntryBase*&> > (this=0x7f685c5fca78, f=..., a=...) at /usr/include/boost/bind/bind.hpp:392
#10 0x000000000222e7ae in boost::_bi::bind_t<void, boost::_mfi::mf2<void, KSyncDBObject, DBTablePartBase*, DBEntryBase*>, boost::_bi::list3<boost::_bi::value<KSyncDBObject*>, boost::arg<1>, boost::arg<2> > >::operator()<DBTablePartBase*, DBEntryBase*> (this=0x7f685c5fca68, a1=@0x7f685c5fc9a0: 0x7f6831a02d40, a2=@0x7f685c5fc998: 0x7f68582db860) at /usr/include/boost/bind/bind_template.hpp:61
#11 0x000000000222cd41 in boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, KSyncDBObject, DBTablePartBase*, DBEntryBase*>, boost::_bi::list3<boost::_bi::value<KSyncDBObject*>, boost::arg<1>, boost::arg<2> > >, void, DBTablePartBase*, DBEntryBase*>::invoke (function_obj_ptr=..., a0=0x7f6831a02d40, a1=0x7f68582db860) at /usr/include/boost/function/function_template.hpp:153
#12 0x0000000002774dda in boost::function2<void, DBTablePartBase*, DBEntryBase*>::operator() (this=0x7f685c5fca60, a0=0x7f6831a02d40, a1=0x7f68582db860) at /usr/include/boost/function/function_template.hpp:767
#13 0x0000000002774193 in DBTableBase::ListenerInfo::RunNotify (this=0x7f68319f8fb0, tpart=0x7f6831a02d40, entry=0x7f68582db860) at controller/src/db/db_table.cc:115
#14 0x0000000002771d97 in DBTableBase::RunNotify (this=0x7f68319f8e00, tpart=0x7f6831a02d40, entry=0x7f68582db860) at controller/src/db/db_table.cc:207
#15 0x000000000277a5eb in DBTablePartBase::RunNotify (this=0x7f6831a02d40) at controller/src/db/db_table_partition.cc:47
#16 0x000000000276ec2a in DBPartition::QueueRunner::Run (this=0x7f68482dad20) at controller/src/db/db_partition.cc:220
#17 0x000000000293886c in TaskImpl::execute (this=0x7f6867937940) at controller/src/base/task.cc:277
#18 0x00007f6865970b3a in ?? () from /usr/lib/libtbb.so.2
#19 0x00007f686596c816 in ?? () from /usr/lib/libtbb.so.2
#20 0x00007f686596bf4b in ?? () from /usr/lib/libtbb.so.2
#21 0x00007f68659680ff in ?? () from /usr/lib/libtbb.so.2
#22 0x00007f68659682f9 in ?? () from /usr/lib/libtbb.so.2
#23 0x00007f6865b8c184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#24 0x00007f6864e6537d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)

[Chandra Sekhar Reddy Mallam (cmallam)]

Attached logs and cores to
bhushana@10.204.216.50:/home/bhushana/Documents/technical/bugs/<bug-ID>

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35522
Submitter: Naveen N (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35522
Committed: http://github.com/Juniper/contrail-controller/commit/bda3c67e8dc79465f5bed5b7a32197c30b620edb
Submitter: Zuul (<email address hidden>)
Branch: master

commit bda3c67e8dc79465f5bed5b7a32197c30b620edb
Author: Naveen N <email address hidden>
Date: Tue Sep 12 16:00:56 2017 +0530

* ip-fabric:ip-fabric VRF shouldnt be deleted from agent

Upon config delete agent might get delete of ip-fabric VRF from
config node, ignore such deletes.
If gateway route points to route being deleted, pick discard
nh for processing.

Change-Id: I9f05f9305f48725618cf1b54a94bb06af34e7ed9
Closes-bug: #1712499