DPDK: agent core on one of nodes Cannot access memory at address 0x303e22676e69727c

[Vinod Nair]

On Upgrading from 3.0.2 Build 43 - 44 kilo , see agent crash on one node with the below trace

Cannot access memory at address 0x303e22676e69727c
eading symbols from ./contrail-vrouter-agent...done.

warning: core file may not match specified executable file.
[New LWP 30591]
[New LWP 30595]
[New LWP 30592]
[New LWP 4576]
[New LWP 11792]
[New LWP 11790]
[New LWP 25339]
[New LWP 4809]
[New LWP 25338]
[New LWP 30593]
[New LWP 30572]
[New LWP 11793]
[New LWP 28614]
[New LWP 30589]
[New LWP 30588]
[New LWP 30590]
[New LWP 11791]
[New LWP 30594]
Cannot access memory at address 0x303e22676e69727c
Cannot access memory at address 0x303e22676e697274
(gdb) bt
#0 0x00007fe9c6b73c37 in ?? ()
#1 0x00007fe9c6b77028 in ?? ()
#2 0x0000000000000020 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb)

Core: /cs-shared/bugs/1587258
Version : 3.0.3.44 Kilo Ubunu 14.04.4

[Vinod Nair (vinodnair)]

see it again on upgrade from 43 to 45
core in the same location
f7bd774ccc9 in __GI___libc_sigaction (sig=4032, act=0xfd7, oact=0x6) at ../sysdeps/unix/sysv/linux/x86_64/sigaction.c:53
#1 0x00007f7bd806bd04 in std::_Rb_tree_insert_and_rebalance(bool, std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::_Rb_tree_node_base&) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x00007f7bda4d2000 in ?? ()
#3 0x00007f7bd7896830 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000001245cd0 in ?? ()
#5 0x00000000011f2580 in ?? ()
#6 0x00007f7bcaffa340 in ?? ()
#7 0x0000000000000011 in ?? ()
#8 0x00007f7bd7745b86 in __assert_fail_base (fmt=0x0, assertion=0x0, file=<optimized out>, line=<optimized out>, function=<optimized out>) at assert.c:68
#9 0x00007f7bd7745c32 in __assert_fail_base (fmt=0x11f1bb0 "controller/src/vnsw/agent/oper/inet_unicast_route.cc", assertion=0x15d <error: Cannot access memory at address 0x15d>,
    file=0xa1339e <InetUnicastRouteEntry::ModifyEcmpPath(boost::asio::ip::address const&, unsigned char, std::set<std::string, std::less<std::string>, std::allocator<std::string> > const&, unsigned int, bool, std::string const&, std::vector<int, std::allocator<int> >, std::vector<std::string, std::allocator<std::string> > const&, PathPreference const&, unsigned int, EcmpLoadBalance const&, DBRequest&, Agent*, AgentPath*)+414> "轤\331\377H\213<$H\211\303H\205\377t\005\350\\\231\331\377H\211\337\350\024\233\331\377\017\037@", line=32635,
    function=0x4 <error: Cannot access memory at address 0x4>) at assert.c:53
#10 0x00007f7bcaffa340 in ?? ()
#11 0x00007f7bcaffa320 in ?? ()
#12 0x0000000000a13d86 in InetUnicastRouteEntry::DeleteComponentNH(Agent*, AgentPath*) ()
#13 0x0000000000a1433e in InetUnicastRouteEntry::EcmpDeletePath(AgentPath*) ()
#14 0x00000000009d1654 in AgentRouteTable::DeletePathFromPeer(DBTablePartBase*, AgentRoute*, AgentPath*) ()
#15 0x00000000009d2c0d in AgentRouteTable::Input(DBTablePartition*, DBClient*, DBRequest*) ()
#16 0x0000000000a1153b in InetUnicastAgentRouteTable::Delete(Peer const*, std::string const&, boost::asio::ip::address const&, unsigned char) ()
#17 0x0000000000a8d2c3 in VmInterface::DeleteIpv4InstanceIp(bool, unsigned int, VrfEntry*) ()
#18 0x0000000000a96184 in VmInterface::DeleteL3(bool, VrfEntry*, boost::asio::ip::address_v4 const&, bool, bool, boost::asio::ip::address_v6 const&, boost::asio::ip::address_v4 const&, unsigned char, int, boost::asio::ip::address_v4 const&) ()
#19 0x0000000000a983e4 in VmInterface::ApplyConfig(bool, bool, bool, VrfEntry*, boost::asio::ip::address_v4 const&, int, bool, bool, boost::asio::ip::address_v6 const&, boost::asio::ip::address_v4 const&, unsigned char, bool, bool, bool, boost::asio::ip::address_v4 const&, bool, bool) ()
#20 0x0000000000a987b8 in VmInterface::Resync(InterfaceTable const*, VmInterfaceData const*) ()
#21 0x00000000009fcd3a in AgentOperDBTable::Resync(DBEntry*, DBRequest const*) ()
#22 0x0000000000acec7c in AgentDBTable::Input(DBTablePartition*, DBClient*, DBRequest*) ()
#23 0x000000000109f2b6 in DBPartition::QueueRunner::Run() ()
#24 0x00000000011b0cff in TaskImpl::execute() ()
#25 0x00007f7bd831bb3a in ?? () from /usr/...

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20830
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20833
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20830
Committed: http://github.org/Juniper/contrail-controller/commit/9502e065f43ed1ce32ca1bf6e5bf846cc0ca136c
Submitter: Zuul
Branch: R3.0

commit 9502e065f43ed1ce32ca1bf6e5bf846cc0ca136c
Author: Hari <email address hidden>
Date: Wed Jun 1 13:50:39 2016 +0530

If VRF is deleted, do not assert for NULL NH while modifying ecmp path.

Change-Id: Id92166fd23de764a0ebc8766fa0f7b5ca589ca5d
closes-bug: #1587258

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20833
Committed: http://github.org/Juniper/contrail-controller/commit/a6e8bbecb89666fd97a2706e7ca3a8da4341f634
Submitter: Zuul
Branch: master

commit a6e8bbecb89666fd97a2706e7ca3a8da4341f634
Author: Hari <email address hidden>
Date: Wed Jun 1 13:50:39 2016 +0530

If VRF is deleted, do not assert for NULL NH while modifying ecmp path.

Change-Id: Id92166fd23de764a0ebc8766fa0f7b5ca589ca5d
closes-bug: #1587258