Cores were generated while upgrading to R2.21 from R2.20

[Daisuke Nakajima]

Cores were generated while upgrading to R2.21 build 102 from R2.20 build 64.

[Hari Prasad Killi (haripk)]

Core was generated by `/usr/bin/contrail-tor-agent --config_file /etc/contrail/contrail-tor-agent-1.co'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000000000084cf8e in AgentRouteTable::Input (this=0x7ffb58026bd0, part=0x7ffb58026d80, client=0x0, req=0x7ffb607ab460)
    at controller/src/vnsw/agent/oper/agent_route.cc:330
330 controller/src/vnsw/agent/oper/agent_route.cc: No such file or directory.

(gdb) bt
#0 0x000000000084cf8e in AgentRouteTable::Input (this=0x7ffb58026bd0, part=0x7ffb58026d80, client=0x0, req=0x7ffb607ab460)
    at controller/src/vnsw/agent/oper/agent_route.cc:330
#1 0x000000000084c7e5 in AgentRouteTable::Input (this=0x7ffb680343a0, part=<optimized out>, client=0x0, req=0x7ffb607ab460)
    at controller/src/vnsw/agent/oper/agent_route.cc:314
#2 0x0000000000d0af36 in DBPartition::QueueRunner::Run (this=0x7ffb61278d60) at controller/src/db/db_partition.cc:187
#3 0x0000000000e0a1c0 in TaskImpl::execute (this=0x7ffb6ee3ba40) at controller/src/base/task.cc:238

(gdb) p *req
$1 = {oper = DBRequest::DB_ENTRY_DELETE, key = {_M_ptr = 0x7ffb6000e9b0}, data = {_M_ptr = 0x0}}

(gdb) p /x *(AgentRouteKey *)0x7ffb6000e9b0
$3 = (EvpnRouteKey) {<AgentRouteKey> = {<AgentKey> = {<DBRequestKey> = {_vptr.DBRequestKey = 0xe2f830 <vtable for EvpnRouteKey+16>}, sub_op_ = 0x0},
    peer_ = 0x7ffb60013e00, vrf_name_ = {static npos = <optimized out>,
      _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7ffb58017af8}}}, dmac_ = {
    static kZeroMac = {static kZeroMac = <same as static member of an already seen type>, static kBroadcastMac = {
        static kZeroMac = <same as static member of an already seen type>, static kBroadcastMac = <same as static member of an already seen type>, addr_ = {
          ether_addr_octet = {0xff, 0xff, 0xff, 0xff, 0xff, 0xff}}}, addr_ = {ether_addr_octet = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}},
    static kBroadcastMac = <same as static member of an already seen type>, addr_ = {ether_addr_octet = {0x0, 0x14, 0xf6, 0x8a, 0x98, 0x3e}}}, ip_addr_ = {
    type_ = 0x0, ipv4_address_ = {addr_ = {s_addr = 0x0}}, ipv6_address_ = {addr_ = {__in6_u = {__u6_addr8 = {0x0 <repeats 16 times>}, __u6_addr16 = {0x0,
            0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, __u6_addr32 = {0x0, 0x0, 0x0, 0x0}}}, scope_id_ = 0x0}}, ethernet_tag_ = 0xb04}

(gdb) p *(Peer *)0x7ffb60013e00
$4 = {_vptr.Peer = 0x33, type_ = 51, name_ = {static npos = <optimized out>,
    _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
      _M_p = 0x7ffb00000000 <error: Cannot access memory at address 0x7ffb00000000>}}, export_to_controller_ = 99}

[Prabhjot Singh Sethi (prabhjot)]

crash happens due to async processing of route delete request, while the peer for the route is already deleted.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14771
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14771
Committed: http://github.org/Juniper/contrail-controller/commit/734223864ec70ca8e04056e8aae4fc7a1e854c4d
Submitter: Zuul
Branch: master

commit 734223864ec70ca8e04056e8aae4fc7a1e854c4d
Author: Prabhjot Singh Sethi <email address hidden>
Date: Mon Nov 2 11:19:35 2015 +0530

Fix ToR-Agent crash on withdrawing HA stale route

Issue:
------
since the route export operations were done async, it
resulted in releasing the ovsdb route peer due to cleanup
before completion of the route operations, causing invalid
pointer access while withdrawing routes

Fix:
----
on DB notification to KSync, enqueue export event for the
entry to a KSync task workqueue, also hold the state of
KSync entry which assures that entry is held in the state
till operation is complete, allowing route operations to
complete before route peer cleanup.

Closes-Bug: 1503124
Change-Id: I32732ce9e32a6fa8bddd3ee2cc8c0bd783afe945

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14811
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14813
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14813
Committed: http://github.org/Juniper/contrail-controller/commit/a3bf35a1534d6989c56fb497b9913416b5f450b6
Submitter: Zuul
Branch: R2.21.x

commit a3bf35a1534d6989c56fb497b9913416b5f450b6
Author: Prabhjot Singh Sethi <email address hidden>
Date: Mon Nov 2 11:19:35 2015 +0530

Fix ToR-Agent crash on withdrawing HA stale route

Issue:
------
since the route export operations were done async, it
resulted in releasing the ovsdb route peer due to cleanup
before completion of the route operations, causing invalid
pointer access while withdrawing routes

Fix:
----
on DB notification to KSync, enqueue export event for the
entry to a KSync task workqueue, also hold the state of
KSync entry which assures that entry is held in the state
till operation is complete, allowing route operations to
complete before route peer cleanup.

Conflicts:
 src/vnsw/agent/ovs_tor_agent/ovsdb_client/ha_stale_l2_route.cc
 src/vnsw/agent/ovs_tor_agent/ovsdb_client/ovsdb_entry.h

Closes-Bug: 1503124
Change-Id: I32732ce9e32a6fa8bddd3ee2cc8c0bd783afe945
(cherry picked from commit 734223864ec70ca8e04056e8aae4fc7a1e854c4d)
(cherry picked from commit 28043f285a55769195b28a56030fc3d4d91fde61)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14811
Committed: http://github.org/Juniper/contrail-controller/commit/28043f285a55769195b28a56030fc3d4d91fde61
Submitter: Zuul
Branch: R2.20

commit 28043f285a55769195b28a56030fc3d4d91fde61
Author: Prabhjot Singh Sethi <email address hidden>
Date: Mon Nov 2 11:19:35 2015 +0530

Fix ToR-Agent crash on withdrawing HA stale route

Issue:
------
since the route export operations were done async, it
resulted in releasing the ovsdb route peer due to cleanup
before completion of the route operations, causing invalid
pointer access while withdrawing routes

Fix:
----
on DB notification to KSync, enqueue export event for the
entry to a KSync task workqueue, also hold the state of
KSync entry which assures that entry is held in the state
till operation is complete, allowing route operations to
complete before route peer cleanup.

Conflicts:
 src/vnsw/agent/ovs_tor_agent/ovsdb_client/ha_stale_l2_route.cc
 src/vnsw/agent/ovs_tor_agent/ovsdb_client/ovsdb_entry.h

Closes-Bug: 1503124
Change-Id: I32732ce9e32a6fa8bddd3ee2cc8c0bd783afe945
(cherry picked from commit 734223864ec70ca8e04056e8aae4fc7a1e854c4d)