Agent crash due to parallel access

[Hari Prasad Killi]

(gdb) bt
#0 next_node (node=<synthetic pointer>) at /usr/include/boost/intrusive/detail/tree_algorithms.hpp:446
#1 next_node (p=<synthetic pointer>) at /usr/include/boost/intrusive/rbtree_algorithms.hpp:352
#2 operator++ (this=<synthetic pointer>) at /usr/include/boost/intrusive/detail/tree_node.hpp:135
#3 KSyncObject::BackRefReEval (this=this@entry=0x7fc0a001ccc8, key=key@entry=0x7fc0a0542de8) at controller/src/ksync/ksync_object.cc:1497
#4 0x0000000000ed256b in KSyncObject::NotifyEvent (this=0x7fc0a001ccc8, entry=0x7fc0a0542de8, event=<optimized out>) at controller/src/ksync/ksync_object.cc:1402
#5 0x0000000000ed272d in KSyncObject::SafeNotifyEvent (this=0x7fc0a001ccc8, entry=0x7fc0a0542de8, event=KSyncEntry::ADD_CHANGE_REQ) at controller/src/ksync/ksync_object.cc:277
#6 0x0000000000ed2777 in KSyncObject::Change (this=<optimized out>, entry=<optimized out>) at controller/src/ksync/ksync_object.cc:228
#7 0x0000000000dbc0af in KSyncFlowIndexManager::Update (this=0x7fc0a00165f8, flow=0x7fc0a1efb388) at controller/src/vnsw/agent/vrouter/ksync/ksync_flow_index_manager.cc:76
#8 0x0000000000cebfdc in FlowTable::AddInternal (this=this@entry=0x7fc0a066d368, flow_req=flow_req@entry=0x7fc0a24f5db8, flow=flow@entry=0x7fc0a1efb388,
    rflow_req=rflow_req@entry=0x7fc0a0cc2098, rflow=<optimized out>, rflow@entry=0x7fc0a105bca8, fwd_flow_update=fwd_flow_update@entry=true,
    rev_flow_update=rev_flow_update@entry=true) at controller/src/vnsw/agent/pkt/flow_table.cc:300
#9 0x0000000000cec448 in FlowTable::Update (this=0x7fc0a066d368, flow=flow@entry=0x7fc0a24f5db8, rflow=0x7fc0a0cc2098) at controller/src/vnsw/agent/pkt/flow_table.cc:184
#10 0x0000000000d02e54 in FlowProto::UpdateFlow (this=<optimized out>, flow=flow@entry=0x7fc0a24f5db8) at controller/src/vnsw/agent/pkt/flow_proto.cc:305
#11 0x0000000000d132f2 in PktFlowInfo::Add (this=this@entry=0x7fc0b5a968b0, pkt=<optimized out>, in=in@entry=0x7fc0b5a96830, out=out@entry=0x7fc0b5a96870)
    at controller/src/vnsw/agent/pkt/pkt_flow_info.cc:1750
#12 0x0000000000d25ae5 in FlowHandler::Run (this=0x7fc0a4375e18) at controller/src/vnsw/agent/pkt/flow_handler.cc:150
#13 0x0000000000d1ef7d in Proto::RunProtoHandler (this=this@entry=0x7fc0a066c918, handler=handler@entry=0x7fc0a4375e18) at controller/src/vnsw/agent/pkt/proto.cc:51
#14 0x0000000000d031a1 in FlowProto::FlowEventHandler (this=0x7fc0a066c918, req=<optimized out>, table=<optimized out>) at controller/src/vnsw/agent/pkt/flow_proto.cc:429
#15 0x0000000000d22e5d in FlowEventQueueBase::Handler (this=0x7fc0a2617628, event=0x7fc0b09bd498) at controller/src/vnsw/agent/pkt/flow_event.cc:105
#16 0x0000000000d081d2 in operator() (a0=0x7fc0b09bd498, this=0x7fc0b5a96b30) at /usr/include/boost/function/function_template.hpp:767
#17 QueueTaskRunner<FlowEvent*, WorkQueue<FlowEvent*> >::RunQueue (this=0x7fc0b03f5458) at controller/src/base/queue_task.h:92
#18 0x0000000001305d6f in TaskImpl::execute (this=0x7fc0b7455b40) at controller/src/base/task.cc:262

(gdb) p back_ref_tree_
$1 = boost::intrusive::set<KSyncBackReference> with 0 elements

back_ref_tree_ is updated while iteration is in progress.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36760
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36761
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36768
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36760
Committed: http://github.com/Juniper/contrail-controller/commit/ef13c6b5ec0250df8bf7be0241a240c799e6b86f
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit ef13c6b5ec0250df8bf7be0241a240c799e6b86f
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36813
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36822
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36768
Committed: http://github.com/Juniper/contrail-controller/commit/849be7c017bf13b7c054d0cc5b436a6a87c3b0ed
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 849be7c017bf13b7c054d0cc5b436a6a87c3b0ed
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36761
Committed: http://github.com/Juniper/contrail-controller/commit/ae7bb5713543b07d0e2d0aa0d1c9b8d2d99fe4b3
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit ae7bb5713543b07d0e2d0aa0d1c9b8d2d99fe4b3
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36822
Committed: http://github.com/Juniper/contrail-controller/commit/46c9d4683f5a5d75a3271c395980a8dc2e64dba6
Submitter: Zuul (<email address hidden>)
Branch: master

commit 46c9d4683f5a5d75a3271c395980a8dc2e64dba6
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36813
Committed: http://github.com/Juniper/contrail-controller/commit/7e1054b88f43183ae484a52574996adc83556f2a
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 7e1054b88f43183ae484a52574996adc83556f2a
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37773
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37774
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/37774
Committed: http://github.com/Juniper/contrail-controller/commit/745426628a50197b76a057f7012e35d18d50d233
Submitter: Zuul (<email address hidden>)
Branch: R3.0.3.x

commit 745426628a50197b76a057f7012e35d18d50d233
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709
(cherry picked from commit 849be7c017bf13b7c054d0cc5b436a6a87c3b0ed)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/37773
Committed: http://github.com/Juniper/contrail-controller/commit/73a4537cacc504d79727709764bbd93d448de429
Submitter: Zuul (<email address hidden>)
Branch: R3.0

commit 73a4537cacc504d79727709764bbd93d448de429
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709
(cherry picked from commit 849be7c017bf13b7c054d0cc5b436a6a87c3b0ed)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/38193
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/38193
Committed: http://github.com/Juniper/contrail-controller/commit/158da92f29af86e5345d79a51e864271dbb9594f
Submitter: Zuul (<email address hidden>)
Branch: R3.2.3.x

commit 158da92f29af86e5345d79a51e864271dbb9594f
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709
(cherry picked from commit ef13c6b5ec0250df8bf7be0241a240c799e6b86f)