vrouter-agent crashed at KSyncFlowIndexManager::UpdateFlowHandle(FlowTableKSyncEntry*

[Sandip Dey]

Build R3.1 24 ubuntu14.04

There are multiple vrouter cores in nodel7 and few other nodes with the below bt.
If someone can take a look.

logs saved at http://10.204.216.50/Docs/bugs/<bug-id>

#0 0x00007f77e2b4ecc9 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f77e2b520d8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f77e2b47b86 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f77e2b47c32 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000da083b in KSyncFlowIndexManager::UpdateFlowHandle(FlowTableKSyncEntry*, unsigned int, unsigned char) ()
#5 0x0000000000cdf332 in FlowTable::ProcessKSyncFlowEvent(FlowEventKSync const*, FlowEntry*) ()
#6 0x0000000000cdf803 in FlowTable::ProcessFlowEvent(FlowEvent const*, FlowEntry*, FlowEntry*) ()
#7 0x0000000000cf45ea in FlowProto::FlowKSyncMsgHandler(FlowEvent*, FlowTable*) ()
#8 0x0000000000cf499e in FlowProto::FlowEventHandler(FlowEvent*, FlowTable*) ()
#9 0x0000000000d14ecd in FlowEventQueueBase::Handler(FlowEvent*) ()
#10 0x0000000000cf9762 in QueueTaskRunner<FlowEvent*, WorkQueue<FlowEvent*> >::RunQueue() ()
#11 0x00000000012e518f in TaskImpl::execute() ()
#12 0x00007f77e371db3a in ?? () from /usr/lib/libtbb.so.2
#13 0x00007f77e3719816 in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f77e3718f4b in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f77e37150ff in ?? () from /usr/lib/libtbb.so.2
#16 0x00007f77e37152f9 in ?? () from /usr/lib/libtbb.so.2
#17 0x00007f77e3939182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#18 0x00007f77e2c1247d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Regards
Sandip

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23480
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23480
Committed: http://github.org/Juniper/contrail-controller/commit/2a743d785986f7c276927b59749c1acb28534c70
Submitter: Zuul
Branch: master

commit 2a743d785986f7c276927b59749c1acb28534c70
Author: Prabhjot Singh Sethi <email address hidden>
Date: Sat Aug 20 00:08:14 2016 +0530

Fix Vrouter Agent crash @ update flow handle

Issue:
------
Even though Hash id is part of the key for ksync flow
entry, we do allow change of hash id to -1 when other
ksync entry for the same flow ends up receiving same
flow handle to handle two entries with same key.
however if there was a pending response(Add/Change Ack)
from vrouter we will end up processing the ack as a
index allocation from value -1 to flow handle for
delete ksync entry. This can end up acquiring the
index from the active entry and cause invalid state
for ksync flow entry, resulting in crash.

Fix:
----
Save the last vrouter hash id (hash id used while
sending message to vrouter) and compare the ack/response
against this hash id to evaluate if response has a new
index allocation.

Closes-Bug: 1613553
Change-Id: I21a12fbd5684de871eabf5cd6dd1568c0a9d6c52

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23513
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23514
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23515
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23514
Committed: http://github.org/Juniper/contrail-controller/commit/067a5f2127ec4f9d207d9fde0ba4e803ebcb2ca4
Submitter: Zuul
Branch: R3.0

commit 067a5f2127ec4f9d207d9fde0ba4e803ebcb2ca4
Author: Prabhjot Singh Sethi <email address hidden>
Date: Sat Aug 20 00:08:14 2016 +0530

Fix Vrouter Agent crash @ update flow handle

Issue:
------
Even though Hash id is part of the key for ksync flow
entry, we do allow change of hash id to -1 when other
ksync entry for the same flow ends up receiving same
flow handle to handle two entries with same key.
however if there was a pending response(Add/Change Ack)
from vrouter we will end up processing the ack as a
index allocation from value -1 to flow handle for
delete ksync entry. This can end up acquiring the
index from the active entry and cause invalid state
for ksync flow entry, resulting in crash.

Fix:
----
Save the last vrouter hash id (hash id used while
sending message to vrouter) and compare the ack/response
against this hash id to evaluate if response has a new
index allocation.

Conflicts:
 src/vnsw/agent/pkt/test/test_flow_mgr_instances.cc

Closes-Bug: 1613553
Change-Id: I21a12fbd5684de871eabf5cd6dd1568c0a9d6c52
(cherry picked from commit 2a743d785986f7c276927b59749c1acb28534c70)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23513
Committed: http://github.org/Juniper/contrail-controller/commit/390bc9f56f86f13f7e7c54a0cfddfb276ac8f245
Submitter: Zuul
Branch: R3.1

commit 390bc9f56f86f13f7e7c54a0cfddfb276ac8f245
Author: Prabhjot Singh Sethi <email address hidden>
Date: Sat Aug 20 00:08:14 2016 +0530

Fix Vrouter Agent crash @ update flow handle

Issue:
------
Even though Hash id is part of the key for ksync flow
entry, we do allow change of hash id to -1 when other
ksync entry for the same flow ends up receiving same
flow handle to handle two entries with same key.
however if there was a pending response(Add/Change Ack)
from vrouter we will end up processing the ack as a
index allocation from value -1 to flow handle for
delete ksync entry. This can end up acquiring the
index from the active entry and cause invalid state
for ksync flow entry, resulting in crash.

Fix:
----
Save the last vrouter hash id (hash id used while
sending message to vrouter) and compare the ack/response
against this hash id to evaluate if response has a new
index allocation.

Closes-Bug: 1613553
Change-Id: I21a12fbd5684de871eabf5cd6dd1568c0a9d6c52
(cherry picked from commit 2a743d785986f7c276927b59749c1acb28534c70)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23515
Committed: http://github.org/Juniper/contrail-controller/commit/f05357fec7410fab2316bba417f96d253ad10f47
Submitter: Zuul
Branch: R3.0.2.x

commit f05357fec7410fab2316bba417f96d253ad10f47
Author: Prabhjot Singh Sethi <email address hidden>
Date: Sat Aug 20 00:08:14 2016 +0530

Fix Vrouter Agent crash @ update flow handle

Issue:
------
Even though Hash id is part of the key for ksync flow
entry, we do allow change of hash id to -1 when other
ksync entry for the same flow ends up receiving same
flow handle to handle two entries with same key.
however if there was a pending response(Add/Change Ack)
from vrouter we will end up processing the ack as a
index allocation from value -1 to flow handle for
delete ksync entry. This can end up acquiring the
index from the active entry and cause invalid state
for ksync flow entry, resulting in crash.

Fix:
----
Save the last vrouter hash id (hash id used while
sending message to vrouter) and compare the ack/response
against this hash id to evaluate if response has a new
index allocation.

Conflicts:
 src/vnsw/agent/pkt/test/test_flow_mgr_instances.cc

Closes-Bug: 1613553
Change-Id: I21a12fbd5684de871eabf5cd6dd1568c0a9d6c52
(cherry picked from commit 2a743d785986f7c276927b59749c1acb28534c70)