vrouter-agent crashed in FlowEntryFreeList::Allocate

[Vinod Nair]

Agent crashed with the below back trace.

debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fd5c5c29c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007fd5c5c29c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fd5c5c2d028 in __GI_abort () at abort.c:89
#2 0x00007fd5c5c22bf6 in __assert_fail_base (fmt=0x7fd5c5d733b8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x136c6f0 "table_->ConcurrencyCheck(table_->flow_task_id()) == true", file=file@entry=0x136c668 "controller/src/vnsw/agent/pkt/flow_table.cc", line=line@entry=899,
    function=function@entry=0x136cba0 <FlowEntryFreeList::Allocate(FlowKey const&)::__PRETTY_FUNCTION__> "FlowEntry* FlowEntryFreeList::Allocate(const FlowKey&)") at assert.c:92
#3 0x00007fd5c5c22ca2 in __GI___assert_fail (assertion=0x136c6f0 "table_->ConcurrencyCheck(table_->flow_task_id()) == true", file=0x136c668 "controller/src/vnsw/agent/pkt/flow_table.cc",
    line=899, function=0x136cba0 <FlowEntryFreeList::Allocate(FlowKey const&)::__PRETTY_FUNCTION__> "FlowEntry* FlowEntryFreeList::Allocate(const FlowKey&)") at assert.c:101
#4 0x0000000000cdf5f0 in FlowEntryFreeList::Allocate (this=0x7fd5ba97e910, key=...) at controller/src/vnsw/agent/pkt/flow_table.cc:899
#5 0x0000000000cd31ed in FlowEntry::Allocate (key=..., flow_table=<optimized out>) at controller/src/vnsw/agent/pkt/flow_entry.cc:427
#6 0x0000000000d058ec in PktFlowInfo::Add (this=this@entry=0x7fd5bdea8880, pkt=0x7fd3b97f12f0, in=in@entry=0x7fd5bdea8800, out=out@entry=0x7fd5bdea8840)
    at controller/src/vnsw/agent/pkt/pkt_flow_info.cc:1665
#7 0x0000000000d1893d in FlowHandler::Run (this=0x7fd3bcea4ba0) at controller/src/vnsw/agent/pkt/flow_handler.cc:150
#8 0x0000000000d11bc4 in RunProtoHandler (handler=0x7fd3bcea4ba0, this=0x7fd5b8af9210) at controller/src/vnsw/agent/pkt/proto.cc:51
#9 Proto::ProcessProto (this=this@entry=0x7fd5b8af9210, msg_info=...) at controller/src/vnsw/agent/pkt/proto.cc:66
#10 0x0000000000cf5693 in FlowProto::FlowEventHandler (this=0x7fd5b8af9210, req=0x7fd3b97f1440, table=<optimized out>) at controller/src/vnsw/agent/pkt/flow_proto.cc:406
#11 0x0000000000d15b6d in FlowEventQueueBase::Handler (this=0x7fd590803bb0, event=0x7fd3b97f1440) at controller/src/vnsw/agent/pkt/flow_event.cc:105
#12 0x0000000000cfa3f2 in operator() (a0=0x7fd3b97f1440, this=0x7fd5bdea8b30) at /usr/include/boost/function/function_template.hpp:767
#13 QueueTaskRunner<FlowEvent*, WorkQueue<FlowEvent*> >::RunQueue (this=0x7fd3bd20c820) at controller/src/base/queue_task.h:92
#14 0x00000000012df94f in TaskImpl::execute (this=0x7fd5bf46c140) at controller/src/base/task.cc:262
#15 0x00007fd5c67f8b3a in ?? () from /usr/lib/libtbb.so.2
#16 0x00007fd5c67f4816 in ?? () from /usr/lib/libtbb.so.2
#17 0x00007fd5c67f3f4b in ?? () from /usr/lib/libtbb.so.2
#18 0x00007fd5c67f00ff in ?? () from /usr/lib/libtbb.so.2
#19 0x00007fd5c67f02f9 in ?? () from /usr/lib/libtbb.so.2
#20 0x00007fd5c6a14184 in start_thread (arg=0x7fd5bdea9700) at pthread_create.c:312
#21 0x00007fd5c5ced37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) quit

Traffic was udp dns, fatflow

Version: 3.1 Mitaka
core in /cs-shared/bugs/1619433

[Prabhjot Singh Sethi (prabhjot)]

issue potentially happened due to re-parsing of packet once with Fat-flow config and once without it, ideally we should not re-parse the packet (which internally recompute the source or dest port to 0 based on fat flow config availability)

(gdb) p key
$8 = <IPv4 nh=32 sip=192.168.121.114 dip=20.0.0.2 proto=17 sport=0 dport=53 >

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23868
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23868
Committed: http://github.org/Juniper/contrail-controller/commit/d32524d9e07040cf60aa5f81e44a5a56fcf6596c
Submitter: Zuul
Branch: master

commit d32524d9e07040cf60aa5f81e44a5a56fcf6596c
Author: Prabhjot Singh Sethi <email address hidden>
Date: Fri Sep 2 09:05:57 2016 -0700

Fix inconsistent state in flow handling from pkt

Issue:
------
pkt is already parsed and enqueue to the right flow table
partition, where while parsing it also calculate source
and destination ports based on fat flow configuration from
interface, using the flow key it identifies flow table
partition
But now in processing flow module do parsing of pkt again
where fat flow config is available on interface (which was
not available earlier) where it will end up changing the
port to zero and can cause a different table hash than the
flow event queue processing resulting in in-consistent
state and crash.

Fix:
----
remove code causing re-parsing of pkt.

Change-Id: I36aacf35d83c06e3329531d20b8dd6ad6a828f19
Closes-Bug: 1619433

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23918
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23919
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23920
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23918
Committed: http://github.org/Juniper/contrail-controller/commit/255c61854e9fe216870ae7d2a337e71414366e16
Submitter: Zuul
Branch: R3.1

commit 255c61854e9fe216870ae7d2a337e71414366e16
Author: Prabhjot Singh Sethi <email address hidden>
Date: Fri Sep 2 09:05:57 2016 -0700

Fix inconsistent state in flow handling from pkt

Issue:
------
pkt is already parsed and enqueue to the right flow table
partition, where while parsing it also calculate source
and destination ports based on fat flow configuration from
interface, using the flow key it identifies flow table
partition
But now in processing flow module do parsing of pkt again
where fat flow config is available on interface (which was
not available earlier) where it will end up changing the
port to zero and can cause a different table hash than the
flow event queue processing resulting in in-consistent
state and crash.

Fix:
----
remove code causing re-parsing of pkt.

Closes-Bug: 1619433
Change-Id: I36aacf35d83c06e3329531d20b8dd6ad6a828f19
(cherry picked from commit d32524d9e07040cf60aa5f81e44a5a56fcf6596c)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23920
Committed: http://github.org/Juniper/contrail-controller/commit/b773dc03eb7ed409639a63f889ef3804c3b3bd04
Submitter: Zuul
Branch: R3.0.2.x

commit b773dc03eb7ed409639a63f889ef3804c3b3bd04
Author: Prabhjot Singh Sethi <email address hidden>
Date: Fri Sep 2 09:05:57 2016 -0700

Fix inconsistent state in flow handling from pkt

Issue:
------
pkt is already parsed and enqueue to the right flow table
partition, where while parsing it also calculate source
and destination ports based on fat flow configuration from
interface, using the flow key it identifies flow table
partition
But now in processing flow module do parsing of pkt again
where fat flow config is available on interface (which was
not available earlier) where it will end up changing the
port to zero and can cause a different table hash than the
flow event queue processing resulting in in-consistent
state and crash.

Fix:
----
remove code causing re-parsing of pkt.

Closes-Bug: 1619433
Change-Id: I36aacf35d83c06e3329531d20b8dd6ad6a828f19
(cherry picked from commit d32524d9e07040cf60aa5f81e44a5a56fcf6596c)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23919
Committed: http://github.org/Juniper/contrail-controller/commit/550d0250e773bffb4f5a3c5e6d8d5d2c7f6ebb34
Submitter: Zuul
Branch: R3.0

commit 550d0250e773bffb4f5a3c5e6d8d5d2c7f6ebb34
Author: Prabhjot Singh Sethi <email address hidden>
Date: Fri Sep 2 09:05:57 2016 -0700

Fix inconsistent state in flow handling from pkt

Issue:
------
pkt is already parsed and enqueue to the right flow table
partition, where while parsing it also calculate source
and destination ports based on fat flow configuration from
interface, using the flow key it identifies flow table
partition
But now in processing flow module do parsing of pkt again
where fat flow config is available on interface (which was
not available earlier) where it will end up changing the
port to zero and can cause a different table hash than the
flow event queue processing resulting in in-consistent
state and crash.

Fix:
----
remove code causing re-parsing of pkt.

Closes-Bug: 1619433
Change-Id: I36aacf35d83c06e3329531d20b8dd6ad6a828f19
(cherry picked from commit d32524d9e07040cf60aa5f81e44a5a56fcf6596c)