agent crash at FlowEntry::LogFlow

[Vedamurthy Joshi]

R3.0.2.0 Build 28 Ubuntu 14.04 Kilo

Was seen on a setup with 1M flow table and sending n X n hping traffic between vms
Core will be in http://10.204.216.50/Docs/bugs/#
Log level was set to SYS_DEBUG on this node in agent conf file

(gdb) bt
#0 0x0000000000c239c0 in FlowEntry::LogFlow (this=this@entry=0x7f35b20756e0, event=event@entry=FlowEventLog::FLOW_EVICT, ksync=0x0, flow_handle=302301, gen_id=gen_id@entry=8 '\b')
    at controller/src/vnsw/agent/pkt/flow_entry.cc:2385
#1 0x0000000000cdc65d in KSyncFlowIndexManager::AcquireIndexUnLocked (this=this@entry=0x7f364c014950, index=<optimized out>, gen_id=<optimized out>, flow=flow@entry=0x7f35b20756e0)
    at controller/src/vnsw/agent/vrouter/ksync/ksync_flow_index_manager.cc:208
#2 0x0000000000cdcc18 in KSyncFlowIndexManager::CreateInternal (this=this@entry=0x7f364c014950, flow=flow@entry=0x7f35b20756e0)
    at controller/src/vnsw/agent/vrouter/ksync/ksync_flow_index_manager.cc:240
#3 0x0000000000cdcd6e in KSyncFlowIndexManager::Update (this=0x7f364c014950, flow=0x7f35b20756e0) at controller/src/vnsw/agent/vrouter/ksync/ksync_flow_index_manager.cc:66
#4 0x0000000000c291e5 in FlowTable::AddInternal (this=0x7f3640df2800, flow_req=0x7f35b20756e0, flow=0x7f35b20756e0, rflow_req=0x0, rflow=<optimized out>, fwd_flow_update=<optimized out>,
    rev_flow_update=false) at controller/src/vnsw/agent/pkt/flow_table.cc:291
#5 0x0000000000c29403 in FlowTable::Add (this=0x7f3640df2800, flow=flow@entry=0x7f35b20756e0, rflow=rflow@entry=0x0) at controller/src/vnsw/agent/pkt/flow_table.cc:160
#6 0x0000000000c3f517 in FlowProto::FlowEventHandler (this=<optimized out>, req=0x7f35a54f43e0, table=0x7f3640df2800) at controller/src/vnsw/agent/pkt/flow_proto.cc:420
#7 0x0000000000c45c9f in operator() (a0=0x7f35a54f43e0, this=0x7f3662ffab30) at /usr/include/boost/function/function_template.hpp:767
#8 RunQueue (this=0x7f36448cb6e0) at controller/src/base/queue_task.h:87
#9 QueueTaskRunner<FlowEvent*, WorkQueue<FlowEvent*> >::Run (this=0x7f36448cb6e0) at controller/src/base/queue_task.h:66
#10 0x00000000011a9134 in TaskImpl::execute (this=0x7f366899bd40) at controller/src/base/task.cc:259
#11 0x00007f366fd52b3a in ?? () from /usr/lib/libtbb.so.2
#12 0x00007f366fd4e816 in ?? () from /usr/lib/libtbb.so.2
#13 0x00007f366fd4df4b in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f366fd4a0ff in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f366fd4a2f9 in ?? () from /usr/lib/libtbb.so.2
#16 0x00007f366ff6e182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#17 0x00007f366f24747d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)

[Prabhjot Singh Sethi (prabhjot)]

issue happens with logging when an entry was evicted before processing the AddReq. even though the states are handled properly Log API ends up accessing NULL pointer

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19724
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19725
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19724
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19725
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19725
Committed: http://github.org/Juniper/contrail-controller/commit/b829113b47e8bdaa043bc75e3782e0e6bcd24d16
Submitter: Zuul
Branch: R3.0

commit b829113b47e8bdaa043bc75e3782e0e6bcd24d16
Author: Prabhjot Singh Sethi <email address hidden>
Date: Fri Apr 29 08:40:56 2016 -0700

Fix NULL ptr access while logging

Issue:
------
Flow Add req was processed after the flow was replaced
with later gen-id, causing the flow to raise an evict
event while creating the ksync entry itself, where
LogFlow is called with NULL KSyncEntry pointer which
doesnot have NULL check safety

Fix:
----
Add NULL check safe gaurd for KSyncEntry pointer
while logging

Closes-Bug: 1573917
Change-Id: I32eb720ee0ca335b8a75544896bc781fd663a135
(cherry picked from commit afc1a6cc15e866dfc387745688f99528ca67ca7f)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19724
Committed: http://github.org/Juniper/contrail-controller/commit/afc1a6cc15e866dfc387745688f99528ca67ca7f
Submitter: Zuul
Branch: master

commit afc1a6cc15e866dfc387745688f99528ca67ca7f
Author: Prabhjot Singh Sethi <email address hidden>
Date: Fri Apr 29 08:40:56 2016 -0700

Fix NULL ptr access while logging

Issue:
------
Flow Add req was processed after the flow was replaced
with later gen-id, causing the flow to raise an evict
event while creating the ksync entry itself, where
LogFlow is called with NULL KSyncEntry pointer which
doesnot have NULL check safety

Fix:
----
Add NULL check safe gaurd for KSyncEntry pointer
while logging

Closes-Bug: 1573917
Change-Id: I32eb720ee0ca335b8a75544896bc781fd663a135