3.0: Agent crash @ fd_ == VmFlowRef::kInvalidFd"

[amit surana]

Solution scripts doing flow setup/teardown.

3.0 b2712

core file location: 10.84.5.112:/cs-shared/bugs/<bug-id>

ImportError: No module named 'libstdcxx'
(gdb) bt
#0 0x00007f28e8d21cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f28e8d250d8 in __GI_abort () at abort.c:89
#2 0x00007f28e8d1ab86 in __assert_fail_base (fmt=0x7f28e8e6b830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x11fd44e "fd_ == VmFlowRef::kInvalidFd", file=file@entry=0x11fd620 "controller/src/vnsw/agent/pkt/flow_entry.cc",
    line=line@entry=116,
    function=function@entry=0x11fdb80 <VmFlowRef::operator=(VmFlowRef const&)::__PRETTY_FUNCTION__> "void VmFlowRef::operator=(const VmFlowRef&)")
    at assert.c:92
#3 0x00007f28e8d1ac32 in __GI___assert_fail (assertion=0x11fd44e "fd_ == VmFlowRef::kInvalidFd",
    file=0x11fd620 "controller/src/vnsw/agent/pkt/flow_entry.cc", line=116,
    function=0x11fdb80 <VmFlowRef::operator=(VmFlowRef const&)::__PRETTY_FUNCTION__> "void VmFlowRef::operator=(const VmFlowRef&)") at assert.c:101
#4 0x0000000000c0c735 in VmFlowRef::operator= (this=<optimized out>, rhs=...) at controller/src/vnsw/agent/pkt/flow_entry.cc:116
#5 0x0000000000c12b75 in operator= (this=0x7f28cde78900) at controller/src/vnsw/agent/pkt/flow_entry.h:218
#6 FlowEntry::Copy (this=this@entry=0x7f28cde788a0, rhs=rhs@entry=0x7f28cdea1020) at controller/src/vnsw/agent/pkt/flow_entry.cc:371
#7 0x0000000000c1923e in Copy (rhs=0x7f28cdea1020, lhs=0x7f28cde788a0, this=0x7f28cc4488b0) at controller/src/vnsw/agent/pkt/flow_table.cc:117
#8 FlowTable::AddInternal (this=0x7f28cc4488b0, flow_req=0x7f28cdea1020, flow=0x7f28cde788a0, rflow_req=0x7f28cdea0b80, rflow=0x7f28cdea0b80,
    update=<optimized out>) at controller/src/vnsw/agent/pkt/flow_table.cc:165
#9 0x0000000000c19430 in FlowTable::Add (this=this@entry=0x7f28cc4488b0, flow_req=flow_req@entry=0x7f28cdea1020, flow=flow@entry=0x7f28cde788a0,
    rflow_req=rflow_req@entry=0x7f28cdea0b80, rflow=0x7f28cdea0b80, update=update@entry=false) at controller/src/vnsw/agent/pkt/flow_table.cc:232
#10 0x0000000000c1954f in FlowTable::Add (this=0x7f28cc4488b0, flow=flow@entry=0x7f28cdea1020, rflow=0x7f28cdea0b80)
    at controller/src/vnsw/agent/pkt/flow_table.cc:137
#11 0x0000000000c2ea54 in FlowProto::AddFlow (this=<optimized out>, flow=flow@entry=0x7f28cdea1020) at controller/src/vnsw/agent/pkt/flow_proto.cc:177
#12 0x0000000000c3a4f1 in PktFlowInfo::Add (this=this@entry=0x7f28c05726c0, pkt=<optimized out>, in=in@entry=0x7f28c0572640, out=out@entry=0x7f28c0572680)
    at controller/src/vnsw/agent/pkt/pkt_flow_info.cc:1732
#13 0x0000000000c4824c in FlowHandler::Run (this=0x7f28701756b0) at controller/src/vnsw/agent/pkt/flow_handler.cc:128
#14 0x0000000000c44848 in RunProtoHandler (handler=0x7f28701756b0, this=0x7f28c0572880) at controller/src/vnsw/agent/pkt/proto.cc:51
#15 Proto::ProcessProto (this=this@entry=0x7f28cc469760, msg_info=...) at controller/src/vnsw/agent/pkt/proto.cc:67
#16 0x0000000000c2e572 in FlowProto::FlowEventHandler (this=0x7f28cc469760, req=..., table=0x7f28cc4488b0)
    at controller/src/vnsw/agent/pkt/flow_proto.cc:263
#17 0x0000000000c338f8 in operator() (a0=<error reading variable: access outside bounds of object referenced via synthetic pointer>, this=0x7f28c0572900)
    at /usr/include/boost/function/function_template.hpp:767
#18 QueueTaskRunner<FlowEvent, WorkQueue<FlowEvent> >::RunQueue (this=0x1fb5d10) at controller/src/base/queue_task.h:87
#19 0x00000000011793ec in TaskImpl::execute (this=0x7f28e258bc40) at controller/src/base/task.cc:253
#20 0x00007f28e98f0b3a in ?? () from /usr/lib/libtbb.so.2
#21 0x00007f28e98ec816 in ?? () from /usr/lib/libtbb.so.2
#22 0x00007f28e98ebf4b in ?? () from /usr/lib/libtbb.so.2
#23 0x00007f28e98e80ff in ?? () from /usr/lib/libtbb.so.2
#24 0x00007f28e98e82f9 in ?? () from /usr/lib/libtbb.so.2
#25 0x00007f28e9b0c182 in start_thread (arg=0x7f28c0573700) at pthread_create.c:312
#26 0x00007f28e8de547d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[amit surana (asurana-t)]

making this critical as its completely blocking solution script run. Agent crashes every time within a few minutes of starting traffic.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17241
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17241
Committed: http://github.org/Juniper/contrail-controller/commit/f43cfd4066649aafff74089183d7175adb8d0bfa
Submitter: Zuul
Branch: master

commit f43cfd4066649aafff74089183d7175adb8d0bfa
Author: Hari <email address hidden>
Date: Mon Feb 15 15:55:34 2016 +0530

Remove incorrect assert.

As updates on linklocal flows are not supported, an assert is added
in VmFlowRef assignment operator. But this also gets hit when vrouter
evicts the older flow and same flow is being establised again. Removing
the assert and updating the other fields.

Change-Id: If5079cd4a8a147cf82f391a036b7fb482f6a5cc1
closes-bug: 1544767