Agent crash @ FlowMgmtEntry::Delete

[amit surana]

3.1.1.0-32

core file:

core at: 10.84.5.31:/cs-shared/bugs/<bug-id>/

#0 0x00007f6226530c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f6226534028 in __GI_abort () at abort.c:89
#2 0x00007f6226529bf6 in __assert_fail_base (fmt=0x7f622667a3b8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x1318a20 "!node_algorithms::inited(real_value_traits::to_node_ptr(value))",
    file=file@entry=0x13189f8 "/usr/include/boost/intrusive/list.hpp", line=line@entry=1275,
    function=function@entry=0x1371ac0 <boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<FlowMgmtKeyNode, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &FlowMgmtKeyNode::hook_>, unsigned long, true> >::iterator_to(FlowMgmtKeyNode&)::__PRETTY_FUNCTION__> "boost::intrusive::list_impl<Config>::iterator boost::intrusive::list_impl<Config>::iterator_to(boost::intrusive::list_impl<Config>::reference) [with Config = boost::intrusive::listopt<boost::intrusive"...) at assert.c:92
#3 0x00007f6226529ca2 in __GI___assert_fail (assertion=0x1318a20 "!node_algorithms::inited(real_value_traits::to_node_ptr(value))",
    file=0x13189f8 "/usr/include/boost/intrusive/list.hpp", line=1275,
    function=0x1371ac0 <boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<FlowMgmtKeyNode, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &FlowMgmtKeyNode::hook_>, unsigned long, true> >::iterator_to(FlowMgmtKeyNode&)::__PRETTY_FUNCTION__> "boost::intrusive::list_impl<Config>::iterator boost::intrusive::list_impl<Config>::iterator_to(boost::intrusive::list_impl<Config>::reference) [with Config = boost::intrusive::listopt<boost::intrusive"...) at assert.c:101
#4 0x00000000007fad44 in iterator_to (this=<optimized out>, value=...) at /usr/include/boost/intrusive/list.hpp:1275
#5 FlowMgmtEntry::Delete (node=<optimized out>, flow=<optimized out>, this=<optimized out>) at controller/src/vnsw/agent/pkt/flow_mgmt.cc:1055
#6 0x0000000000ce175d in FlowMgmtEntry::Delete (this=<optimized out>, flow=<optimized out>, node=<optimized out>)
    at controller/src/vnsw/agent/pkt/flow_mgmt.cc:1054
#7 0x0000000000ce4bfa in FlowMgmtTree::Delete (this=0x7f61fc8047e0, key=0x7f6204cc1ff0, flow=0x7f61fc1b0560, node=0x7f62042c1ca0)
    at controller/src/vnsw/agent/pkt/flow_mgmt.cc:985
#8 0x0000000000ce8687 in FlowMgmtManager::DeleteFlowMgmtKey (this=this@entry=0x7f61fc803f60, flow=0x7f61fc1b0560, info=info@entry=0x7f61e09f75c0,
    key=0x7f6204cc1ff0, node=node@entry=0x7f62042c1ca0) at controller/src/vnsw/agent/pkt/flow_mgmt.cc:861
#9 0x0000000000ce88c7 in FlowMgmtManager::DeleteFlow (this=this@entry=0x7f61fc803f60, flow=..., params=...)
    at controller/src/vnsw/agent/pkt/flow_mgmt.cc:664
#10 0x0000000000ce9029 in FlowMgmtManager::RequestHandler (this=0x7f61fc803f60, req=...) at controller/src/vnsw/agent/pkt/flow_mgmt.cc:440
#11 0x0000000000ceb0ec in operator() (a1=..., p=<optimized out>, this=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:165
#12 operator()<bool, boost::_mfi::mf1<bool, FlowMgmtManager, boost::shared_ptr<FlowMgmtRequest> >, boost::_bi::list1<boost::shared_ptr<FlowMgmtRequest>&> > (a=<synthetic pointer>, f=..., this=<optimized out>) at /usr/include/boost/bind/bind.hpp:303
#13 operator()<boost::shared_ptr<FlowMgmtRequest> > (a1=..., this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:32
#14 boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, FlowMgmtManager, boost::shared_ptr<FlowMgmtRequest> >, boost::_bi::list2<boost::_bi::value<FlowMgmtManager*>, boost::arg<1> > >, bool, boost::shared_ptr<FlowMgmtRequest> >::invoke (function_obj_ptr=..., a0=...)
    at /usr/include/boost/function/function_template.hpp:132
#15 0x0000000000cf0bd4 in operator() (a0=..., this=0x7f61f0d7aaa0) at /usr/include/boost/function/function_template.hpp:767
#16 QueueTaskRunner<boost::shared_ptr<FlowMgmtRequest>, WorkQueue<boost::shared_ptr<FlowMgmtRequest> > >::RunQueue (this=0x7f62047e8060)
    at controller/src/base/queue_task.h:92
#17 0x00000000012e3c8f in TaskImpl::execute (this=0x7f621fc26140) at controller/src/base/task.cc:262
#18 0x00007f62270ffb3a in ?? () from /usr/lib/libtbb.so.2
#19 0x00007f62270fb816 in ?? () from /usr/lib/libtbb.so.2
#20 0x00007f62270faf4b in ?? () from /usr/lib/libtbb.so.2
#21 0x00007f62270f70ff in ?? () from /usr/lib/libtbb.so.2
#22 0x00007f62270f72f9 in ?? () from /usr/lib/libtbb.so.2
#23 0x00007f622731b184 in start_thread (arg=0x7f61f0d7b700) at pthread_create.c:312
#24 0x00007f62265f437d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25636
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25636
Committed: http://github.org/Juniper/contrail-controller/commit/ea23d2128217d6939b1b03f70197dc1023818b87
Submitter: Zuul
Branch: master

commit ea23d2128217d6939b1b03f70197dc1023818b87
Author: Manish Singh <email address hidden>
Date: Thu Nov 3 12:43:35 2016 +0530

Agent crash @ FlowMgmtEntry::Delete

Problem:
Note that Flow management module and controller update module run in parallel.
In BGP service flows, to identify flow mgmt tree, CN info is needed.
When a bgp service flow is getting added keys are extracted and then flow is
added. Both these routines use GetCNIndex to identify which bgp service flow
mgmt tree to use. This can cause issue as extraction of keys may have taken some
index but by the time flow node gets added in tree CN which has been picked up
is down. In turn GetCNIndex will return invalid index and flow node does not get
added. Because of CN delete, flow delete request would have been pushed. On
serving delete request key is found but flow node is absent. This results in
crash.

Solution:
During addition of flow node use CN index from key instead of calling
GetCNIndex.

Change-Id: Ifd9adac9cdb4a03e621cf62eff009e3e55d882a5
Closes-bug: #1630011

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25761
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25762
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25761
Committed: http://github.org/Juniper/contrail-controller/commit/8a6b5eaabcba3d836f584dde0a59300b1b7387e6
Submitter: Zuul
Branch: R3.2

commit 8a6b5eaabcba3d836f584dde0a59300b1b7387e6
Author: Manish Singh <email address hidden>
Date: Thu Nov 3 12:43:35 2016 +0530

Agent crash @ FlowMgmtEntry::Delete

Problem:
Note that Flow management module and controller update module run in parallel.
In BGP service flows, to identify flow mgmt tree, CN info is needed.
When a bgp service flow is getting added keys are extracted and then flow is
added. Both these routines use GetCNIndex to identify which bgp service flow
mgmt tree to use. This can cause issue as extraction of keys may have taken some
index but by the time flow node gets added in tree CN which has been picked up
is down. In turn GetCNIndex will return invalid index and flow node does not get
added. Because of CN delete, flow delete request would have been pushed. On
serving delete request key is found but flow node is absent. This results in
crash.

Solution:
During addition of flow node use CN index from key instead of calling
GetCNIndex.

Change-Id: Ifd9adac9cdb4a03e621cf62eff009e3e55d882a5
Closes-bug: #1630011
(cherry picked from commit ea23d2128217d6939b1b03f70197dc1023818b87)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25762
Committed: http://github.org/Juniper/contrail-controller/commit/ba02baf9e694badd977da35c898291ed36b2d916
Submitter: Zuul
Branch: R3.1

commit ba02baf9e694badd977da35c898291ed36b2d916
Author: Manish Singh <email address hidden>
Date: Thu Nov 3 12:43:35 2016 +0530

Agent crash @ FlowMgmtEntry::Delete

Problem:
Note that Flow management module and controller update module run in parallel.
In BGP service flows, to identify flow mgmt tree, CN info is needed.
When a bgp service flow is getting added keys are extracted and then flow is
added. Both these routines use GetCNIndex to identify which bgp service flow
mgmt tree to use. This can cause issue as extraction of keys may have taken some
index but by the time flow node gets added in tree CN which has been picked up
is down. In turn GetCNIndex will return invalid index and flow node does not get
added. Because of CN delete, flow delete request would have been pushed. On
serving delete request key is found but flow node is absent. This results in
crash.

Solution:
During addition of flow node use CN index from key instead of calling
GetCNIndex.

Change-Id: Ifd9adac9cdb4a03e621cf62eff009e3e55d882a5
Closes-bug: #1630011
(cherry picked from commit ea23d2128217d6939b1b03f70197dc1023818b87)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/29096
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/29096
Committed: http://github.org/Juniper/contrail-controller/commit/8acab1fc4111777e2f0b899586c569f5adf483a3
Submitter: Zuul (<email address hidden>)
Branch: R3.0.3.x

commit 8acab1fc4111777e2f0b899586c569f5adf483a3
Author: Manish Singh <email address hidden>
Date: Thu Nov 3 12:43:35 2016 +0530

Agent crash @ FlowMgmtEntry::Delete

Problem:
Note that Flow management module and controller update module run in parallel.
In BGP service flows, to identify flow mgmt tree, CN info is needed.
When a bgp service flow is getting added keys are extracted and then flow is
added. Both these routines use GetCNIndex to identify which bgp service flow
mgmt tree to use. This can cause issue as extraction of keys may have taken some
index but by the time flow node gets added in tree CN which has been picked up
is down. In turn GetCNIndex will return invalid index and flow node does not get
added. Because of CN delete, flow delete request would have been pushed. On
serving delete request key is found but flow node is absent. This results in
crash.

Solution:
During addition of flow node use CN index from key instead of calling
GetCNIndex.

Change-Id: Ifd9adac9cdb4a03e621cf62eff009e3e55d882a5
Closes-bug: #1630011
(cherry picked from commit ea23d2128217d6939b1b03f70197dc1023818b87)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/29122
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/29122
Committed: http://github.org/Juniper/contrail-controller/commit/8f6ee4041db768af9c9b801078d88846fe84d970
Submitter: Zuul (<email address hidden>)
Branch: R3.0

commit 8f6ee4041db768af9c9b801078d88846fe84d970
Author: Manish Singh <email address hidden>
Date: Thu Nov 3 12:43:35 2016 +0530

Agent crash @ FlowMgmtEntry::Delete

Problem:
Note that Flow management module and controller update module run in parallel.
In BGP service flows, to identify flow mgmt tree, CN info is needed.
When a bgp service flow is getting added keys are extracted and then flow is
added. Both these routines use GetCNIndex to identify which bgp service flow
mgmt tree to use. This can cause issue as extraction of keys may have taken some
index but by the time flow node gets added in tree CN which has been picked up
is down. In turn GetCNIndex will return invalid index and flow node does not get
added. Because of CN delete, flow delete request would have been pushed. On
serving delete request key is found but flow node is absent. This results in
crash.

Solution:
During addition of flow node use CN index from key instead of calling
GetCNIndex.

Change-Id: Ifd9adac9cdb4a03e621cf62eff009e3e55d882a5
Closes-bug: #1630011
(cherry picked from commit ea23d2128217d6939b1b03f70197dc1023818b87)
(cherry picked from commit 8acab1fc4111777e2f0b899586c569f5adf483a3)