Stabilize bgp_stress_test

Bug #1563673 reported by Ananth Suryanarayana
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
Trunk
Fix Committed
Medium
Ananth Suryanarayana

Bug Description

This could be due to recent GR changes, but not entirely sure. Happens once every 4 runs or so.

(gdb) bt
#0 0x00002aaaacdfacc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00002aaaacdfe0d8 in __GI_abort () at abort.c:89
#2 0x00002aaaacdf3b86 in __assert_fail_base (fmt=0x2aaaacf453d0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x1f20dfc "!trigger_", file=file@entry=0x1f20dd8 "controller/src/base/task_trigger.cc", line=line@entry=42, function=function@entry=0x1f20e70 <TaskTrigger::~TaskTrigger()::__PRETTY_FUNCTION__> "TaskTrigger::~TaskTrigger()") at assert.c:92
#3 0x00002aaaacdf3c32 in __GI___assert_fail (assertion=0x1f20dfc "!trigger_", file=0x1f20dd8 "controller/src/base/task_trigger.cc", line=42, function=0x1f20e70 <TaskTrigger::~TaskTrigger()::__PRETTY_FUNCTION__> "TaskTrigger::~TaskTrigger()") at assert.c:101
#4 0x0000000001c545ea in TaskTrigger::~TaskTrigger (this=0x2aab3c0069f0, __in_chrg=<optimized out>) at controller/src/base/task_trigger.cc:42
#5 0x00000000014ce93b in BgpPeer::~BgpPeer (this=0x2aab3c006960, __in_chrg=<optimized out>) at controller/src/bgp/bgp_peer.cc:438
#6 0x00000000012dc89c in BgpPeerTest::~BgpPeerTest (this=0x2aab3c006960, __in_chrg=<optimized out>) at controller/src/bgp/test/bgp_server_test_util.cc:197
#7 0x00000000012dc932 in BgpPeerTest::~BgpPeerTest (this=0x2aab3c006960, __in_chrg=<optimized out>) at controller/src/bgp/test/bgp_server_test_util.cc:198
#8 0x0000000001c94675 in PeerManager::DestroyIPeer (this=0x26eda50, ipeer=0x2aab3c006960) at controller/src/bgp/routing-instance/peer_manager.cc:113
#9 0x00000000012ddabf in PeerManagerTest::DestroyIPeer (this=0x26eda50, ipeer=0x2aab3c006960) at controller/src/bgp/test/bgp_server_test_util.cc:297
#10 0x00000000014f0e31 in BgpPeer::DeleteActor::Destroy (this=0x2aab3c007970) at controller/src/bgp/bgp_peer.cc:240
#11 0x0000000001c30382 in LifetimeManager::DeleteExecutor (this=0x26db220, actor_ref=...) at controller/src/base/lifetime.cc:229
#12 0x0000000001c33b52 in boost::_mfi::mf1<bool, LifetimeManager, LifetimeManager::LifetimeActorRef>::operator() (this=0x2aaab4ed0b18, p=0x26db220, a1=...) at /usr/include/boost/bind/mem_fn_template.hpp:165
#13 0x0000000001c3352e in boost::_bi::list2<boost::_bi::value<LifetimeManager*>, boost::arg<1> >::operator()<bool, boost::_mfi::mf1<bool, LifetimeManager, LifetimeManager::LifetimeActorRef>, boost::_bi::list1<LifetimeManager::LifetimeActorRef&> > (this=0x2aaab4ed0b28, f=..., a=...) at /usr/include/boost/bind/bind.hpp:303
#14 0x0000000001c32ef0 in boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, LifetimeManager, LifetimeManager::LifetimeActorRef>, boost::_bi::list2<boost::_bi::value<LifetimeManager*>, boost::arg<1> > >::operator()<LifetimeManager::LifetimeActorRef> (this=0x2aaab4ed0b18, a1=...) at /usr/include/boost/bind/bind_template.hpp:32
#15 0x0000000001c329f3 in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, LifetimeManager, LifetimeManager::LifetimeActorRef>, boost::_bi::list2<boost::_bi::value<LifetimeManager*>, boost::arg<1> > >, bool, LifetimeManager::LifetimeActorRef>::invoke (function_obj_ptr=..., a0=...) at /usr/include/boost/function/function_template.hpp:132
#16 0x0000000001c3423b in boost::function1<bool, LifetimeManager::LifetimeActorRef>::operator() (this=0x2aaab4ed0b10, a0=...) at /usr/include/boost/function/function_template.hpp:767
#17 0x0000000001c33e68 in QueueTaskRunner<LifetimeManager::LifetimeActorRef, WorkQueue<LifetimeManager::LifetimeActorRef> >::RunQueue (this=0x2aab14000f00) at controller/src/base/queue_task.h:87
#18 0x0000000001c33c8a in QueueTaskRunner<LifetimeManager::LifetimeActorRef, WorkQueue<LifetimeManager::LifetimeActorRef> >::Run (this=0x2aab14000f00) at controller/src/base/queue_task.h:66
#19 0x0000000001c42ec9 in TaskImpl::execute (this=0x2aaab3a5e140) at controller/src/base/task.cc:254
#20 0x00002aaaabc13b3a in ?? () from /usr/lib/libtbb.so.2
#21 0x00002aaaabc0f816 in ?? () from /usr/lib/libtbb.so.2
#22 0x00002aaaabc0ef4b in ?? () from /usr/lib/libtbb.so.2
#23 0x00002aaaabc0b0ff in ?? () from /usr/lib/libtbb.so.2
#24 0x00002aaaabc0b2f9 in ?? () from /usr/lib/libtbb.so.2
#25 0x00002aaaab9dd182 in start_thread (arg=0x2aaab4ed1700) at pthread_create.c:312
#26 0x00002aaaacebf00d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) f
#4 0x0000000001c545ea in TaskTrigger::~TaskTrigger (this=0x2aab3c0069f0, __in_chrg=<optimized out>) at controller/src/base/task_trigger.cc:42
42 assert(!trigger_);
(gdb)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/18874
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/18874
Committed: http://github.org/Juniper/contrail-controller/commit/afc72b25130b47cf685f0cf97559f8e5373eefd9
Submitter: Zuul
Branch: master

commit afc72b25130b47cf685f0cf97559f8e5373eefd9
Author: Ananth Suryanarayana <email address hidden>
Date: Tue Mar 29 23:03:18 2016 -0700

Move bgp_stress_test to flaky_test_suite until root cause is solved

Change-Id: I86c01bcb8c9c52abd2125bd7d0ee25ee1370adb2
Partial-Bug: #1563673

Changed in juniperopenstack:
milestone: none → r3.1.0.0-fcs
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/18898
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/18898
Committed: http://github.org/Juniper/contrail-controller/commit/5a173d5568e516994ea43e71888829e33a2954a6
Submitter: Zuul
Branch: master

commit 5a173d5568e516994ea43e71888829e33a2954a6
Author: Ananth Suryanarayana <email address hidden>
Date: Wed Mar 30 12:24:15 2016 -0700

Defer bgp peer deletion by taking pre-emption into account

BGP Peer deletion should be deferred until all close paths are complete.
This is controlled by different booleans such as defer_close_ in bgp_peer.cc
and some in bgp_peer_close.cc. But, there is a time lag between when
defer_close_ is reset and when bgp_peer_close is notified to proceed
(via task trigger_). Hence, we must also check if the task is triggerd to
deduce whether the close is still in progress.

Change-Id: Ied61f970b509241a80728a8b5f0510711b86eb28
Closes-Bug: #1563673

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/40788
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/40788
Committed: http://github.com/Juniper/contrail-controller/commit/cc4307679e9693774b2fa99be92e6d2ada8e3057
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit cc4307679e9693774b2fa99be92e6d2ada8e3057
Author: Ananth Suryanarayana <email address hidden>
Date: Sun Mar 18 22:15:38 2018 -0700

Stabilize bgp_stress_test and move from flaky to normal tests list

BGP Peer Admin state was modified directly from the main thread breaking
concurrency policy (in fact, cheating it using ConcurrencyScope())

Fixed by ensuring that BGP Admin state is always modified by the script off
bgp::Config task, using task_util::TaskFire() utility function

Change-Id: I885b5fe7f2d418178b21d0d6df041653bcb13af1
Partial-Bug: 1563673

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.