CN: crash at LifetimeActor::DependencyAdd

Bug #1668369 reported by amit surana
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
High
Nischal Sheth
R3.0.3.x
Fix Committed
High
Nischal Sheth
R3.1
Fix Committed
High
Nischal Sheth
R3.2
Fix Committed
High
Nischal Sheth
Trunk
Fix Committed
High
Nischal Sheth

Bug Description

With 2k BGPaaS peers, upon restarting config services, CN crash could occur with the following bt:

#0 0x00007f996ad3cc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f996ad40028 in __GI_abort () at abort.c:89
#2 0x00007f996ad35bf6 in __assert_fail_base (fmt=0x7f996ae863b8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x110649c "!deleted_",
    file=file@entry=0x1106450 "controller/src/base/lifetime.cc", line=line@entry=129,
    function=function@entry=0x1106920 <LifetimeActor::DependencyAdd(DependencyRef<LifetimeRefBase, LifetimeActor>*)::__PRETTY_FUNCTION__> "void LifetimeActor::DependencyAdd(DependencyRef<LifetimeRefBase, LifetimeActor>*)") at assert.c:92
#3 0x00007f996ad35ca2 in __GI___assert_fail (assertion=0x110649c "!deleted_", file=0x1106450 "controller/src/base/lifetime.cc", line=129,
    function=0x1106920 <LifetimeActor::DependencyAdd(DependencyRef<LifetimeRefBase, LifetimeActor>*)::__PRETTY_FUNCTION__> "void LifetimeActor::DependencyAdd(DependencyRef<LifetimeRefBase, LifetimeActor>*)") at assert.c:101
#4 0x000000000076fa45 in LifetimeActor::DependencyAdd (this=0x7f99181c5030, node=0x7f98c4acb230) at controller/src/base/lifetime.cc:129
#5 0x00000000007701a5 in DependencyRef<LifetimeRefBase, LifetimeActor>::DependencyRef (this=0x7f98c4acb230, self=0x7f98c4acb228, ptr=0x7f99181c5030)
    at controller/src/base/dependency.h:22
#6 0x000000000076f47a in LifetimeRefBase::LifetimeRefBase (this=0x7f98c4acb228, actor=0x7f99181c5030) at controller/src/base/lifetime.cc:12
#7 0x0000000000a139c5 in LifetimeRef<BgpPeer>::LifetimeRef (this=0x7f98c4acb228, ptr=0x7f98c4ac2ff0, actor=0x7f99181c5030)
    at controller/src/base/lifetime.h:103
#8 0x00000000009eb032 in BgpPeer::BgpPeer (this=0x7f98c4ac2ff0, server=0x26e1e70, instance=0x7f991892c1b0, config=0x7f98c4b15950)
    at controller/src/bgp/bgp_peer.cc:598
#9 0x00000000009af838 in boost::factory<BgpPeer*, int boost::detail::none_helper::*, (boost::factory_alloc_propagation)0>::operator()<BgpServer*, RoutingInstance*, BgpNeighborConfig const*> (this=0x2686948, a0=@0x7f98f2bf8be0: 0x26e1e70, a1=@0x7f98f2bf8bd8: 0x7f991892c1b0, a2=@0x7f98f2bf8bd0: 0x7f98c4b15950)
    at /usr/include/boost/functional/factory.hpp:141
#10 0x00000000009ad67e in boost::detail::function::function_obj_invoker3<boost::factory<BgpPeer*, int boost::detail::none_helper::*, (boost::factory_alloc_propagation)0>, BgpPeer*, BgpServer*, RoutingInstance*, BgpNeighborConfig const*>::invoke (function_obj_ptr=..., a0=0x26e1e70, a1=0x7f991892c1b0, a2=0x7f98c4b15950)
    at /usr/include/boost/function/function_template.hpp:132
#11 0x00000000006699d9 in boost::function3<BgpPeer*, BgpServer*, RoutingInstance*, BgpNeighborConfig const*>::operator() (this=0x2686940, a0=0x26e1e70,
    a1=0x7f991892c1b0, a2=0x7f98c4b15950) at /usr/include/boost/function/function_template.hpp:767
#12 0x000000000066839b in BgpObjectFactory::Create<BgpPeer> (a0=0x26e1e70, a1=0x7f991892c1b0, a2=0x7f98c4b15950) at controller/src/bgp/bgp_factory.h:66
#13 0x000000000065afc7 in PeerManager::PeerLocate (this=0x7f992c3499c0, server=0x26e1e70, config=0x7f98c4b15950)
    at controller/src/bgp/routing-instance/peer_manager.cc:41
#14 0x0000000000afcdff in BgpServer::ConfigUpdater::ProcessNeighborConfig (this=0x26ea690, neighbor_config=0x7f98c4b15950, event=BgpConfigManager::CFG_ADD)
    at controller/src/bgp/bgp_server.cc:202
#15 0x0000000000b08b98 in boost::_mfi::mf2<void, BgpServer::ConfigUpdater, BgpNeighborConfig const*, BgpConfigManager::EventType>::operator() (
    this=0x7f98f2bf9428, p=0x26ea690, a1=0x7f98c4b15950, a2=BgpConfigManager::CFG_ADD) at /usr/include/boost/bind/mem_fn_template.hpp:280
#16 0x0000000000b07d88 in boost::_bi::list3<boost::_bi::value<BgpServer::ConfigUpdater*>, boost::arg<1>, boost::arg<2> >::operator()<boost::_mfi::mf2<void, BgpServer::ConfigUpdater, BgpNeighborConfig const*, BgpConfigManager::EventType>, boost::_bi::list2<BgpNeighborConfig const*&, BgpConfigManager::EventType&> > (
    this=0x7f98f2bf9438, f=..., a=...) at /usr/include/boost/bind/bind.hpp:392
#17 0x0000000000b0710e in boost::_bi::bind_t<void, boost::_mfi::mf2<void, BgpServer::ConfigUpdater, BgpNeighborConfig const*, BgpConfigManager::EventType>, boost::_bi::list3<boost::_bi::value<BgpServer::ConfigUpdater*>, boost::arg<1>, boost::arg<2> > >::operator()<BgpNeighborConfig const*, BgpConfigManager::EventType>
    (this=0x7f98f2bf9428, a1=@0x7f98f2bf92f0: 0x7f98c4b15950, a2=@0x7f98f2bf92ec: BgpConfigManager::CFG_ADD) at /usr/include/boost/bind/bind_template.hpp:61
#18 0x0000000000b063f2 in boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, BgpServer::ConfigUpdater, BgpNeighborConfig const*, BgpConfigManager::EventType>, boost::_bi::list3<boost::_bi::value<BgpServer::ConfigUpdater*>, boost::arg<1>, boost::arg<2> > >, void, BgpNeighborConfig const*, BgpConfigManager::EventType>::invoke (function_obj_ptr=..., a0=0x7f98c4b15950, a1=BgpConfigManager::CFG_ADD)
    at /usr/include/boost/function/function_template.hpp:153
#19 0x0000000000990860 in boost::function2<void, BgpNeighborConfig const*, BgpConfigManager::EventType>::operator() (this=0x7f98f2bf9420, a0=0x7f98c4b15950,
    a1=BgpConfigManager::CFG_ADD) at /usr/include/boost/function/function_template.hpp:767
#20 0x000000000098e5a7 in BgpConfigManager::Notify<BgpNeighborConfig> (this=0x26ea190, config=0x7f98c4b15950, event=BgpConfigManager::CFG_ADD)
    at controller/src/bgp/bgp_config.cc:471
#21 0x0000000000bb2f22 in BgpIfmapInstanceConfig::AddNeighbor (this=0x7f98c4b09750, manager=0x26ea190, neighbor=0x7f98c4b15950)
    at controller/src/bgp/bgp_config_ifmap.cc:1158

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/29147
Submitter: Nischal Sheth (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/29148
Submitter: Nischal Sheth (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0.3.x

Review in progress for https://review.opencontrail.org/29149
Submitter: Nischal Sheth (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/29150
Submitter: Nischal Sheth (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/29151
Submitter: Nischal Sheth (<email address hidden>)

Nischal Sheth (nsheth)
tags: added: bgpaas
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/29150
Committed: http://github.org/Juniper/contrail-controller/commit/c0f71cbb8c7b1a900eaf35d220a4579321c6175a
Submitter: Zuul (<email address hidden>)
Branch: master

commit c0f71cbb8c7b1a900eaf35d220a4579321c6175a
Author: Nischal Sheth <email address hidden>
Date: Mon Feb 27 14:54:15 2017 -0800

Fix race condition in BgpPeer creation for non-master instances

If a new incarnation of a routing-instance and neighbors in the
routing-instance are configured before the previous incarnation of
the routing-instance has been destroyed, we end up trying to create
a neighbor that tries to take a LifetimeRef to a deleted instance.

Ignore neighbor create/update notifications if the routing-instance
is marked deleted. The neighbors under the routing-instance will get
created when the instance itself gets resurrected.

Change-Id: I93d74cf8f903fa8f6a1949c990a70481d654ecb3
Closes-Bug: 1668369

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/29179
Submitter: Nischal Sheth (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/29179
Committed: http://github.org/Juniper/contrail-controller/commit/b2ba96f20c13fe8efc4be3ac04ac148b703dc709
Submitter: Zuul (<email address hidden>)
Branch: master

commit b2ba96f20c13fe8efc4be3ac04ac148b703dc709
Author: Nischal Sheth <email address hidden>
Date: Tue Feb 28 09:20:40 2017 -0800

Add one more test for BGPaaS neighbor create/delete

Change-Id: I33067eb623efed68d189a35b1199f99d4e0f55ae
Closes-Bug: 1668369

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/29229
Submitter: Nischal Sheth (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/29151
Committed: http://github.org/Juniper/contrail-controller/commit/22eb7823343635bef73b0694c4c20e741f6a1308
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 22eb7823343635bef73b0694c4c20e741f6a1308
Author: Nischal Sheth <email address hidden>
Date: Mon Feb 27 14:54:15 2017 -0800

Fix race condition in BgpPeer creation for non-master instances

If a new incarnation of a routing-instance and neighbors in the
routing-instance are configured before the previous incarnation of
the routing-instance has been destroyed, we end up trying to create
a neighbor that tries to take a LifetimeRef to a deleted instance.

Ignore neighbor create/update notifications if the routing-instance
is marked deleted. The neighbors under the routing-instance will get
created when the instance itself gets resurrected.

Change-Id: I93d74cf8f903fa8f6a1949c990a70481d654ecb3
Closes-Bug: 1668369

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/29147
Committed: http://github.org/Juniper/contrail-controller/commit/53f90e5baf6772e7d5ce05434501bbbff67af6a7
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 53f90e5baf6772e7d5ce05434501bbbff67af6a7
Author: Nischal Sheth <email address hidden>
Date: Mon Feb 27 11:27:58 2017 -0800

Fix race condition in BgpPeer creation for non-master instances

Two problems related to BgpPeer creation in non-master instances
are fixed.

1. If a new incarnation of a routing-instance and neighbors in the
routing-instance are configured before the previous incarnation of
the routing-instance has been destroyed, we end up trying to create
a neighbor that tries to take a LifetimeRef to a deleted instance.

2. When the previous incarnation of a routing-instance is destroyed
and a new incarnaiton is created, also create neighbors that belong
to the routing-instance.

Change-Id: Ic85893ed301b9221e07dfd740974b8e925835bb6
Closes-Bug: 1668369

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/29148
Committed: http://github.org/Juniper/contrail-controller/commit/af2ca41928b89452809294e5dcef940b7473b37b
Submitter: Zuul (<email address hidden>)
Branch: R3.0

commit af2ca41928b89452809294e5dcef940b7473b37b
Author: Nischal Sheth <email address hidden>
Date: Mon Feb 27 11:27:58 2017 -0800

Fix race condition in BgpPeer creation for non-master instances

Two problems related to BgpPeer creation in non-master instances
are fixed.

1. If a new incarnation of a routing-instance and neighbors in the
routing-instance are configured before the previous incarnation of
the routing-instance has been destroyed, we end up trying to create
a neighbor that tries to take a LifetimeRef to a deleted instance.

2. When the previous incarnation of a routing-instance is destroyed
and a new incarnaiton is created, also create neighbors that belong
to the routing-instance.

Change-Id: Ic85893ed301b9221e07dfd740974b8e925835bb6
Closes-Bug: 1668369

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/29229
Committed: http://github.org/Juniper/contrail-controller/commit/16b0ef5291af38c5f429646d64346d0e53ec27ed
Submitter: Zuul (<email address hidden>)
Branch: master

commit 16b0ef5291af38c5f429646d64346d0e53ec27ed
Author: Nischal Sheth <email address hidden>
Date: Wed Mar 1 12:33:46 2017 -0800

Add more verification in BGPaaS instance resurrection tests

Change-Id: I4ff6d339734d49c4c358778762700110d7f4428c
Closes-Bug: 1668369

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/29149
Committed: http://github.org/Juniper/contrail-controller/commit/c60e6ab6f18f63750db312d75d94824567d598f3
Submitter: Zuul (<email address hidden>)
Branch: R3.0.3.x

commit c60e6ab6f18f63750db312d75d94824567d598f3
Author: Nischal Sheth <email address hidden>
Date: Mon Feb 27 11:27:58 2017 -0800

Fix race condition in BgpPeer creation for non-master instances

Two problems related to BgpPeer creation in non-master instances
are fixed.

1. If a new incarnation of a routing-instance and neighbors in the
routing-instance are configured before the previous incarnation of
the routing-instance has been destroyed, we end up trying to create
a neighbor that tries to take a LifetimeRef to a deleted instance.

2. When the previous incarnation of a routing-instance is destroyed
and a new incarnaiton is created, also create neighbors that belong
to the routing-instance.

Change-Id: Ic85893ed301b9221e07dfd740974b8e925835bb6
Closes-Bug: 1668369

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.