control-node core at _M_data during provisioning

[Vedamurthy Joshi]

R2.20 Build 7 Ubuntu 14.04 multi-node icehouse setup

Right after provisioning, control-node had crashed on two control nodes in my cluster
Crash files and logs will be in http://10.204.216.50/Docs/bugs/#

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-control'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 _M_data (this=0xc8) at /usr/include/c++/4.8/bits/basic_string.h:293
293 { return _M_dataplus._M_p; }
(gdb) bt
#0 _M_data (this=0xc8) at /usr/include/c++/4.8/bits/basic_string.h:293
#1 begin (this=0xc8) at /usr/include/c++/4.8/bits/basic_string.h:617
#2 range_begin<std::basic_string<char> const> (c=...) at /usr/include/boost/range/begin.hpp:49
#3 begin<std::basic_string<char> > (r=...) at /usr/include/boost/range/begin.hpp:119
#4 adl_begin<std::basic_string<char> const> (r=...) at /usr/include/boost/range/iterator_range_core.hpp:58
#5 iterator_range<std::basic_string<char> > (r=..., this=<optimized out>) at /usr/include/boost/range/iterator_range_core.hpp:201
#6 make_iterator_range<std::basic_string<char> > (r=...) at /usr/include/boost/range/iterator_range_core.hpp:567
#7 make_range<std::basic_string<char> const> (r=...) at /usr/include/boost/range/as_literal.hpp:93
#8 as_literal<std::basic_string<char> > (r=...) at /usr/include/boost/range/as_literal.hpp:109
#9 equals<std::basic_string<char>, char [4], boost::algorithm::is_iequal> (Comp=..., Test=..., Input=...) at /usr/include/boost/algorithm/string/predicate.hpp:289
#10 iequals<std::basic_string<char>, char [4]> (Loc=..., Test=..., Input=...) at /usr/include/boost/algorithm/string/predicate.hpp:346
#11 KeyChainType (value=...) at controller/src/bgp/bgp_config_ifmap.cc:72
#12 BuildKeyChain (neighbor=neighbor@entry=0x7f1920022810, values=...) at controller/src/bgp/bgp_config_ifmap.cc:81
#13 0x000000000071f404 in MakeBgpNeighborConfig (instance=0x2b45ac0, remote_name=..., local_name=..., local_router=local_router@entry=0x0,
    remote_router=remote_router@entry=0x7f192000e8c0, session=session@entry=0x7f1920011890) at controller/src/bgp/bgp_config_ifmap.cc:201
#14 0x000000000071fc2e in BgpIfmapPeeringConfig::BuildNeighbors (this=this@entry=0x7f1920010f70, manager=manager@entry=0x2b29960, local_rt_config=local_rt_config@entry=0x0, peername=...,
    remote_rt_config=0x7f192000e8c0, peering=peering@entry=0x7f1920007580, map=map@entry=0x7f19371b5900) at controller/src/bgp/bgp_config_ifmap.cc:227
#15 0x00000000007201df in BgpIfmapPeeringConfig::Update (this=this@entry=0x7f1920010f70, manager=manager@entry=0x2b29960, peering=0x7f1920007580)
    at controller/src/bgp/bgp_config_ifmap.cc:271
#16 0x00000000007255dc in BgpIfmapConfigManager::ProcessBgpPeering (this=0x2b29960, delta=...) at controller/src/bgp/bgp_config_ifmap.cc:1360
#17 0x000000000071d237 in operator() (a0=..., this=0x2b29cd8) at /usr/include/boost/function/function_template.hpp:767
#18 BgpIfmapConfigManager::ProcessChanges (this=this@entry=0x2b29960, change_list=...) at controller/src/bgp/bgp_config_ifmap.cc:1374
#19 0x000000000071d2f4 in BgpIfmapConfigManager::ConfigHandler (this=0x2b29960) at controller/src/bgp/bgp_config_ifmap.cc:1388
#20 0x0000000000ab427f in operator() (this=<optimized out>) at /usr/include/boost/function/function_template.hpp:767
#21 TaskTrigger::WorkerTask::Run (this=0x7f1920011520) at controller/src/base/task_trigger.cc:19
#22 0x0000000000aaea80 in TaskImpl::execute (this=0x7f1937d77c40) at controller/src/base/task.cc:232
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-control'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 _M_data (this=0xc8) at /usr/include/c++/4.8/bits/basic_string.h:293
293 { return _M_dataplus._M_p; }
(gdb) bt
#0 _M_data (this=0xc8) at /usr/include/c++/4.8/bits/basic_string.h:293
#1 begin (this=0xc8) at /usr/include/c++/4.8/bits/basic_string.h:617
#2 range_begin<std::basic_string<char> const> (c=...) at /usr/include/boost/range/begin.hpp:49
#3 begin<std::basic_string<char> > (r=...) at /usr/include/boost/range/begin.hpp:119
#4 adl_begin<std::basic_string<char> const> (r=...) at /usr/include/boost/range/iterator_range_core.hpp:58
#5 iterator_range<std::basic_string<char> > (r=..., this=<optimized out>) at /usr/include/boost/range/iterator_range_core.hpp:201
#6 make_iterator_range<std::basic_string<char> > (r=...) at /usr/include/boost/range/iterator_range_core.hpp:567
#7 make_range<std::basic_string<char> const> (r=...) at /usr/include/boost/range/as_literal.hpp:93
#8 as_literal<std::basic_string<char> > (r=...) at /usr/include/boost/range/as_literal.hpp:109
#9 equals<std::basic_string<char>, char [4], boost::algorithm::is_iequal> (Comp=..., Test=..., Input=...) at /usr/include/boost/algorithm/string/predicate.hpp:289
#10 iequals<std::basic_string<char>, char [4]> (Loc=..., Test=..., Input=...) at /usr/include/boost/algorithm/string/predicate.hpp:346
#11 KeyChainType (value=...) at controller/src/bgp/bgp_config_ifmap.cc:72
#12 BuildKeyChain (neighbor=neighbor@entry=0x7f1920022810, values=...) at controller/src/bgp/bgp_config_ifmap.cc:81
#13 0x000000000071f404 in MakeBgpNeighborConfig (instance=0x2b45ac0, remote_name=..., local_name=..., local_router=local_router@entry=0x0,
    remote_router=remote_router@entry=0x7f192000e8c0, session=session@entry=0x7f1920011890) at controller/src/bgp/bgp_config_ifmap.cc:201
#14 0x000000000071fc2e in BgpIfmapPeeringConfig::BuildNeighbors (this=this@entry=0x7f1920010f70, manager=manager@entry=0x2b29960, local_rt_config=local_rt_config@entry=0x0, peername=...,
    remote_rt_config=0x7f192000e8c0, peering=peering@entry=0x7f1920007580, map=map@entry=0x7f19371b5900) at controller/src/bgp/bgp_config_ifmap.cc:227
#15 0x00000000007201df in BgpIfmapPeeringConfig::Update (this=this@entry=0x7f1920010f70, manager=manager@entry=0x2b29960, peering=0x7f1920007580)
    at controller/src/bgp/bgp_config_ifmap.cc:271
#16 0x00000000007255dc in BgpIfmapConfigManager::ProcessBgpPeering (this=0x2b29960, delta=...) at controller/src/bgp/bgp_config_ifmap.cc:1360
#17 0x000000000071d237 in operator() (a0=..., this=0x2b29cd8) at /usr/include/boost/function/function_template.hpp:767
#18 BgpIfmapConfigManager::ProcessChanges (this=this@entry=0x2b29960, change_list=...) at controller/src/bgp/bgp_config_ifmap.cc:1374
#19 0x000000000071d2f4 in BgpIfmapConfigManager::ConfigHandler (this=0x2b29960) at controller/src/bgp/bgp_config_ifmap.cc:1388
#20 0x0000000000ab427f in operator() (this=<optimized out>) at /usr/include/boost/function/function_template.hpp:767
#21 TaskTrigger::WorkerTask::Run (this=0x7f1920011520) at controller/src/base/task_trigger.cc:19
#22 0x0000000000aaea80 in TaskImpl::execute (this=0x7f1937d77c40) at controller/src/base/task.cc:232

[Tapan Karwa (tkarwa)]

http://10.204.216.50/Docs/bugs/1450330/

/github-build/R2.20/7/ubuntu-14-04/icehouse/store/sandbox/build/production/control-node/contrail-control

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/9796
Submitter: Tapan Karwa (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/9797
Submitter: Tapan Karwa (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/9797
Committed: http://github.org/Juniper/contrail-controller/commit/5cb7b1292a85684666dd083095e0d137e30b3a18
Submitter: Zuul
Branch: R2.20

commit 5cb7b1292a85684666dd083095e0d137e30b3a18
Author: Tapan Karwa <email address hidden>
Date: Thu Apr 30 15:46:19 2015 -0700

We need to check if the local bgp_router has its object and PARAMETERS before
accessing it.

Change-Id: I5edda0f588cf3fc0a72f7fe76eda93db3f37eaca
Closes-Bug: 1450330

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/9796
Committed: http://github.org/Juniper/contrail-controller/commit/c50b4a07e222b1b1590f0c61bfa8dc49d8b78b69
Submitter: Zuul
Branch: master

commit c50b4a07e222b1b1590f0c61bfa8dc49d8b78b69
Author: Tapan Karwa <email address hidden>
Date: Thu Apr 30 15:46:19 2015 -0700

We need to check if the local bgp_router has its object and PARAMETERS before
accessing it.

Change-Id: I5edda0f588cf3fc0a72f7fe76eda93db3f37eaca
Closes-Bug: 1450330