[Ubuntu R2.20.64] Control node crash @ BgpXmppChannel::XmppPeer::~XmppPeer

[chhandak]

Trigger:
Observed the crash after reboot of all the compute node

Backtrace:
(gdb) bt
#0 0x00007fea0ca98cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fea0ca9c0d8 in __GI_abort () at abort.c:89
#2 0x00007fea0ca91b86 in __assert_fail_base (fmt=0x7fea0cbe2830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xb0035e "GetRefCount() == 0",
    file=file@entry=0xb28218 "controller/src/bgp/bgp_xmpp_channel.cc", line=line@entry=323,
    function=function@entry=0xb293c0 <BgpXmppChannel::XmppPeer::~XmppPeer()::__PRETTY_FUNCTION__> "virtual BgpXmppChannel::XmppPeer::~XmppPeer()") at assert.c:92
#3 0x00007fea0ca91c32 in __GI___assert_fail (assertion=0xb0035e "GetRefCount() == 0", file=0xb28218 "controller/src/bgp/bgp_xmpp_channel.cc", line=323,
    function=0xb293c0 <BgpXmppChannel::XmppPeer::~XmppPeer()::__PRETTY_FUNCTION__> "virtual BgpXmppChannel::XmppPeer::~XmppPeer()") at assert.c:101
#4 0x000000000041a0c4 in BgpXmppChannel::XmppPeer::~XmppPeer (this=<optimized out>, __in_chrg=<optimized out>) at controller/src/bgp/bgp_xmpp_channel.cc:323
#5 0x0000000000782250 in ~XmppPeer (this=<optimized out>, __in_chrg=<optimized out>) at controller/src/bgp/bgp_xmpp_channel.cc:322
#6 BgpXmppChannel::XmppPeer::~XmppPeer (this=<optimized out>, __in_chrg=<optimized out>) at controller/src/bgp/bgp_xmpp_channel.cc:324
#7 0x0000000000776287 in checked_delete<BgpXmppChannel::XmppPeer> (x=<optimized out>) at /usr/include/boost/checked_delete.hpp:34
#8 ~scoped_ptr (this=0x7fe9bc0438f0, __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/scoped_ptr.hpp:82
#9 BgpXmppChannel::~BgpXmppChannel (this=0x7fe9bc0438d0, __in_chrg=<optimized out>) at controller/src/bgp/bgp_xmpp_channel.cc:474
#10 0x00000000007763c9 in BgpXmppChannel::~BgpXmppChannel (this=0x7fe9bc0438d0, __in_chrg=<optimized out>) at controller/src/bgp/bgp_xmpp_channel.cc:484
#11 0x000000000074c1bd in BgpXmppChannelManager::DeleteExecutor (this=<optimized out>, channel=<optimized out>) at controller/src/bgp/bgp_xmpp_channel.cc:2316
#12 0x000000000078ce25 in operator() (a0=0x7fe9bc0438d0, this=0x7fe9f57f4b40) at /usr/include/boost/function/function_template.hpp:767
#13 RunQueue (this=0x7fe7ae695410) at controller/src/base/queue_task.h:81
#14 QueueTaskRunner<BgpXmppChannel*, WorkQueue<BgpXmppChannel*> >::Run (this=0x7fe7ae695410) at controller/src/base/queue_task.h:64
#15 0x0000000000ab5720 in TaskImpl::execute (this=0x7fea0f6f2140) at controller/src/base/task.cc:238
#16 0x00007fea0d86fb3a in ?? () from /usr/lib/libtbb.so.2
#17 0x00007fea0d86b816 in ?? () from /usr/lib/libtbb.so.2
#18 0x00007fea0d86af4b in ?? () from /usr/lib/libtbb.so.2
#19 0x00007fea0d8670ff in ?? () from /usr/lib/libtbb.so.2
#20 0x00007fea0d8672f9 in ?? () from /usr/lib/libtbb.so.2
#21 0x00007fea0da8b182 in start_thread (arg=0x7fe9f57f5700) at pthread_create.c:312
#22 0x00007fea0cb5c47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Scale profile
-------------------
TOR: 32(4 QFX , rest are dummy)
VN: 64K
LIF: 64K (8K on QFX)
VMI:128K (16K on QFX)
VTEP: 4K
MAC:164K

[chhandak (chhandak)]

Logs saved at http://mayamruga.englab.juniper.net/bugs/1470838

To access to the core:

ssh to bhushana@10.204.216.50 Password bhu@123

cd /home/bhushana/Documents/technical/bugs/1470838

Ammu1803