Hi Tapan,

This crash can be reproduced consistently on scale setup. Observed again
on R2.20.81 
Copied the core @  http://mayamruga.englab.juniper.net/bugs
<http://mayamruga.englab.juniper.net/bugs/%3Cbug-ID%3E>/1453369

Setup nodei6(10.204.217.118, root/c0ntrail123) still is in issue
reproduced state.

Thanks and Regards,
Chhandak 

On 6/15/15, 10:32 PM, "OpenContrail Admin" <email address hidden>
wrote:

>** Changed in: juniperopenstack/r2.20
>    Milestone: r2.20-fcs => r2.21
>
>** Tags added: quench
>
>-- 
>You received this bug notification because you are a member of Contrail
>Systems engineering, which is subscribed to Juniper Openstack.
>https://bugs.launchpad.net/bugs/1453369
>
>Title:
>  [Build R2.20 10 Juno] TOR Scale:Control node crash @
>  IFMapExporter::StateUpdateOnDequeue
>
>Status in Juniper Openstack distribution:
>  New
>Status in Juniper Openstack r2.20 series:
>  New
>Status in Juniper Openstack trunk series:
>  New
>
>Bug description:
>  
>  Trigger
>  -------------
>  System had around 16K Vmi. Deleting all of them in one go with UI.
>
>  Backtrace
>  ----------------
>  (gdb) bt
>  #0  0x00007f547dd66cc9 in __GI_raise (sig=sig@entry=6) at
>../nptl/sysdeps/unix/sysv/linux/raise.c:56
>  #1  0x00007f547dd6a0d8 in __GI_abort () at abort.c:89
>  #2  0x00007f547dd5fb86 in __assert_fail_base (fmt=0x7f547deb0830
>"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
>assertion=assertion@entry=0xadebe1 "state->advertised().empty()",
>      file=file@entry=0xadec78 "controller/src/ifmap/ifmap_exporter.cc",
>line=line@entry=548,
>      function=function@entry=0xadf000
><IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&,
>bool)::__PRETTY_FUNCTION__> "void
>IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)")
>at assert.c:92
>  #3  0x00007f547dd5fc32 in __GI___assert_fail (assertion=0xadebe1
>"state->advertised().empty()", file=0xadec78
>"controller/src/ifmap/ifmap_exporter.cc", line=548,
>      function=0xadf000
><IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&,
>bool)::__PRETTY_FUNCTION__> "void
>IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)")
>at assert.c:101
>  #4  0x000000000045df76 in IFMapExporter::StateUpdateOnDequeue
>(this=0x2d92150, update=update@entry=0x7f545688de70, dequeue_set=...,
>is_delete=<optimized out>)
>      at controller/src/ifmap/ifmap_exporter.cc:548
>  #5  0x000000000048add2 in IFMapUpdateSender::ProcessUpdate
>(this=this@entry=0x2d93600, update=update@entry=0x7f545688de70,
>base_send_set=...)
>      at controller/src/ifmap/ifmap_update_sender.cc:225
>  #6  0x000000000048b314 in IFMapUpdateSender::Send (this=0x2d93600,
>imarker=<optimized out>) at
>controller/src/ifmap/ifmap_update_sender.cc:184
>  #7  0x000000000048badb in IFMapUpdateSender::SendTask::Run
>(this=0x2dcf520) at controller/src/ifmap/ifmap_update_sender.cc:41
>  #8  0x0000000000ab2ad0 in TaskImpl::execute (this=0x7f54774c2040) at
>controller/src/base/task.cc:232
>  #9  0x00007f547eb3db3a in ?? () from /usr/lib/libtbb.so.2
>  #10 0x00007f547eb39816 in ?? () from /usr/lib/libtbb.so.2
>  #11 0x00007f547eb38f4b in ?? () from /usr/lib/libtbb.so.2
>  #12 0x00007f547eb350ff in ?? () from /usr/lib/libtbb.so.2
>  #13 0x00007f547eb352f9 in ?? () from /usr/lib/libtbb.so.2
>  #14 0x00007f547ed59182 in start_thread (arg=0x7f544dbf6700) at
>pthread_create.c:312
>  #15 0x00007f547de2a47d in clone () at
>../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>  (gdb) quit
>
>  
>  Contrail-control log during crash
>  -------------------------------------------------
>  2015-05-09 Sat 14:54:14:617.374 IST  nodei6 [Thread 140000083560192,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh SEND Notification with Code
>2 and SubCode 7 ( OPEN Message Error:Unsupported Capability )
>controller/src/bgp/bgp_session.cc 96
>  2015-05-09 Sat 14:54:35:205.862 IST  nodei6 [Thread 140000045774592,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog:  Mode Server:  Event: Tcp
>Connection Closed  peer ip: 192.168.22.4 ( nodei9-1 )
>controller/src/xmpp/xmpp_state_machine.cc 1322
>  2015-05-09 Sat 14:54:42:809.939 IST  nodei6 [Thread 140000710457088,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog:  Mode Server:  PassiveOpen
>in state: Idle peer ip: 192.168.22.4 (  )
>controller/src/xmpp/xmpp_state_machine.cc 1335
>  2015-05-09 Sat 14:54:50:564.235 IST  nodei6 [Thread 140000718853888,
>Pid 18669]: BGP [SYS_WARN]: BgpPeerMessageLog: BGP Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh RECV Unsupported Capability:
>MpExtension (1) controller/src/bgp/bgp_proto.cc 112
>  2015-05-09 Sat 14:54:50:564.660 IST  nodei6 [Thread 140000718853888,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh RECV Notification with Code
>6 and SubCode 5 ( Cease:Connection is rejected by the peer )
>controller/src/bgp/state_machine.cc 307
>  2015-05-09 Sat 14:54:50:564.968 IST  nodei6 [Thread 140000718853888,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh SEND Notification with Code
>2 and SubCode 7 ( OPEN Message Error:Unsupported Capability )
>controller/src/bgp/bgp_session.cc 96
>  2015-05-09 Sat 14:55:29:444.147 IST  nodei6 [Thread 140000096155392,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog:  Mode Server:  Event: Tcp
>Connection Closed  peer ip: 192.168.22.5 ( nodei10 )
>controller/src/xmpp/xmpp_state_machine.cc 1322
>  2015-05-09 Sat 14:55:31:681.300 IST  nodei6 [Thread 140000731449088,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:nodei8 RECV Notification with Code
>6 and SubCode 3 ( Cease:Administrator has unconfigured the peer )
>controller/src/bgp/state_machine.cc 307
>  2015-05-09 Sat 14:55:39:548.516 IST  nodei6 [Thread 140000752441088,
>Pid 18669]: BGP [SYS_WARN]: BgpPeerMessageLog: BGP Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh RECV Unsupported Capability:
>MpExtension (1) controller/src/bgp/bgp_proto.cc 112
>  2015-05-09 Sat 14:55:39:548.843 IST  nodei6 [Thread 140000752441088,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh RECV Notification with Code
>6 and SubCode 5 ( Cease:Connection is rejected by the peer )
>controller/src/bgp/state_machine.cc 307
>  2015-05-09 Sat 14:55:39:549.138 IST  nodei6 [Thread 140000752441088,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh SEND Notification with Code
>2 and SubCode 7 ( OPEN Message Error:Unsupported Capability )
>controller/src/bgp/bgp_session.cc 96
>  2015-05-09 Sat 14:55:39:597.924 IST  nodei6 [Thread 140000714655488,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog:  Mode Server:  PassiveOpen
>in state: Idle peer ip: 192.168.22.5 (  )
>controller/src/xmpp/xmpp_state_machine.cc 1335
>  2015-05-09 Sat 14:55:40:694.229 IST  nodei6 [Thread 140000054171392,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog:  Mode Server:  Event: Tcp
>Connection Closed  peer ip: 192.168.22.4 ( nodei9 )
>controller/src/xmpp/xmpp_state_machine.cc 1322
>  2015-05-09 Sat 14:55:46:241.071 IST  nodei6 [Thread 140000037377792,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog:  Mode Server:  Event: Tcp
>Connection Closed  peer ip: 192.168.22.4 ( nodei9-1 )
>controller/src/xmpp/xmpp_state_machine.cc 1322
>  2015-05-09 Sat 14:56:05:916.364 IST  nodei6 [Thread 140012986161088,
>Pid 31844]: SANDESH: Logging: DISABLED -> ENABLED
>  2015-05-09 Sat 14:56:05:916.563 IST  nodei6 [Thread 140012986161088,
>Pid 31844]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_NOTICE ]
>log4level: [ TRACE ] -> [ WARN ]
>  2015-05-09 Sat 14:56:05:917.715 IST  nodei6 [Thread 140012986161088,
>Pid 31844]: Starting Bgp Server at port 179
>  2015-05-09 Sat 14:56:05:930.579 IST  nodei6 [Thread 140012986161088,
>Pid 31844]: SANDESH: No Client: 1431163565930447
>SandeshModuleClientTrace: data= [ name =
>nodei6:Control:contrail-control:0 client_info= [  status = Idle
>successful_connections = 0 pid = 31844 http_port = 8083 start_time =
>1431163565930223 collector_name =  primary = 0.0.0.0:0 secondary =
>0.0.0.0:0 rx_socket_stats= [  bytes = 0 calls = 0 average_bytes = 0
>blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration =
>errors = 0 ] tx_socket_stats= [  bytes = 0 calls = 0 average_bytes = 0
>blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration =
>errors = 0 ] ] ]
>                  
>                  
>                  
>
>  Testbed
>  --------------
>
>  env.roledefs = {
>      'all': [host1, host2, host3, host4, host5, host6],
>      'cfgm': [host1, host2, host3],
>      'openstack': [host1, host2, host3],
>      'webui': [host2],
>      'control': [host1, host3],
>      'compute': [host4, host5, host6],
>      'tsn': [host4, host5],
>      'toragent': [host4, host5],
>      'collector': [host1, host3],
>      'database': [host1, host2, host3],
>      'build': [host_build],
>  }
>
>  env.hostnames = {
>      'all': ['nodei6', 'nodei7', 'nodei8', 'nodei9', 'nodei10',
>'nodei19']
>  }
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/juniperopenstack/+bug/1453369/+subscriptions