Juniper Openstack

  1. Series trunk
  2. Bug #1453369
  3. Comment #6

Comment 6 for bug 1453369

Revision history for this message
chhandak (chhandak) wrote : Re: [Bug 1453369] Re: [Build R2.20 10 Juno] TOR Scale:Control node crash @ IFMapExporter::StateUpdateOnDequeue

Hi Tapan.

This crash can be consistently reproducible .

When ever we delete logical interface in chunk (using webui), I have hit
the crash.

You can have a look at the crash @ nodei6 (10.204.217.118) root/c0ntrail123

Thanks and Regards,
Chhandak

On 6/15/15, 10:32 PM, "OpenContrail Admin" <email address hidden>
wrote:

>** Changed in: juniperopenstack/r2.20
> Milestone: r2.20-fcs => r2.21
>
>** Tags added: quench
>
>--
>You received this bug notification because you are a member of Contrail
>Systems engineering, which is subscribed to Juniper Openstack.
>https://bugs.launchpad.net/bugs/1453369
>
>Title:
> [Build R2.20 10 Juno] TOR Scale:Control node crash @
> IFMapExporter::StateUpdateOnDequeue
>
>Status in Juniper Openstack distribution:
> New
>Status in Juniper Openstack r2.20 series:
> New
>Status in Juniper Openstack trunk series:
> New
>
>Bug description:
>
> Trigger
> -------------
> System had around 16K Vmi. Deleting all of them in one go with UI.
>
> Backtrace
> ----------------
> (gdb) bt
> #0 0x00007f547dd66cc9 in __GI_raise (sig=sig@entry=6) at
>../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1 0x00007f547dd6a0d8 in __GI_abort () at abort.c:89
> #2 0x00007f547dd5fb86 in __assert_fail_base (fmt=0x7f547deb0830
>"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
>assertion=assertion@entry=0xadebe1 "state->advertised().empty()",
> file=file@entry=0xadec78 "controller/src/ifmap/ifmap_exporter.cc",
>line=line@entry=548,
> function=function@entry=0xadf000
><IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&,
>bool)::__PRETTY_FUNCTION__> "void
>IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)")
>at assert.c:92
> #3 0x00007f547dd5fc32 in __GI___assert_fail (assertion=0xadebe1
>"state->advertised().empty()", file=0xadec78
>"controller/src/ifmap/ifmap_exporter.cc", line=548,
> function=0xadf000
><IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&,
>bool)::__PRETTY_FUNCTION__> "void
>IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)")
>at assert.c:101
> #4 0x000000000045df76 in IFMapExporter::StateUpdateOnDequeue
>(this=0x2d92150, update=update@entry=0x7f545688de70, dequeue_set=...,
>is_delete=<optimized out>)
> at controller/src/ifmap/ifmap_exporter.cc:548
> #5 0x000000000048add2 in IFMapUpdateSender::ProcessUpdate
>(this=this@entry=0x2d93600, update=update@entry=0x7f545688de70,
>base_send_set=...)
> at controller/src/ifmap/ifmap_update_sender.cc:225
> #6 0x000000000048b314 in IFMapUpdateSender::Send (this=0x2d93600,
>imarker=<optimized out>) at
>controller/src/ifmap/ifmap_update_sender.cc:184
> #7 0x000000000048badb in IFMapUpdateSender::SendTask::Run
>(this=0x2dcf520) at controller/src/ifmap/ifmap_update_sender.cc:41
> #8 0x0000000000ab2ad0 in TaskImpl::execute (this=0x7f54774c2040) at
>controller/src/base/task.cc:232
> #9 0x00007f547eb3db3a in ?? () from /usr/lib/libtbb.so.2
> #10 0x00007f547eb39816 in ?? () from /usr/lib/libtbb.so.2
> #11 0x00007f547eb38f4b in ?? () from /usr/lib/libtbb.so.2
> #12 0x00007f547eb350ff in ?? () from /usr/lib/libtbb.so.2
> #13 0x00007f547eb352f9 in ?? () from /usr/lib/libtbb.so.2
> #14 0x00007f547ed59182 in start_thread (arg=0x7f544dbf6700) at
>pthread_create.c:312
> #15 0x00007f547de2a47d in clone () at
>../sysdeps/unix/sysv/linux/x86_64/clone.S:111
> (gdb) quit
>
>
> Contrail-control log during crash
> -------------------------------------------------
> 2015-05-09 Sat 14:54:14:617.374 IST nodei6 [Thread 140000083560192,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh SEND Notification with Code
>2 and SubCode 7 ( OPEN Message Error:Unsupported Capability )
>controller/src/bgp/bgp_session.cc 96
> 2015-05-09 Sat 14:54:35:205.862 IST nodei6 [Thread 140000045774592,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp
>Connection Closed peer ip: 192.168.22.4 ( nodei9-1 )
>controller/src/xmpp/xmpp_state_machine.cc 1322
> 2015-05-09 Sat 14:54:42:809.939 IST nodei6 [Thread 140000710457088,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: PassiveOpen
>in state: Idle peer ip: 192.168.22.4 ( )
>controller/src/xmpp/xmpp_state_machine.cc 1335
> 2015-05-09 Sat 14:54:50:564.235 IST nodei6 [Thread 140000718853888,
>Pid 18669]: BGP [SYS_WARN]: BgpPeerMessageLog: BGP Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh RECV Unsupported Capability:
>MpExtension (1) controller/src/bgp/bgp_proto.cc 112
> 2015-05-09 Sat 14:54:50:564.660 IST nodei6 [Thread 140000718853888,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh RECV Notification with Code
>6 and SubCode 5 ( Cease:Connection is rejected by the peer )
>controller/src/bgp/state_machine.cc 307
> 2015-05-09 Sat 14:54:50:564.968 IST nodei6 [Thread 140000718853888,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh SEND Notification with Code
>2 and SubCode 7 ( OPEN Message Error:Unsupported Capability )
>controller/src/bgp/bgp_session.cc 96
> 2015-05-09 Sat 14:55:29:444.147 IST nodei6 [Thread 140000096155392,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp
>Connection Closed peer ip: 192.168.22.5 ( nodei10 )
>controller/src/xmpp/xmpp_state_machine.cc 1322
> 2015-05-09 Sat 14:55:31:681.300 IST nodei6 [Thread 140000731449088,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:nodei8 RECV Notification with Code
>6 and SubCode 3 ( Cease:Administrator has unconfigured the peer )
>controller/src/bgp/state_machine.cc 307
> 2015-05-09 Sat 14:55:39:548.516 IST nodei6 [Thread 140000752441088,
>Pid 18669]: BGP [SYS_WARN]: BgpPeerMessageLog: BGP Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh RECV Unsupported Capability:
>MpExtension (1) controller/src/bgp/bgp_proto.cc 112
> 2015-05-09 Sat 14:55:39:548.843 IST nodei6 [Thread 140000752441088,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh RECV Notification with Code
>6 and SubCode 5 ( Cease:Connection is rejected by the peer )
>controller/src/bgp/state_machine.cc 307
> 2015-05-09 Sat 14:55:39:549.138 IST nodei6 [Thread 140000752441088,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer
>default-domain:default-project:ip-fabric:__default__:nodei6:default-domain
>:default-project:ip-fabric:__default__:walsh SEND Notification with Code
>2 and SubCode 7 ( OPEN Message Error:Unsupported Capability )
>controller/src/bgp/bgp_session.cc 96
> 2015-05-09 Sat 14:55:39:597.924 IST nodei6 [Thread 140000714655488,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: PassiveOpen
>in state: Idle peer ip: 192.168.22.5 ( )
>controller/src/xmpp/xmpp_state_machine.cc 1335
> 2015-05-09 Sat 14:55:40:694.229 IST nodei6 [Thread 140000054171392,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp
>Connection Closed peer ip: 192.168.22.4 ( nodei9 )
>controller/src/xmpp/xmpp_state_machine.cc 1322
> 2015-05-09 Sat 14:55:46:241.071 IST nodei6 [Thread 140000037377792,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp
>Connection Closed peer ip: 192.168.22.4 ( nodei9-1 )
>controller/src/xmpp/xmpp_state_machine.cc 1322
> 2015-05-09 Sat 14:56:05:916.364 IST nodei6 [Thread 140012986161088,
>Pid 31844]: SANDESH: Logging: DISABLED -> ENABLED
> 2015-05-09 Sat 14:56:05:916.563 IST nodei6 [Thread 140012986161088,
>Pid 31844]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_NOTICE ]
>log4level: [ TRACE ] -> [ WARN ]
> 2015-05-09 Sat 14:56:05:917.715 IST nodei6 [Thread 140012986161088,
>Pid 31844]: Starting Bgp Server at port 179
> 2015-05-09 Sat 14:56:05:930.579 IST nodei6 [Thread 140012986161088,
>Pid 31844]: SANDESH: No Client: 1431163565930447
>SandeshModuleClientTrace: data= [ name =
>nodei6:Control:contrail-control:0 client_info= [ status = Idle
>successful_connections = 0 pid = 31844 http_port = 8083 start_time =
>1431163565930223 collector_name = primary = 0.0.0.0:0 secondary =
>0.0.0.0:0 rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0
>blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration =
>errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0
>blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration =
>errors = 0 ] ] ]
>
>
>
>
> Testbed
> --------------
>
> env.roledefs = {
> 'all': [host1, host2, host3, host4, host5, host6],
> 'cfgm': [host1, host2, host3],
> 'openstack': [host1, host2, host3],
> 'webui': [host2],
> 'control': [host1, host3],
> 'compute': [host4, host5, host6],
> 'tsn': [host4, host5],
> 'toragent': [host4, host5],
> 'collector': [host1, host3],
> 'database': [host1, host2, host3],
> 'build': [host_build],
> }
>
> env.hostnames = {
> 'all': ['nodei6', 'nodei7', 'nodei8', 'nodei9', 'nodei10',
>'nodei19']
> }
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/juniperopenstack/+bug/1453369/+subscriptions