[Build R2.20 10 Juno] TOR Scale:Control node crash @ IFMapExporter::StateUpdateOnDequeue

[chhandak]

Trigger
-------------
System had around 16K Vmi. Deleting all of them in one go with UI.

Backtrace
----------------
(gdb) bt
#0 0x00007f547dd66cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f547dd6a0d8 in __GI_abort () at abort.c:89
#2 0x00007f547dd5fb86 in __assert_fail_base (fmt=0x7f547deb0830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xadebe1 "state->advertised().empty()",
    file=file@entry=0xadec78 "controller/src/ifmap/ifmap_exporter.cc", line=line@entry=548,
    function=function@entry=0xadf000 <IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&, bool)::__PRETTY_FUNCTION__> "void IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)") at assert.c:92
#3 0x00007f547dd5fc32 in __GI___assert_fail (assertion=0xadebe1 "state->advertised().empty()", file=0xadec78 "controller/src/ifmap/ifmap_exporter.cc", line=548,
    function=0xadf000 <IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&, bool)::__PRETTY_FUNCTION__> "void IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)") at assert.c:101
#4 0x000000000045df76 in IFMapExporter::StateUpdateOnDequeue (this=0x2d92150, update=update@entry=0x7f545688de70, dequeue_set=..., is_delete=<optimized out>)
    at controller/src/ifmap/ifmap_exporter.cc:548
#5 0x000000000048add2 in IFMapUpdateSender::ProcessUpdate (this=this@entry=0x2d93600, update=update@entry=0x7f545688de70, base_send_set=...)
    at controller/src/ifmap/ifmap_update_sender.cc:225
#6 0x000000000048b314 in IFMapUpdateSender::Send (this=0x2d93600, imarker=<optimized out>) at controller/src/ifmap/ifmap_update_sender.cc:184
#7 0x000000000048badb in IFMapUpdateSender::SendTask::Run (this=0x2dcf520) at controller/src/ifmap/ifmap_update_sender.cc:41
#8 0x0000000000ab2ad0 in TaskImpl::execute (this=0x7f54774c2040) at controller/src/base/task.cc:232
#9 0x00007f547eb3db3a in ?? () from /usr/lib/libtbb.so.2
#10 0x00007f547eb39816 in ?? () from /usr/lib/libtbb.so.2
#11 0x00007f547eb38f4b in ?? () from /usr/lib/libtbb.so.2
#12 0x00007f547eb350ff in ?? () from /usr/lib/libtbb.so.2
#13 0x00007f547eb352f9 in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f547ed59182 in start_thread (arg=0x7f544dbf6700) at pthread_create.c:312
#15 0x00007f547de2a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) quit

Contrail-control log during crash
-------------------------------------------------
2015-05-09 Sat 14:54:14:617.374 IST nodei6 [Thread 140000083560192, Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer default-domain:default-project:ip-fabric:__default__:nodei6:default-domain:default-project:ip-fabric:__default__:walsh SEND Notification with Code 2 and SubCode 7 ( OPEN Message Error:Unsupported Capability ) controller/src/bgp/bgp_session.cc 96
2015-05-09 Sat 14:54:35:205.862 IST nodei6 [Thread 140000045774592, Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp Connection Closed peer ip: 192.168.22.4 ( nodei9-1 ) controller/src/xmpp/xmpp_state_machine.cc 1322
2015-05-09 Sat 14:54:42:809.939 IST nodei6 [Thread 140000710457088, Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: PassiveOpen in state: Idle peer ip: 192.168.22.4 ( ) controller/src/xmpp/xmpp_state_machine.cc 1335
2015-05-09 Sat 14:54:50:564.235 IST nodei6 [Thread 140000718853888, Pid 18669]: BGP [SYS_WARN]: BgpPeerMessageLog: BGP Peer default-domain:default-project:ip-fabric:__default__:nodei6:default-domain:default-project:ip-fabric:__default__:walsh RECV Unsupported Capability: MpExtension (1) controller/src/bgp/bgp_proto.cc 112
2015-05-09 Sat 14:54:50:564.660 IST nodei6 [Thread 140000718853888, Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer default-domain:default-project:ip-fabric:__default__:nodei6:default-domain:default-project:ip-fabric:__default__:walsh RECV Notification with Code 6 and SubCode 5 ( Cease:Connection is rejected by the peer ) controller/src/bgp/state_machine.cc 307
2015-05-09 Sat 14:54:50:564.968 IST nodei6 [Thread 140000718853888, Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer default-domain:default-project:ip-fabric:__default__:nodei6:default-domain:default-project:ip-fabric:__default__:walsh SEND Notification with Code 2 and SubCode 7 ( OPEN Message Error:Unsupported Capability ) controller/src/bgp/bgp_session.cc 96
2015-05-09 Sat 14:55:29:444.147 IST nodei6 [Thread 140000096155392, Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp Connection Closed peer ip: 192.168.22.5 ( nodei10 ) controller/src/xmpp/xmpp_state_machine.cc 1322
2015-05-09 Sat 14:55:31:681.300 IST nodei6 [Thread 140000731449088, Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer default-domain:default-project:ip-fabric:__default__:nodei6:default-domain:default-project:ip-fabric:__default__:nodei8 RECV Notification with Code 6 and SubCode 3 ( Cease:Administrator has unconfigured the peer ) controller/src/bgp/state_machine.cc 307
2015-05-09 Sat 14:55:39:548.516 IST nodei6 [Thread 140000752441088, Pid 18669]: BGP [SYS_WARN]: BgpPeerMessageLog: BGP Peer default-domain:default-project:ip-fabric:__default__:nodei6:default-domain:default-project:ip-fabric:__default__:walsh RECV Unsupported Capability: MpExtension (1) controller/src/bgp/bgp_proto.cc 112
2015-05-09 Sat 14:55:39:548.843 IST nodei6 [Thread 140000752441088, Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer default-domain:default-project:ip-fabric:__default__:nodei6:default-domain:default-project:ip-fabric:__default__:walsh RECV Notification with Code 6 and SubCode 5 ( Cease:Connection is rejected by the peer ) controller/src/bgp/state_machine.cc 307
2015-05-09 Sat 14:55:39:549.138 IST nodei6 [Thread 140000752441088, Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer default-domain:default-project:ip-fabric:__default__:nodei6:default-domain:default-project:ip-fabric:__default__:walsh SEND Notification with Code 2 and SubCode 7 ( OPEN Message Error:Unsupported Capability ) controller/src/bgp/bgp_session.cc 96
2015-05-09 Sat 14:55:39:597.924 IST nodei6 [Thread 140000714655488, Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: PassiveOpen in state: Idle peer ip: 192.168.22.5 ( ) controller/src/xmpp/xmpp_state_machine.cc 1335
2015-05-09 Sat 14:55:40:694.229 IST nodei6 [Thread 140000054171392, Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp Connection Closed peer ip: 192.168.22.4 ( nodei9 ) controller/src/xmpp/xmpp_state_machine.cc 1322
2015-05-09 Sat 14:55:46:241.071 IST nodei6 [Thread 140000037377792, Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp Connection Closed peer ip: 192.168.22.4 ( nodei9-1 ) controller/src/xmpp/xmpp_state_machine.cc 1322
2015-05-09 Sat 14:56:05:916.364 IST nodei6 [Thread 140012986161088, Pid 31844]: SANDESH: Logging: DISABLED -> ENABLED
2015-05-09 Sat 14:56:05:916.563 IST nodei6 [Thread 140012986161088, Pid 31844]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_NOTICE ] log4level: [ TRACE ] -> [ WARN ]
2015-05-09 Sat 14:56:05:917.715 IST nodei6 [Thread 140012986161088, Pid 31844]: Starting Bgp Server at port 179
2015-05-09 Sat 14:56:05:930.579 IST nodei6 [Thread 140012986161088, Pid 31844]: SANDESH: No Client: 1431163565930447 SandeshModuleClientTrace: data= [ name = nodei6:Control:contrail-control:0 client_info= [ status = Idle successful_connections = 0 pid = 31844 http_port = 8083 start_time = 1431163565930223 collector_name = primary = 0.0.0.0:0 secondary = 0.0.0.0:0 rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] ]

Testbed
--------------

env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6],
    'cfgm': [host1, host2, host3],
    'openstack': [host1, host2, host3],
    'webui': [host2],
    'control': [host1, host3],
    'compute': [host4, host5, host6],
    'tsn': [host4, host5],
    'toragent': [host4, host5],
    'collector': [host1, host3],
    'database': [host1, host2, host3],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodei6', 'nodei7', 'nodei8', 'nodei9', 'nodei10', 'nodei19']
}

[chhandak (chhandak)]

Logs saved at http://mayamruga.englab.juniper.net/bugs/1453369

To access to the core:

ssh to bhushana@10.204.216.50 Password bhu@123

cd /home/bhushana/Documents/technical/bugs/1453369

[Tapan Karwa (tkarwa)]

Dup of 1430091

[Vedamurthy Joshi (vedujoshi)]

Tapan,
 R2.20 Build 10 already had fix for 1430091.
Does it mean 1430091 is not resolved yet or 1453369 is a new issue.

I see similar crash on 2.20 build 30 as well.

Could you pls check ?

Vedu

On 5/11/15, 9:09 PM, "Tapan Karwa" <email address hidden> wrote:

>*** This bug is a duplicate of bug 1430091 ***
> https://bugs.launchpad.net/bugs/1430091
>
>Dup of 1430091
>
>** This bug has been marked a duplicate of bug 1430091
> control-node assertion in IFMapExporter::StateUpdateOnDequeue on
>deleting logical interfaces
>
>--
>You received this bug notification because you are a member of Contrail
>Systems engineering, which is subscribed to Juniper Openstack.
>Matching subscriptions: Juniper Openstack subscription
>https://bugs.launchpad.net/bugs/1453369
>
>Title:
> [Build R2.20 10 Juno] TOR Scale:Control node crash @
> IFMapExporter::StateUpdateOnDequeue
>
>Status in Juniper Openstack distribution:
> New
>Status in Juniper Openstack r2.20 series:
> New
>Status in Juniper Openstack trunk series:
> New
>
>Bug description:
>
> Trigger
> -------------
> System had around 16K Vmi. Deleting all of them in one go with UI.
>
> Backtrace
> ----------------
> (gdb) bt
> #0 0x00007f547dd66cc9 in __GI_raise (sig=sig@entry=6) at
>../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1 0x00007f547dd6a0d8 in __GI_abort () at abort.c:89
> #2 0x00007f547dd5fb86 in __assert_fail_base (fmt=0x7f547deb0830
>"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
>assertion=assertion@entry=0xadebe1 "state->advertised().empty()",
> file=file@entry=0xadec78 "controller/src/ifmap/ifmap_exporter.cc",
>line=line@entry=548,
> function=function@entry=0xadf000
><IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&,
>bool)::__PRETTY_FUNCTION__> "void
>IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)")
>at assert.c:92
> #3 0x00007f547dd5fc32 in __GI___assert_fail (assertion=0xadebe1
>"state->advertised().empty()", file=0xadec78
>"controller/src/ifmap/ifmap_exporter.cc", line=548,
> function=0xadf000
><IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&,
>bool)::__PRETTY_FUNCTION__> "void
>IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)")
>at assert.c:101
> #4 0x000000000045df76 in IFMapExporter::StateUpdateOnDequeue
>(this=0x2d92150, update=update@entry=0x7f545688de70, dequeue_set=...,
>is_delete=<optimized out>)
> at controller/src/ifmap/ifmap_exporter.cc:548
> #5 0x000000000048add2 in IFMapUpdateSender::ProcessUpdate
>(this=this@entry=0x2d93600, update=update@entry=0x7f545688de70,
>base_send_set=...)
> at controller/src/ifmap/ifmap_update_sender.cc:225
> #6 0x000000000048b314 in IFMapUpdateSender::Send (this=0x2d93600,
>imarker=<optimized out>) at
>controller/src/ifmap/ifmap_update_sender.cc:184
> #7 0x000000000048badb in IFMapUpdateSender::SendTask::Run
>(this=0x2dcf520) at controller/src/ifmap/ifmap_update_sender.cc:41
> #8 0x0000000000ab2ad0 in TaskImpl::execute (this=0x7f54774c2040) at
>controller/src/base/task.cc:232
> #9 0x00007f547eb3db3a in ?? () from /usr/lib/libtbb.so.2
> #10 0x00007f547eb39816 in ?? () from /usr/lib/libtbb.so.2
> #11 0x00007f547eb38...

[Vedamurthy Joshi (vedujoshi)]

R2.2 Build 10 already had fix from bug 1430091

Uploaded new core from R2.2 Build 30 to http://10.204.216.50/Docs/bugs/1453369/may28

[chhandak (chhandak)]

Observed the issue with Build R2.20.43. Seen the crash while deleting LIF and VMI

[chhandak (chhandak)]

Hi Tapan.

This crash can be consistently reproducible .

When ever we delete logical interface in chunk (using webui), I have hit
the crash.

You can have a look at the crash @ nodei6 (10.204.217.118) root/c0ntrail123

Thanks and Regards,
Chhandak

On 6/15/15, 10:32 PM, "OpenContrail Admin" <email address hidden>
wrote:

>** Changed in: juniperopenstack/r2.20
> Milestone: r2.20-fcs => r2.21
>
>** Tags added: quench
>
>--
>You received this bug notification because you are a member of Contrail
>Systems engineering, which is subscribed to Juniper Openstack.
>https://bugs.launchpad.net/bugs/1453369
>
>Title:
> [Build R2.20 10 Juno] TOR Scale:Control node crash @
> IFMapExporter::StateUpdateOnDequeue
>
>Status in Juniper Openstack distribution:
> New
>Status in Juniper Openstack r2.20 series:
> New
>Status in Juniper Openstack trunk series:
> New
>
>Bug description:
>
> Trigger
> -------------
> System had around 16K Vmi. Deleting all of them in one go with UI.
>
> Backtrace
> ----------------
> (gdb) bt
> #0 0x00007f547dd66cc9 in __GI_raise (sig=sig@entry=6) at
>../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1 0x00007f547dd6a0d8 in __GI_abort () at abort.c:89
> #2 0x00007f547dd5fb86 in __assert_fail_base (fmt=0x7f547deb0830
>"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
>assertion=assertion@entry=0xadebe1 "state->advertised().empty()",
> file=file@entry=0xadec78 "controller/src/ifmap/ifmap_exporter.cc",
>line=line@entry=548,
> function=function@entry=0xadf000
><IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&,
>bool)::__PRETTY_FUNCTION__> "void
>IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)")
>at assert.c:92
> #3 0x00007f547dd5fc32 in __GI___assert_fail (assertion=0xadebe1
>"state->advertised().empty()", file=0xadec78
>"controller/src/ifmap/ifmap_exporter.cc", line=548,
> function=0xadf000
><IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&,
>bool)::__PRETTY_FUNCTION__> "void
>IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)")
>at assert.c:101
> #4 0x000000000045df76 in IFMapExporter::StateUpdateOnDequeue
>(this=0x2d92150, update=update@entry=0x7f545688de70, dequeue_set=...,
>is_delete=<optimized out>)
> at controller/src/ifmap/ifmap_exporter.cc:548
> #5 0x000000000048add2 in IFMapUpdateSender::ProcessUpdate
>(this=this@entry=0x2d93600, update=update@entry=0x7f545688de70,
>base_send_set=...)
> at controller/src/ifmap/ifmap_update_sender.cc:225
> #6 0x000000000048b314 in IFMapUpdateSender::Send (this=0x2d93600,
>imarker=<optimized out>) at
>controller/src/ifmap/ifmap_update_sender.cc:184
> #7 0x000000000048badb in IFMapUpdateSender::SendTask::Run
>(this=0x2dcf520) at controller/src/ifmap/ifmap_update_sender.cc:41
> #8 0x0000000000ab2ad0 in TaskImpl::execute (this=0x7f54774c2040) at
>controller/src/base/task.cc:232
> #9 0x00007f547eb3db3a in ?? () from /usr/lib/libtbb.so.2
> #10 0x00007f547eb39816 in ?? () from /usr/lib/libtbb.so.2
> #11 0x00007f547eb38f4b in ?? () from /usr/lib/libtbb.so.2
> #12 0x00007f547eb350ff in ?? () from /usr/lib/libtbb.so.2
> #13 0x00007f547eb352f9 in ?? () from /usr/lib/lib...

[Tapan Karwa (tkarwa)]

Working on something right now.
As requested, can you please upload any core you see, add that info to the bug along with the version number.
Thanks.

On Jun 23, 2015, at 5:02 AM, Chhandak Mukherjee wrote:

> Hi Tapan.
>
> This crash can be consistently reproducible .
>
> When ever we delete logical interface in chunk (using webui), I have hit
> the crash.
>
> You can have a look at the crash @ nodei6 (10.204.217.118) root/c0ntrail123
>
> Thanks and Regards,
> Chhandak
>
>
>
> On 6/15/15, 10:32 PM, "OpenContrail Admin" <email address hidden>
> wrote:
>
>> ** Changed in: juniperopenstack/r2.20
>> Milestone: r2.20-fcs => r2.21
>>
>> ** Tags added: quench
>>
>> --
>> You received this bug notification because you are a member of Contrail
>> Systems engineering, which is subscribed to Juniper Openstack.
>> https://bugs.launchpad.net/bugs/1453369
>>
>> Title:
>> [Build R2.20 10 Juno] TOR Scale:Control node crash @
>> IFMapExporter::StateUpdateOnDequeue
>>
>> Status in Juniper Openstack distribution:
>> New
>> Status in Juniper Openstack r2.20 series:
>> New
>> Status in Juniper Openstack trunk series:
>> New
>>
>> Bug description:
>>
>> Trigger
>> -------------
>> System had around 16K Vmi. Deleting all of them in one go with UI.
>>
>> Backtrace
>> ----------------
>> (gdb) bt
>> #0 0x00007f547dd66cc9 in __GI_raise (sig=sig@entry=6) at
>> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>> #1 0x00007f547dd6a0d8 in __GI_abort () at abort.c:89
>> #2 0x00007f547dd5fb86 in __assert_fail_base (fmt=0x7f547deb0830
>> "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
>> assertion=assertion@entry=0xadebe1 "state->advertised().empty()",
>> file=file@entry=0xadec78 "controller/src/ifmap/ifmap_exporter.cc",
>> line=line@entry=548,
>> function=function@entry=0xadf000
>> <IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&,
>> bool)::__PRETTY_FUNCTION__> "void
>> IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)")
>> at assert.c:92
>> #3 0x00007f547dd5fc32 in __GI___assert_fail (assertion=0xadebe1
>> "state->advertised().empty()", file=0xadec78
>> "controller/src/ifmap/ifmap_exporter.cc", line=548,
>> function=0xadf000
>> <IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&,
>> bool)::__PRETTY_FUNCTION__> "void
>> IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)")
>> at assert.c:101
>> #4 0x000000000045df76 in IFMapExporter::StateUpdateOnDequeue
>> (this=0x2d92150, update=update@entry=0x7f545688de70, dequeue_set=...,
>> is_delete=<optimized out>)
>> at controller/src/ifmap/ifmap_exporter.cc:548
>> #5 0x000000000048add2 in IFMapUpdateSender::ProcessUpdate
>> (this=this@entry=0x2d93600, update=update@entry=0x7f545688de70,
>> base_send_set=...)
>> at controller/src/ifmap/ifmap_update_sender.cc:225
>> #6 0x000000000048b314 in IFMapUpdateSender::Send (this=0x2d93600,
>> imarker=<optimized out>) at
>> controller/src/ifmap/ifmap_update_sender.cc:184
>> #7 0x000000000048badb in IFMapUpdateSender::SendTask::Run
>> (this=0x2dcf520) at controller/src/ifmap/ifmap_update_sender.cc:41
>> #8 0x0000000000ab2ad0 in TaskImpl::execute (this=0x7f54774c2040) at
>> controlle...

[chhandak (chhandak)]

Observed the core again with build R2.20.81

Core Copied @ Logs saved at http://mayamruga.englab.juniper.net/bugs/1453369

Setup : nodei6(root/c0ntrail123) is still having the core.

[chhandak (chhandak)]

Hi Tapan,

This crash can be reproduced consistently on scale setup. Observed again
on R2.20.81
Copied the core @ http://mayamruga.englab.juniper.net/bugs
<http://mayamruga.englab.juniper.net/bugs/%3Cbug-ID%3E>/1453369

Setup nodei6(10.204.217.118, root/c0ntrail123) still is in issue
reproduced state.

Thanks and Regards,
Chhandak

On 6/15/15, 10:32 PM, "OpenContrail Admin" <email address hidden>
wrote:

>** Changed in: juniperopenstack/r2.20
> Milestone: r2.20-fcs => r2.21
>
>** Tags added: quench
>
>--
>You received this bug notification because you are a member of Contrail
>Systems engineering, which is subscribed to Juniper Openstack.
>https://bugs.launchpad.net/bugs/1453369
>
>Title:
> [Build R2.20 10 Juno] TOR Scale:Control node crash @
> IFMapExporter::StateUpdateOnDequeue
>
>Status in Juniper Openstack distribution:
> New
>Status in Juniper Openstack r2.20 series:
> New
>Status in Juniper Openstack trunk series:
> New
>
>Bug description:
>
> Trigger
> -------------
> System had around 16K Vmi. Deleting all of them in one go with UI.
>
> Backtrace
> ----------------
> (gdb) bt
> #0 0x00007f547dd66cc9 in __GI_raise (sig=sig@entry=6) at
>../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1 0x00007f547dd6a0d8 in __GI_abort () at abort.c:89
> #2 0x00007f547dd5fb86 in __assert_fail_base (fmt=0x7f547deb0830
>"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
>assertion=assertion@entry=0xadebe1 "state->advertised().empty()",
> file=file@entry=0xadec78 "controller/src/ifmap/ifmap_exporter.cc",
>line=line@entry=548,
> function=function@entry=0xadf000
><IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&,
>bool)::__PRETTY_FUNCTION__> "void
>IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)")
>at assert.c:92
> #3 0x00007f547dd5fc32 in __GI___assert_fail (assertion=0xadebe1
>"state->advertised().empty()", file=0xadec78
>"controller/src/ifmap/ifmap_exporter.cc", line=548,
> function=0xadf000
><IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&,
>bool)::__PRETTY_FUNCTION__> "void
>IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)")
>at assert.c:101
> #4 0x000000000045df76 in IFMapExporter::StateUpdateOnDequeue
>(this=0x2d92150, update=update@entry=0x7f545688de70, dequeue_set=...,
>is_delete=<optimized out>)
> at controller/src/ifmap/ifmap_exporter.cc:548
> #5 0x000000000048add2 in IFMapUpdateSender::ProcessUpdate
>(this=this@entry=0x2d93600, update=update@entry=0x7f545688de70,
>base_send_set=...)
> at controller/src/ifmap/ifmap_update_sender.cc:225
> #6 0x000000000048b314 in IFMapUpdateSender::Send (this=0x2d93600,
>imarker=<optimized out>) at
>controller/src/ifmap/ifmap_update_sender.cc:184
> #7 0x000000000048badb in IFMapUpdateSender::SendTask::Run
>(this=0x2dcf520) at controller/src/ifmap/ifmap_update_sender.cc:41
> #8 0x0000000000ab2ad0 in TaskImpl::execute (this=0x7f54774c2040) at
>controller/src/base/task.cc:232
> #9 0x00007f547eb3db3a in ?? () from /usr/lib/libtbb.so.2
> #10 0x00007f547eb39816 in ?? () from /usr/lib/libtbb.so.2
> #11 0x00007f547eb38f4b in ?? () from /usr/lib/libtbb.so.2
> #12 0x00007f547eb350ff in ?...

[chhandak (chhandak)]

Observed the crash in R2.21.92 build as well. Can be hit while deleting multiple LIF or VN in scale environment