[Build R2.20 10 Juno] TOR Scale:Control node crash @ IFMapExporter::StateUpdateOnDequeue
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R2.20 |
New
|
High
|
Tapan Karwa | |||
Trunk |
New
|
High
|
Tapan Karwa |
Bug Description
Trigger
-------------
System had around 16K Vmi. Deleting all of them in one go with UI.
Backtrace
----------------
(gdb) bt
#0 0x00007f547dd66cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/
#1 0x00007f547dd6a0d8 in __GI_abort () at abort.c:89
#2 0x00007f547dd5fb86 in __assert_fail_base (fmt=0x7f547deb0830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=
file=
function=
#3 0x00007f547dd5fc32 in __GI___assert_fail (assertion=0xadebe1 "state-
function=
#4 0x000000000045df76 in IFMapExporter:
at controller/
#5 0x000000000048add2 in IFMapUpdateSend
at controller/
#6 0x000000000048b314 in IFMapUpdateSend
#7 0x000000000048badb in IFMapUpdateSend
#8 0x0000000000ab2ad0 in TaskImpl::execute (this=0x7f54774
#9 0x00007f547eb3db3a in ?? () from /usr/lib/
#10 0x00007f547eb39816 in ?? () from /usr/lib/
#11 0x00007f547eb38f4b in ?? () from /usr/lib/
#12 0x00007f547eb350ff in ?? () from /usr/lib/
#13 0x00007f547eb352f9 in ?? () from /usr/lib/
#14 0x00007f547ed59182 in start_thread (arg=0x7f544dbf
#15 0x00007f547de2a47d in clone () at ../sysdeps/
(gdb) quit
Contrail-control log during crash
-------
2015-05-09 Sat 14:54:14:617.374 IST nodei6 [Thread 140000083560192, Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotifica
2015-05-09 Sat 14:54:35:205.862 IST nodei6 [Thread 140000045774592, Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp Connection Closed peer ip: 192.168.22.4 ( nodei9-1 ) controller/
2015-05-09 Sat 14:54:42:809.939 IST nodei6 [Thread 140000710457088, Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: PassiveOpen in state: Idle peer ip: 192.168.22.4 ( ) controller/
2015-05-09 Sat 14:54:50:564.235 IST nodei6 [Thread 140000718853888, Pid 18669]: BGP [SYS_WARN]: BgpPeerMessageLog: BGP Peer default-
2015-05-09 Sat 14:54:50:564.660 IST nodei6 [Thread 140000718853888, Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotifica
2015-05-09 Sat 14:54:50:564.968 IST nodei6 [Thread 140000718853888, Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotifica
2015-05-09 Sat 14:55:29:444.147 IST nodei6 [Thread 140000096155392, Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp Connection Closed peer ip: 192.168.22.5 ( nodei10 ) controller/
2015-05-09 Sat 14:55:31:681.300 IST nodei6 [Thread 140000731449088, Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotifica
2015-05-09 Sat 14:55:39:548.516 IST nodei6 [Thread 140000752441088, Pid 18669]: BGP [SYS_WARN]: BgpPeerMessageLog: BGP Peer default-
2015-05-09 Sat 14:55:39:548.843 IST nodei6 [Thread 140000752441088, Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotifica
2015-05-09 Sat 14:55:39:549.138 IST nodei6 [Thread 140000752441088, Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotifica
2015-05-09 Sat 14:55:39:597.924 IST nodei6 [Thread 140000714655488, Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: PassiveOpen in state: Idle peer ip: 192.168.22.5 ( ) controller/
2015-05-09 Sat 14:55:40:694.229 IST nodei6 [Thread 140000054171392, Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp Connection Closed peer ip: 192.168.22.4 ( nodei9 ) controller/
2015-05-09 Sat 14:55:46:241.071 IST nodei6 [Thread 140000037377792, Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp Connection Closed peer ip: 192.168.22.4 ( nodei9-1 ) controller/
2015-05-09 Sat 14:56:05:916.364 IST nodei6 [Thread 140012986161088, Pid 31844]: SANDESH: Logging: DISABLED -> ENABLED
2015-05-09 Sat 14:56:05:916.563 IST nodei6 [Thread 140012986161088, Pid 31844]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_NOTICE ] log4level: [ TRACE ] -> [ WARN ]
2015-05-09 Sat 14:56:05:917.715 IST nodei6 [Thread 140012986161088, Pid 31844]: Starting Bgp Server at port 179
2015-05-09 Sat 14:56:05:930.579 IST nodei6 [Thread 140012986161088, Pid 31844]: SANDESH: No Client: 1431163565930447 SandeshModuleCl
Testbed
--------------
env.roledefs = {
'all': [host1, host2, host3, host4, host5, host6],
'cfgm': [host1, host2, host3],
'openstack': [host1, host2, host3],
'webui': [host2],
'control': [host1, host3],
'compute': [host4, host5, host6],
'tsn': [host4, host5],
'toragent': [host4, host5],
'collector': [host1, host3],
'database': [host1, host2, host3],
'build': [host_build],
}
env.hostnames = {
'all': ['nodei6', 'nodei7', 'nodei8', 'nodei9', 'nodei10', 'nodei19']
}
Changed in juniperopenstack: | |
assignee: | nobody → Tapan Karwa (tkarwa) |
importance: | Undecided → High |
tags: | added: scale |
summary: |
- [Build R2.20 10 Juno] Control node crash @ + [Build R2.20 10 Juno] TOR Scale:Control node crash @ IFMapExporter::StateUpdateOnDequeue |
information type: | Proprietary → Public |
tags: | added: quench |
Logs saved at http:// mayamruga. englab. juniper. net/bugs/ 1453369
To access to the core:
ssh to bhushana@ 10.204. 216.50 Password bhu@123
cd /home/bhushana/ Documents/ technical/ bugs/1453369