Control Node crashing on HA setup
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R2.0 |
Fix Committed
|
High
|
Tapan Karwa | |||
R2.1 |
Won't Fix
|
High
|
Tapan Karwa | |||
R2.20 |
Fix Committed
|
High
|
Tapan Karwa | |||
Trunk |
Fix Committed
|
High
|
Tapan Karwa |
Bug Description
This is 3CN and 3 TSN setup. The cores files can be found at 192.168.61.1 (/var/www/html/pub)
root@Host1-CN1:~#
root@Host1-CN1:~#
root@Host1-CN1:~#
root@Host1-CN1:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------
contrail-analytics 2.20-79 79
contrail-config 2.20-79 79
contrail-
contrail-control 2.20-79 79
contrail-dns 2.20-79 79
contrail-f5 2.20-79 79
contrail-
contrail-heat 2.20-79 79
contrail-
contrail-lib 2.20-79 79
contrail-nodemgr 2.20-79 79
contrail-
contrail-nova-vif 2.20-79 79
contrail-openstack 2.20-79 79
contrail-
contrail-
contrail-
contrail-
contrail-
contrail-
contrail-
contrail-
contrail-setup 2.20-79 79
contrail-utils 2.20-79 79
contrail-
contrail-
contrail-
contrail-
contrail-
contrail-
contrail-web-core 2.20-79 79
ifmap-
ifmap-server 0.3.2-1contrail1 79
neutron-
nova-api 1:2014.
nova-common 1:2014.
nova-compute 1:2014.
nova-compute-kvm 1:2014.
nova-compute-
nova-conductor 1:2014.
nova-console 1:2014.
nova-consoleauth 1:2014.
nova-novncproxy 1:2014.
nova-objectstore 1:2014.
nova-scheduler 1:2014.
python-contrail 2.20-79 79
python-
python-
python-nova 1:2014.
python-
root@Host1-CN1:~# contrail-status
vRouter is NOT PRESENT
== Contrail vRouter ==
supervisor-vrouter: inactive (disabled on boot)
unix://
== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-
contrail-dns active
contrail-named active
== Contrail Analytics ==
supervisor-
contrail-
contrail-
contrail-collector active
contrail-
contrail-
contrail-topology active
== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-
contrail-
contrail-
contrail-schema backup
contrail-
ifmap active
== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-
== Contrail Support Services ==
supervisor-
rabbitmq-server active
========Run time service failures=
/var/crashes/
========Run time service failures=
/var/crashes/
root@Host1-CN1:~# gdb contrail-control /var/crashes/
GNU gdb (Ubuntu 7.7.1-0ubuntu5~
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://
Find the GDB manual and other documentation resources online at:
<http://
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from contrail-
warning: core file may not match specified executable file.
[New LWP 5387]
[New LWP 5390]
[New LWP 5384]
[New LWP 5396]
[New LWP 5391]
[New LWP 5388]
[New LWP 5395]
[New LWP 5401]
[New LWP 5400]
[New LWP 5393]
[New LWP 5371]
[New LWP 5377]
[New LWP 5374]
[New LWP 2267]
[New LWP 5403]
[New LWP 5399]
[New LWP 5376]
[New LWP 5392]
[New LWP 5382]
[New LWP 5381]
[New LWP 5373]
[New LWP 5402]
[New LWP 5383]
[New LWP 5386]
[New LWP 5380]
[New LWP 5398]
[New LWP 5394]
[New LWP 5397]
[New LWP 5372]
[New LWP 5385]
[New LWP 5379]
[New LWP 5375]
[New LWP 5389]
[New LWP 5378]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_
Core was generated by `/usr/bin/
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fd5391a8cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/
56 ../nptl/
(gdb) bt full
#0 0x00007fd5391a8cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/
resultvar = 0
pid = 2267
selftid = 5387
#1 0x00007fd5391ac0d8 in __GI_abort () at abort.c:89
save_stage = 2
act = {__sigaction_
__val = {140553764138268, 11498000, 559, 215838300, 140553762780387, 4294967296, 140553505167536,
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x00007fd5391a1b86 in __assert_fail_base (fmt=0x7fd5392f2830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=
file=
function=
at assert.c:92
str = 0x7fd4c096ab40 "\340\303d\
total = 4096
#3 0x00007fd5391a1c32 in __GI___assert_fail (assertion=0xaf70cd "state-
file=0xaf7210 "controller/
function=
at assert.c:101
No locals.
#4 0x000000000045f776 in ?? ()
No symbol table info available.
#5 0x0000000000491804 in ?? ()
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#6 0x000000000049204b in ?? ()
No symbol table info available.
#7 0x0000000000acd040 in ?? ()
No symbol table info available.
#8 0x00007fd539f7fb3a in ?? () from /usr/lib/
No symbol table info available.
#9 0x00007fd539f7b816 in ?? () from /usr/lib/
No symbol table info available.
#10 0x00007fd539f7af4b in ?? () from /usr/lib/
No symbol table info available.
#11 0x00007fd539f770ff in ?? () from /usr/lib/
No symbol table info available.
#12 0x00007fd539f772f9 in ?? () from /usr/lib/
No symbol table info available.
#13 0x00007fd53a19b182 in start_thread (arg=0x7fd529bf
__res = <optimized out>
pd = 0x7fd529bf6700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140553505171200, 752422925856766
0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#14 0x00007fd53926c47d in clone () at ../sysdeps/
---Type <return> to continue, or q <return> to quit---
No locals.
(gdb) quit
root@Host1-CN1:~# gdb /usr/bin/
GNU gdb (Ubuntu 7.7.1-0ubuntu5~
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://
Find the GDB manual and other documentation resources online at:
<http://
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/
warning: core file may not match specified executable file.
[New LWP 5387]
[New LWP 5390]
[New LWP 5384]
[New LWP 5396]
[New LWP 5391]
[New LWP 5388]
[New LWP 5395]
[New LWP 5401]
[New LWP 5400]
[New LWP 5393]
[New LWP 5371]
[New LWP 5377]
[New LWP 5374]
[New LWP 2267]
[New LWP 5403]
[New LWP 5399]
[New LWP 5376]
[New LWP 5392]
[New LWP 5382]
[New LWP 5381]
[New LWP 5373]
[New LWP 5402]
[New LWP 5383]
[New LWP 5386]
[New LWP 5380]
[New LWP 5398]
[New LWP 5394]
[New LWP 5397]
[New LWP 5372]
[New LWP 5385]
[New LWP 5379]
[New LWP 5375]
[New LWP 5389]
[New LWP 5378]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_
Core was generated by `/usr/bin/
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fd5391a8cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/
56 ../nptl/
(gdb) bt
#0 0x00007fd5391a8cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/
#1 0x00007fd5391ac0d8 in __GI_abort () at abort.c:89
#2 0x00007fd5391a1b86 in __assert_fail_base (fmt=0x7fd5392f2830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=
line=
at assert.c:92
#3 0x00007fd5391a1c32 in __GI___assert_fail (assertion=0xaf70cd "state-
file=0xaf7210 "controller/
function=
#4 0x000000000045f776 in ?? ()
#5 0x0000000000491804 in ?? ()
#6 0x000000000049204b in ?? ()
#7 0x0000000000acd040 in ?? ()
#8 0x00007fd539f7fb3a in ?? () from /usr/lib/
#9 0x00007fd539f7b816 in ?? () from /usr/lib/
#10 0x00007fd539f7af4b in ?? () from /usr/lib/
#11 0x00007fd539f770ff in ?? () from /usr/lib/
#12 0x00007fd539f772f9 in ?? () from /usr/lib/
#13 0x00007fd53a19b182 in start_thread (arg=0x7fd529bf
#14 0x00007fd53926c47d in clone () at ../sysdeps/
(gdb) quit
root@Host1-CN1:~# ls -altr /var/crashes
total 1621292
drwxr-xr-x 14 root root 4096 Aug 12 09:45 ..
drwxrwxrwx 2 root root 4096 Aug 13 16:16 .
-rw------- 1 contrail contrail 1834303488 Aug 13 16:16 core.contrail-
root@Host1-CN1:~# Write failed: Broken pipe
anoops-mbp:~ anoops$
Changed in juniperopenstack: | |
importance: | Undecided → High |
assignee: | nobody → Nischal Sheth (nsheth) |
tags: |
added: contrail-control removed: qfx |
Changed in juniperopenstack: | |
assignee: | Nischal Sheth (nsheth) → Tapan Karwa (tkarwa) |
information type: | Proprietary → Public |
This looks same as Bug 1453369, it is still open