vsrx2.0 with contrail - Run time service failures due to cores
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Juniper Openstack |
Fix Committed
|
Critical
|
Tapan Karwa | ||
R1.1 |
Fix Committed
|
Critical
|
Tapan Karwa | ||
R2.0 |
Fix Committed
|
Critical
|
Tapan Karwa |
Bug Description
Blocks vsrx2.0 testing
Interfaces went down during testing and it is seen there are several cores
root@kvmsrv08:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-
contrail-
== Contrail Control ==
supervisor-control: active
contrail-control failed
contrail-
contrail-dns active
contrail-named active
== Contrail Analytics ==
supervisor-
contrail-alarm-gen active
contrail-
contrail-
contrail-collector active
contrail-
contrail-
contrail-topology active
== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-
contrail-
contrail-
contrail-schema active
contrail-
ifmap active
== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-
== Contrail Database ==
supervisor-
contrail-database active
contrail-
kafka active
== Contrail Support Services ==
supervisor-
rabbitmq-server active
========Run time service failures=
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
Exception KeyError: KeyError(
root@kvmsrv08:~# openstack-status
== Nova services ==
openstack-nova-api: active
openstack-
openstack-
openstack-
openstack-
openstack-
== Glance services ==
openstack-
openstack-
== Keystone service ==
openstack-keystone: active
== Cinder services ==
openstack-
openstack-
openstack-
== Support services ==
mysql: inactive (disabled on boot)
libvirt-bin: active
rabbitmq-server: active
memcached: inactive (disabled on boot)
== Keystone users ==
Warning keystonerc not sourced
root@kvmsrv08:~#
From contrail controller page
Alerts
kvmsrv08 , Control Node , System Information unavailable
kvmsrv08 , vRouter , 7 Interfaces down
kvmsrv08 , vRouter , 1 core dump, 1 restart
root@kvmsrv08:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------
contrail-analytics 3.0-2550 2550
contrail-config 3.0-2550 2550
contrail-
contrail-control 3.0-2550 2550
contrail-dns 3.0-2550 2550
contrail-f5 3.0-2550 2550
contrail-
contrail-heat 3.0-2550 2550
contrail-
contrail-lib 3.0-2550 2550
contrail-nodemgr 3.0-2550 2550
contrail-
contrail-nova-vif 3.0-2550 2550
contrail-openstack 3.0-2550 2550
contrail-
contrail-
contrail-
contrail-
contrail-
contrail-
contrail-
contrail-setup 3.0-2550 2550
contrail-utils 3.0-2550 2550
contrail-
contrail-
contrail-
contrail-
contrail-
contrail-
contrail-web-core 3.0-2550 2550
ifmap-python-client 0.1-2 2550
ifmap-server 0.3.2-1contrail1 2550
neutron-
nova-api 1:2014.
nova-common 1:2014.
nova-compute 1:2014.
nova-compute-kvm 1:2014.
nova-compute-
nova-conductor 1:2014.
nova-console 1:2014.
nova-consoleauth 1:2014.
nova-novncproxy 1:2014.
nova-objectstore 1:2014.
nova-scheduler 1:2014.
python-contrail 3.0-2550 2550
python-
python-
python-nova 1:2014.
python-
root@kvmsrv08:~#
root@kvmsrv08:~# contrail-status --detail
== Contrail vRouter ==
supervisor-vrouter: active
contrail-
contrail-
== Contrail Control ==
supervisor-control: active
contrail-control failed Exited too quickly (process log may have details)
contrail-
contrail-dns active pid 49569, uptime 9:58:55
contrail-named active pid 3142, uptime 9 days, 10:58:16
== Contrail Analytics ==
supervisor-
contrail-alarm-gen active pid 3140, uptime 9 days, 10:58:16
contrail-
contrail-
contrail-collector active pid 3093, uptime 9 days, 10:58:16
contrail-
contrail-
contrail-topology active pid 3091, uptime 9 days, 10:58:16
== Contrail Config ==
supervisor-config: active
contrail-api:0 active pid 36013, uptime 2 days, 9:07:49
contrail-
contrail-
contrail-
contrail-schema active pid 36015, uptime 2 days, 9:07:49
contrail-
ifmap active pid 36011, uptime 2 days, 9:07:49
== Contrail Web UI ==
supervisor-webui: active
contrail-webui active pid 3021, uptime 9 days, 10:58:16
contrail-
== Contrail Database ==
supervisor-
contrail-database active pid 24724, uptime 2 days, 9:19:05
contrail-
kafka active pid 24723, uptime 2 days, 9:19:05
== Contrail Support Services ==
supervisor-
rabbitmq-server active pid 3001, uptime 9 days, 10:58:16
========Run time service failures=
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
Exception KeyError: KeyError(
root@kvmsrv08:~# contrail-status --debug
== Contrail vRouter ==
supervisor-vrouter: active
supervisor-vrouter: ['contrail-
contrail-
contrail-
contrail-
== Contrail Control ==
supervisor-control: active
supervisor-control: ['contrail-control failed Exited too quickly (process log may have details)', 'contrail-
contrail-control failed
contrail-
contrail-dns active
contrail-named active
== Contrail Analytics ==
supervisor-
supervisor-
contrail-alarm-gen: DEFAULT/
contrail-alarm-gen active
contrail-
contrail-
contrail-collector active
contrail-
contrail-
contrail-topology active
== Contrail Config ==
supervisor-config: active
supervisor-config: ['contrail-api:0 active pid 36013, uptime 2 days, 9:08:13', 'contrail-
contrail-api:0: DEFAULT/
contrail-api:0 active
contrail-
contrail-
contrail-
contrail-
contrail-schema: DEFAULT/
contrail-schema active
contrail-
contrail-
ifmap active
== Contrail Web UI ==
supervisor-webui: active
supervisor-webui: ['contrail-webui active pid 3021, uptime 9 days, 10:58:40', 'contrail-
contrail-webui active
contrail-
== Contrail Database ==
supervisor-
supervisor-
contrail-database active
contrail-
contrail-
kafka active
== Contrail Support Services ==
supervisor-
supervisor-
rabbitmq-server active
========Run time service failures=
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
/var/crashes/
Exception KeyError: KeyError(
root@kvmsrv08:~#
description: | updated |
summary: |
- Run time service failures due to cores + vsrx2.0 with contrail - Run time service failures due to cores |
tags: | added: blocker |
Changed in juniperopenstack: | |
importance: | Undecided → Critical |
Changed in juniperopenstack: | |
assignee: | nobody → Tapan Karwa (tkarwa) |
information type: | Proprietary → Public |
uploaded crash files,gdb did not return much info.
restarted vrouter to proceed with my testing.
root@kvmsrv08: /var/crashes# gdb core.contrail- vroute. 3033.kvmsrv08. 1426789701 14.04.2) 7.7.1 gnu.org/ licenses/ gpl.html> www.gnu. org/software/ gdb/bugs/>. www.gnu. org/software/ gdb/documentati on/>. core.contrail- vroute. 3033.kvmsrv08. 1426789701" : not in executable format: File format not recognized /var/crashes# ls -ltr dns.3141. kvmsrv08. 1426787901 contro. 3139.kvmsrv08. 1426787901 contro. 23948.kvmsrv08. 1426787901 contro. 24331.kvmsrv08. 1426787903 contro. 25182.kvmsrv08. 1426787906 dns.23999. kvmsrv08. 1426787907 contro. 26071.kvmsrv08. 1426787910 dns.25674. kvmsrv08. 1426787913 dns.48380. kvmsrv08. 1426788642 vroute. 3033.kvmsrv08. 1426789701 /var/crashes# gdb core.contrail- contro. 26071.kvmsrv08. 1426787910 14.04.2) 7.7.1 gnu.org/ licenses/ gpl.html> www.gnu. org/software/ gdb/bugs/>. www.gnu. org/software/ gdb/documentati on/>. core.contrail- contro. 26071.kvmsrv08. 1426787910" : not in executable format: File format not recognized /var/c. ..
GNU gdb (Ubuntu 7.7.1-0ubuntu5~
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://
Find the GDB manual and other documentation resources online at:
<http://
For help, type "help".
Type "apropos word" to search for commands related to "word"...
"/var/crashes/
(gdb) bt
No stack.
(gdb) exit
Undefined command: "exit". Try "help".
(gdb) q
root@kvmsrv08:
total 657392
-rw------- 1 contrail contrail 229175296 Mar 19 10:58 core.contrail-
-rw------- 1 contrail contrail 234889216 Mar 19 10:58 core.contrail-
-rw------- 1 contrail contrail 232099840 Mar 19 10:58 core.contrail-
-rw------- 1 contrail contrail 228188160 Mar 19 10:58 core.contrail-
-rw------- 1 contrail contrail 231997440 Mar 19 10:58 core.contrail-
-rw------- 1 contrail contrail 227655680 Mar 19 10:58 core.contrail-
-rw------- 1 contrail contrail 227885056 Mar 19 10:58 core.contrail-
-rw------- 1 contrail contrail 227647488 Mar 19 10:58 core.contrail-
-rw------- 1 contrail contrail 227713024 Mar 19 11:10 core.contrail-
-rw------- 1 root root 237846528 Mar 19 11:28 core.contrail-
root@kvmsrv08:
GNU gdb (Ubuntu 7.7.1-0ubuntu5~
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://
Find the GDB manual and other documentation resources online at:
<http://
For help, type "help".
Type "apropos word" to search for commands related to "word"...
"/var/crashes/
(gdb) bt
No stack.
(gdb) bt
No stack.
(gdb) exit
Undefined command: "exit". Try "help".
(gdb) quit
root@kvmsrv08: