vsrx2.0 with contrail - Run time service failures due to cores

[Jaya]

Blocks vsrx2.0 testing
Interfaces went down during testing and it is seen there are several cores

root@kvmsrv08:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent initializing
contrail-vrouter-nodemgr active

== Contrail Control ==
supervisor-control: active
contrail-control failed
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager active
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor active
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
supervisor-database: active
contrail-database active
contrail-database-nodemgr active
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

========Run time service failures=============
/var/crashes/core.contrail-dns.48380.kvmsrv08.1426788642
/var/crashes/core.contrail-dns.25674.kvmsrv08.1426787913
/var/crashes/core.contrail-contro.25182.kvmsrv08.1426787906
/var/crashes/core.contrail-contro.24331.kvmsrv08.1426787903
/var/crashes/core.contrail-vroute.3033.kvmsrv08.1426789701
/var/crashes/core.contrail-dns.3141.kvmsrv08.1426787901
/var/crashes/core.contrail-dns.23999.kvmsrv08.1426787907
/var/crashes/core.contrail-contro.23948.kvmsrv08.1426787901
/var/crashes/core.contrail-contro.3139.kvmsrv08.1426787901
/var/crashes/core.contrail-contro.26071.kvmsrv08.1426787910
Exception KeyError: KeyError(140199974133584,) in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
root@kvmsrv08:~# openstack-status
== Nova services ==
openstack-nova-api: active
openstack-nova-compute: active
openstack-nova-network: inactive (disabled on boot)
openstack-nova-scheduler: active
openstack-nova-volume: inactive (disabled on boot)
openstack-nova-conductor: active
== Glance services ==
openstack-glance-api: active
openstack-glance-registry: active
== Keystone service ==
openstack-keystone: active
== Cinder services ==
openstack-cinder-api: active
openstack-cinder-scheduler: active
openstack-cinder-volume: inactive (disabled on boot)
== Support services ==
mysql: inactive (disabled on boot)
libvirt-bin: active
rabbitmq-server: active
memcached: inactive (disabled on boot)
== Keystone users ==
Warning keystonerc not sourced
root@kvmsrv08:~#

From contrail controller page

Alerts
kvmsrv08 , Control Node , System Information unavailable
kvmsrv08 , vRouter , 7 Interfaces down
kvmsrv08 , vRouter , 1 core dump, 1 restart

root@kvmsrv08:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.0-2550 2550
contrail-config 3.0-2550 2550
contrail-config-openstack 3.0-2550 2550
contrail-control 3.0-2550 2550
contrail-dns 3.0-2550 2550
contrail-f5 3.0-2550 2550
contrail-fabric-utils 3.0-2550 2550
contrail-heat 3.0-2550 2550
contrail-install-packages 3.0-2550~icehouse 2550
contrail-lib 3.0-2550 2550
contrail-nodemgr 3.0-2550 2550
contrail-nova-networkapi 3.0-2550 2550
contrail-nova-vif 3.0-2550 2550
contrail-openstack 3.0-2550 2550
contrail-openstack-analytics 3.0-2550 2550
contrail-openstack-config 3.0-2550 2550
contrail-openstack-control 3.0-2550 2550
contrail-openstack-dashboard 3.0-2550 2550
contrail-openstack-database 3.0-2550 2550
contrail-openstack-vrouter 3.0-2550 2550
contrail-openstack-webui 3.0-2550 2550
contrail-setup 3.0-2550 2550
contrail-utils 3.0-2550 2550
contrail-vrouter-agent 3.0-2550 2550
contrail-vrouter-common 3.0-2550 2550
contrail-vrouter-dkms 3.0-2550 2550
contrail-vrouter-init 3.0-2550 2550
contrail-vrouter-utils 3.0-2550 2550
contrail-web-controller 3.0-2550 2550
contrail-web-core 3.0-2550 2550
ifmap-python-client 0.1-2 2550
ifmap-server 0.3.2-1contrail1 2550
neutron-plugin-contrail 3.0-2550 2550
nova-api 1:2014.1.3-0ubuntu1~cloud0.2contrail2550
nova-common 1:2014.1.3-0ubuntu1~cloud0.2contrail2550
 nova-compute 1:2014.1.3-0ubuntu1~cloud0.2contrail2550
nova-compute-kvm 1:2014.1.3-0ubuntu1~cloud0.2contrail2550
nova-compute-libvirt 1:2014.1.3-0ubuntu1~cloud0.2contrail2550
nova-conductor 1:2014.1.3-0ubuntu1~cloud0.2contrail2550
nova-console 1:2014.1.3-0ubuntu1~cloud0.2contrail2550
nova-consoleauth 1:2014.1.3-0ubuntu1~cloud0.2contrail2550
nova-novncproxy 1:2014.1.3-0ubuntu1~cloud0.2contrail2550
nova-objectstore 1:2014.1.3-0ubuntu1~cloud0.2contrail2550
nova-scheduler 1:2014.1.3-0ubuntu1~cloud0.2contrail2550
python-contrail 3.0-2550 2550
python-contrail-vrouter-api 3.0-2550 2550
python-neutronclient 2:2.3.4-0ubuntu1.2contrail 2550
python-nova 1:2014.1.3-0ubuntu1~cloud0.2contrail2550
python-opencontrail-vrouter-netns 3.0-2550 2550
root@kvmsrv08:~#

root@kvmsrv08:~# contrail-status --detail
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent initializing pid 21905, uptime 1:41:29
contrail-vrouter-nodemgr active pid 3030, uptime 9 days, 10:58:16

== Contrail Control ==
supervisor-control: active
contrail-control failed Exited too quickly (process log may have details)
contrail-control-nodemgr active pid 3128, uptime 9 days, 10:58:16
contrail-dns active pid 49569, uptime 9:58:55
contrail-named active pid 3142, uptime 9 days, 10:58:16

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen active pid 3140, uptime 9 days, 10:58:16
contrail-analytics-api active pid 3138, uptime 9 days, 10:58:16
contrail-analytics-nodemgr active pid 3089, uptime 9 days, 10:58:16
contrail-collector active pid 3093, uptime 9 days, 10:58:16
contrail-query-engine active pid 3108, uptime 9 days, 10:58:16
contrail-snmp-collector active pid 3090, uptime 9 days, 10:58:16
contrail-topology active pid 3091, uptime 9 days, 10:58:16

== Contrail Config ==
supervisor-config: active
contrail-api:0 active pid 36013, uptime 2 days, 9:07:49
contrail-config-nodemgr active pid 36010, uptime 2 days, 9:07:49
contrail-device-manager active pid 36014, uptime 2 days, 9:07:49
contrail-discovery:0 active pid 36012, uptime 2 days, 9:07:49
contrail-schema active pid 36015, uptime 2 days, 9:07:49
contrail-svc-monitor active pid 54831, uptime 2 days, 7:31:47
ifmap active pid 36011, uptime 2 days, 9:07:49

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active pid 3021, uptime 9 days, 10:58:16
contrail-webui-middleware active pid 3025, uptime 9 days, 10:58:16

== Contrail Database ==
supervisor-database: active
contrail-database active pid 24724, uptime 2 days, 9:19:05
contrail-database-nodemgr active pid 24722, uptime 2 days, 9:19:05
kafka active pid 24723, uptime 2 days, 9:19:05

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active pid 3001, uptime 9 days, 10:58:16

========Run time service failures=============
/var/crashes/core.contrail-dns.48380.kvmsrv08.1426788642
/var/crashes/core.contrail-dns.25674.kvmsrv08.1426787913
/var/crashes/core.contrail-contro.25182.kvmsrv08.1426787906
/var/crashes/core.contrail-contro.24331.kvmsrv08.1426787903
/var/crashes/core.contrail-vroute.3033.kvmsrv08.1426789701
/var/crashes/core.contrail-dns.3141.kvmsrv08.1426787901
/var/crashes/core.contrail-dns.23999.kvmsrv08.1426787907
/var/crashes/core.contrail-contro.23948.kvmsrv08.1426787901
/var/crashes/core.contrail-contro.3139.kvmsrv08.1426787901
/var/crashes/core.contrail-contro.26071.kvmsrv08.1426787910
Exception KeyError: KeyError(139864902106960,) in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
root@kvmsrv08:~# contrail-status --debug
== Contrail vRouter ==
supervisor-vrouter: active
supervisor-vrouter: ['contrail-vrouter-agent active pid 21905, uptime 1:41:52', 'contrail-vrouter-nodemgr active pid 3030, uptime 9 days, 10:58:39', '']
contrail-vrouter-agent: DEFAULT/S.http_server_port not present
contrail-vrouter-agent initializing
contrail-vrouter-nodemgr active

== Contrail Control ==
supervisor-control: active
supervisor-control: ['contrail-control failed Exited too quickly (process log may have details)', 'contrail-control-nodemgr active pid 3128, uptime 9 days, 10:58:39', 'contrail-dns active pid 49569, uptime 9:59:18', 'contrail-named active pid 3142, uptime 9 days, 10:58:39', '']
contrail-control failed
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
supervisor-analytics: ['contrail-alarm-gen active pid 3140, uptime 9 days, 10:58:40', 'contrail-analytics-api active pid 3138, uptime 9 days, 10:58:40', 'contrail-analytics-nodemgr active pid 3089, uptime 9 days, 10:58:40', 'contrail-collector active pid 3093, uptime 9 days, 10:58:40', 'contrail-query-engine active pid 3108, uptime 9 days, 10:58:40', 'contrail-snmp-collector active pid 3090, uptime 9 days, 10:58:40', 'contrail-topology active pid 3091, uptime 9 days, 10:58:40', '']
contrail-alarm-gen: DEFAULT/S.http_server_port not present
contrail-alarm-gen active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==
supervisor-config: active
supervisor-config: ['contrail-api:0 active pid 36013, uptime 2 days, 9:08:13', 'contrail-config-nodemgr active pid 36010, uptime 2 days, 9:08:13', 'contrail-device-manager active pid 36014, uptime 2 days, 9:08:13', 'contrail-discovery:0 active pid 36012, uptime 2 days, 9:08:13', 'contrail-schema active pid 36015, uptime 2 days, 9:08:13', 'contrail-svc-monitor active pid 54831, uptime 2 days, 7:32:11', 'ifmap active pid 36011, uptime 2 days, 9:08:13', '']
contrail-api:0: DEFAULT/S.http_server_port not present
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager: DEFAULT/S.http_server_port not present
contrail-device-manager active
contrail-discovery:0 active
contrail-schema: DEFAULT/S.http_server_port not present
contrail-schema active
contrail-svc-monitor: DEFAULT/S.http_server_port not present
contrail-svc-monitor active
ifmap active

== Contrail Web UI ==
supervisor-webui: active
supervisor-webui: ['contrail-webui active pid 3021, uptime 9 days, 10:58:40', 'contrail-webui-middleware active pid 3025, uptime 9 days, 10:58:40', '']
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
supervisor-database: active
supervisor-database: ['contrail-database active pid 24724, uptime 2 days, 9:19:29', 'contrail-database-nodemgr active pid 24722, uptime 2 days, 9:19:29', 'kafka active pid 24723, uptime 2 days, 9:19:29', '']
contrail-database active
contrail-database-nodemgr: DEFAULT/S.http_server_port not present
contrail-database-nodemgr active
kafka active

== Contrail Support Services ==
supervisor-support-service: active
supervisor-support-service: ['rabbitmq-server active pid 3001, uptime 9 days, 10:58:40', '']
rabbitmq-server active

========Run time service failures=============
/var/crashes/core.contrail-dns.48380.kvmsrv08.1426788642
/var/crashes/core.contrail-dns.25674.kvmsrv08.1426787913
/var/crashes/core.contrail-contro.25182.kvmsrv08.1426787906
/var/crashes/core.contrail-contro.24331.kvmsrv08.1426787903
/var/crashes/core.contrail-vroute.3033.kvmsrv08.1426789701
/var/crashes/core.contrail-dns.3141.kvmsrv08.1426787901
/var/crashes/core.contrail-dns.23999.kvmsrv08.1426787907
/var/crashes/core.contrail-contro.23948.kvmsrv08.1426787901
/var/crashes/core.contrail-contro.3139.kvmsrv08.1426787901
/var/crashes/core.contrail-contro.26071.kvmsrv08.1426787910
Exception KeyError: KeyError(140504852893520,) in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
root@kvmsrv08:~#

[Jaya (jayadevi)]

uploaded crash files,gdb did not return much info.
restarted vrouter to proceed with my testing.

root@kvmsrv08:/var/crashes# gdb core.contrail-vroute.3033.kvmsrv08.1426789701
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
"/var/crashes/core.contrail-vroute.3033.kvmsrv08.1426789701": not in executable format: File format not recognized
(gdb) bt
No stack.
(gdb) exit
Undefined command: "exit". Try "help".
(gdb) q
root@kvmsrv08:/var/crashes# ls -ltr
total 657392
-rw------- 1 contrail contrail 229175296 Mar 19 10:58 core.contrail-dns.3141.kvmsrv08.1426787901
-rw------- 1 contrail contrail 234889216 Mar 19 10:58 core.contrail-contro.3139.kvmsrv08.1426787901
-rw------- 1 contrail contrail 232099840 Mar 19 10:58 core.contrail-contro.23948.kvmsrv08.1426787901
-rw------- 1 contrail contrail 228188160 Mar 19 10:58 core.contrail-contro.24331.kvmsrv08.1426787903
-rw------- 1 contrail contrail 231997440 Mar 19 10:58 core.contrail-contro.25182.kvmsrv08.1426787906
-rw------- 1 contrail contrail 227655680 Mar 19 10:58 core.contrail-dns.23999.kvmsrv08.1426787907
-rw------- 1 contrail contrail 227885056 Mar 19 10:58 core.contrail-contro.26071.kvmsrv08.1426787910
-rw------- 1 contrail contrail 227647488 Mar 19 10:58 core.contrail-dns.25674.kvmsrv08.1426787913
-rw------- 1 contrail contrail 227713024 Mar 19 11:10 core.contrail-dns.48380.kvmsrv08.1426788642
-rw------- 1 root root 237846528 Mar 19 11:28 core.contrail-vroute.3033.kvmsrv08.1426789701
root@kvmsrv08:/var/crashes# gdb core.contrail-contro.26071.kvmsrv08.1426787910
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
"/var/crashes/core.contrail-contro.26071.kvmsrv08.1426787910": not in executable format: File format not recognized
(gdb) bt
No stack.
(gdb) bt
No stack.
(gdb) exit
Undefined command: "exit". Try "help".
(gdb) quit
root@kvmsrv08:/var/c...

[Tapan Karwa (tkarwa)]

All the control-node and dns cores are the exact same backtrace and values.

Will post details soon.
/github-build/mainline/2550/ubuntu-14-04/icehouse/store/sandbox/build/production/control-node/contrail-control
/github-build/mainline/2550/ubuntu-14-04/icehouse/store/sandbox/build/production/dns/contrail-dns

[Tapan Karwa (tkarwa)]

We will have to change this function to include the node's table name too i.e. use node->ToString() instead of name().

std::string IFMapLinkTable::LinkKey(const string &metadata,
                                       IFMapNode *left, IFMapNode *right) {
   ostringstream oss;
   if (left->IsLess(*right)) {
       oss << metadata << "," << left->name() << "," << right->name();
   } else {
       oss << metadata << "," << right->name() << "," << left->name();
   }
   return oss.str();
}

EG:

We are actually trying to add a link between midnode and second (which is left and right below).

(gdb) p left->name_
"attr(default-domain:admin:6060,default-domain:admin:6060)" <<<< note the 2 strings inside the (). they are the same.
(gdb) p right->name_
"default-domain:admin:6060"

But, we use IsLess() and so it returns the link between first and midnode i.e. the left_node_ and right_node_ in the link.

(gdb) p link->left_node_->name_
"default-domain:admin:6060"
(gdb) p link->right_node_->name_
"attr(default-domain:admin:6060,default-domain:admin:6060)"

1.
(gdb) p link->left_node_->table_->name_
"__ifmap__.virtual_network.0"
2.
(gdb) p link->right_node_->table_->name_
"__ifmap__.virtual_network_network_ipam.0"
3.
(gdb) p left->table_->name_
"__ifmap__.virtual_network_network_ipam.0"
4.
(gdb) p right->table_->name_
"__ifmap__.network_ipam.0"

So, #1 and #4 belong to different tables.
But, they have the same name and we use IsLess() to decide on the sequence of the string. Thats the problem.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/8661
Submitter: Tapan Karwa (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/8662
Submitter: Tapan Karwa (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/8663
Submitter: Tapan Karwa (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/8664
Submitter: Tapan Karwa (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/8661
Committed: http://github.org/Juniper/contrail-controller/commit/5ae7a00c790b3d7126decf73167c2ceeddd657a6
Submitter: Zuul
Branch: master

commit 5ae7a00c790b3d7126decf73167c2ceeddd657a6
Author: Tapan Karwa <email address hidden>
Date: Thu Mar 26 11:36:08 2015 -0700

Add table name of the nodes (of the link) to create the key for links in the
link-table.

There can be cases where we add a link between 2 nodes with the same name. This
causes a problem for links with metadata. While adding the link between midnode
and second, we find the link between first and midnode leading to an assert.

Fix is to use the table-name of both the nodes of the link while creating the
link's key.

Change-Id: Ibbaeff7845b74da56abe1488dced24f131d75377
Closes-Bug:1434342

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/8662
Committed: http://github.org/Juniper/contrail-controller/commit/b03cba36f7196ae5dfe29a54f84ab7722b4c317b
Submitter: Zuul
Branch: R2.1

commit b03cba36f7196ae5dfe29a54f84ab7722b4c317b
Author: Tapan Karwa <email address hidden>
Date: Thu Mar 26 11:36:08 2015 -0700

Add table name of the nodes (of the link) to create the key for links in the
link-table.

There can be cases where we add a link between 2 nodes with the same name. This
causes a problem for links with metadata. While adding the link between midnode
and second, we find the link between first and midnode leading to an assert.

Fix is to use the table-name of both the nodes of the link while creating the
link's key.

Change-Id: Ibbaeff7845b74da56abe1488dced24f131d75377
Closes-Bug:1434342

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/8663
Committed: http://github.org/Juniper/contrail-controller/commit/c61ea4b719433ba25468e715101ebeeb69321609
Submitter: Zuul
Branch: R2.0

commit c61ea4b719433ba25468e715101ebeeb69321609
Author: Tapan Karwa <email address hidden>
Date: Thu Mar 26 11:36:08 2015 -0700

Add table name of the nodes (of the link) to create the key for links in the
link-table.

There can be cases where we add a link between 2 nodes with the same name. This
causes a problem for links with metadata. While adding the link between midnode
and second, we find the link between first and midnode leading to an assert.

Fix is to use the table-name of both the nodes of the link while creating the
link's key.

Change-Id: Ibbaeff7845b74da56abe1488dced24f131d75377
Closes-Bug:1434342

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/8664
Committed: http://github.org/Juniper/contrail-controller/commit/15be7ac4506cd46c61d129d8594625b03f4b72c7
Submitter: Zuul
Branch: R1.10

commit 15be7ac4506cd46c61d129d8594625b03f4b72c7
Author: Tapan Karwa <email address hidden>
Date: Thu Mar 26 12:25:33 2015 -0700

Add table name of the nodes (of the link) to create the key for links in the
link-table.

There can be cases where we add a link between 2 nodes with the same name. This
causes a problem for links with metadata. While adding the link between midnode
and second, we find the link between first and midnode leading to an assert.

Fix is to use the table-name of both the nodes of the link while creating the
link's key.

Change-Id: Ibbaeff7845b74da56abe1488dced24f131d75377
Closes-Bug:1434342