Contrail 2.22 : vrouter crash

[Deepak Jeyaraman]

Using 2.22 contrail (see build details below)

1 node control/compute - ccra-06
6 compute nodes - ccra-05, 07, 08, 09, 10, 12

initially had issue pinging the management IPs of instances in ccra-09 (only that compute node had this issue)

later when I cleaned up all the heat stacks (240 total) , checked if all the nova instances were killed. They were killed as expected, heat stacks cleaned up - no issues.

But, still was seeing the instances of ccra-09 all (34 of them) showing up on the contrail dashboard, all these instances management IPs were those that werent pingable initially.

I am attaching the logs / screenshot of the contrail dash

==================

All Logs are here:

jdeepak@ttsv-shell04 /volume/vmcores/sagrawal/jdeepak_temp/NFV/contrail_bug/1520823
$ ls -tlr
total 1932892
-rwxrwxrwx 1 jdeepak software 1971507200 Nov 28 12:59 core.contrail-vroute.3265.ccra-09.1448640291
drwxrwxrwx 4 jdeepak software 4096 Nov 28 13:00 logs
jdeepak@ttsv-shell04 /volume/vmcores/sagrawal/jdeepak_temp/NFV/contrail_bug/1520823

=================

root@ccra-09:~# virsh list
 Id Name State
----------------------------------------------------

root@ccra-09:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent timeout
contrail-vrouter-nodemgr active

========Run time service failures=============
/var/crashes/core.contrail-vroute.3265.ccra-09.1448640291 <<<<<<<<<<
root@ccra-09:~# cd /var/crashes/
root@ccra-09:/var/crashes# ls -ltr
total 1055924
-rw------- 1 root root 1971507200 Nov 27 08:04 core.contrail-vroute.3265.ccra-09.1448640291
root@ccra-09:/var/crashes# uname -a
Linux ccra-09 3.13.0-40-generic #69-Ubuntu SMP Thu Nov 13 17:53:56 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
root@ccra-09:/var/crashes# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04 LTS
Release: 14.04
Codename: trusty

root@ccra-06:/var/log/heat# dpkg -l | grep contrail
ii contrail-analytics 2.22-109 amd64 OpenContrail analytics
ii contrail-config 2.22-109 amd64 OpenContrail configuration management
ii contrail-config-openstack 2.22-109 amd64 OpenContrail configuration OpenStack module
ii contrail-control 2.22-109 amd64 OpenContrail control-node
ii contrail-dns 2.22-109 amd64 OpenContrail control-dns
ii contrail-f5 2.22-109 all Big IP interface for f5 load balancer
ii contrail-fabric-utils 2.22-109 all Contrail Fabric Utilities for cluster management
ii contrail-heat 2.22-109 all OpenStack virtual network service - Opencontrail
ii contrail-install-packages 2.22-109~icehouse all Contrail Installer Packages - Container of debian packages
ii contrail-lib 2.22-109 amd64 OpenContrail libraries
ii contrail-nodemgr 2.22-109 all Contrail Config API Library package
ii contrail-nova-networkapi 2.22-109 all OpenContrail networking api for nova
ii contrail-nova-vif 2.22-109 all OpenContrail interface driver for nova-compute
ii contrail-openstack 2.22-109 all Contrail Openstack composite debian package
ii contrail-openstack-analytics 2.22-109 all Contrail Openstack Analytics composite debian package
ii contrail-openstack-config 2.22-109 all Contrail Openstack Config composite debian package
ii contrail-openstack-control 2.22-109 all Contrail Openstack Control composite debian package
ii contrail-openstack-dashboard 2.22-109 all Openstack Dashboard implementation for Contrail Neutron Plugin.
ii contrail-openstack-database 2.22-109 all Contrail Openstack database composite debian package
ii contrail-openstack-vrouter 2.22-109 all Contrail Openstack vRouter composite debian package
ii contrail-openstack-webui 2.22-109 all Contrail Openstack WebUI composite debian package
ii contrail-setup 2.22-109 all Contrail Setup package with scripts for provisioning
ii contrail-utils 2.22-109 amd64 OpenContrail tools and utilities
ii contrail-vrouter-3.13.0-40-generic 2.22-109 all OpenContrail VRouter - precompiled module version
ii contrail-vrouter-agent 2.22-109 amd64 OpenContrail vrouter agent
ii contrail-vrouter-common 2.22-109 all Contrail vRouter composite debian package
ii contrail-vrouter-init 2.22-109 all OpenContrail compute-node startup and monitoring scripts.
ii contrail-vrouter-utils 2.22-109 amd64 OpenContrail VRouter - Utilities
ii contrail-web-controller 2.22-109 amd64 Contrail Systems Web UI Feature
ii contrail-web-core 2.22-109 amd64 Contrail Systems Web UI
ii ifmap-server 0.3.2-1contrail1 all irond is a IF-MAP MAP server
ii kafka 2.9.2-0.8.2.0-0contrail0 amd64 <insert up to 60 chars description>
ii librdkafka1:amd64 0.8.5-2-0contrail0.14.04 amd64 library implementing the Apache Kafka protocol
ii neutron-plugin-contrail 2.22-109 all OpenStack virtual network service - Opencontrail
ii nodejs 0.8.15-1contrail1 amd64 Node.js event-based server-side javascript engine
ii nova-api 1:2014.1.3-0ubuntu1~cloud0.3contrail all OpenStack Compute - API frontend
ii nova-common 1:2014.1.3-0ubuntu1~cloud0.3contrail all OpenStack Compute - common files
ii nova-compute 1:2014.1.3-0ubuntu1~cloud0.3contrail all OpenStack Compute - compute node base
ii nova-compute-kvm 1:2014.1.3-0ubuntu1~cloud0.3contrail all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 1:2014.1.3-0ubuntu1~cloud0.3contrail all OpenStack Compute - compute node libvirt support
ii nova-conductor 1:2014.1.3-0ubuntu1~cloud0.3contrail all OpenStack Compute - conductor service
ii nova-console 1:2014.1.3-0ubuntu1~cloud0.3contrail all OpenStack Compute - Console
ii nova-consoleauth 1:2014.1.3-0ubuntu1~cloud0.3contrail all OpenStack Compute - Console Authenticator
ii nova-novncproxy 1:2014.1.3-0ubuntu1~cloud0.3contrail all OpenStack Compute - NoVNC proxy
ii nova-objectstore 1:2014.1.3-0ubuntu1~cloud0.3contrail all OpenStack Compute - object store
ii nova-scheduler 1:2014.1.3-0ubuntu1~cloud0.3contrail all OpenStack Compute - virtual machine scheduler
ii python-backports.ssl-match-hostname 3.4.0.2-1contrail1 all The ssl.match_hostname() function from Python 3.4
ii python-certifi 1.0.1-1contrail1 all Python SSL Certificates
ii python-consistent-hash 1.0-0contrail1 amd64 <insert up to 60 chars description>
ii python-contrail 2.22-109 amd64 OpenContrail python-libs
ii python-contrail-vrouter-api 2.22-109 amd64 OpenContrail vrouter agent api
ii python-geventhttpclient 1.1.0-1contrail1 amd64 http client library for gevent
ii python-kafka-python 0.9.2-0contrail0 all Pure Python client for Apache Kafka
ii python-kazoo 1.3.1-1contrail2 all higher level API to Apache Zookeeper (Python 2)
ii python-ncclient 0.4.1-1contrail1 all Python library for NETCONF clients
ii python-neutronclient 2:2.3.4-0ubuntu1.2contrail all client - Neutron is a virtual network service for Openstack
ii python-nova 1:2014.1.3-0ubuntu1~cloud0.3contrail all OpenStack Compute Python libraries
ii python-opencontrail-vrouter-netns 2.22-109 amd64 OpenContrail vrouter network namespace package
ii python-pycassa 1.11.0-1contrail2 all Client library for Apache Cassandra
ii python-redis 2.8.0-1contrail1 all The Python interface to the Redis key-value store.
ii python-xmltodict 0.9.0-1contrail1 all Makes working with XML feel like you are working with JSON
root@ccra-06:/var/log/heat#
root@ccra-06:/var/log/heat#
root@ccra-06:/var/log/heat# uname -a
Linux ccra-06 3.13.0-40-generic #69-Ubuntu SMP Thu Nov 13 17:53:56 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
root@ccra-06:/var/log/heat# lsb_release -r
Release: 14.04
root@ccra-06:/var/log/heat# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04 LTS
Release: 14.04
Codename: trusty
root@ccra-06:/var/log/heat#

[Hari Prasad Killi (haripk)]

The core file is not available. /volume/vmcores/sagrawal/jdeepak_temp/NFV/contrail_bug/1520823/core.contrail-vroute.3265.ccra-09.1448640291 is actually a tar file of the log files. Please try with the latest build and check.