14.1X53D26/Contrail: Observed this tor-agent core on TSN when I rebooted a tor which is connected to tsn/vrouter

[Praveen Shetty]

Observed this tor-agent core when I rebooted a tor which is connected to tsn/vrouter

Senthil looked into this issue and asked me file PR

below is the BT
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f5c913e6cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007f5c913e6cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f5c913ea0d8 in __GI_abort () at abort.c:89
#2 0x00007f5c913dfb86 in __assert_fail_base (fmt=0x7f5c91530830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xf3aaf5 "0",
    file=file@entry=0xf47a80 "controller/src/vnsw/agent/oper/vrf.cc", line=line@entry=314, function=function@entry=0xf47c40 "bool VrfEntry::DeleteTimeout()") at assert.c:92
#3 0x00007f5c913dfc32 in __GI___assert_fail (assertion=0xf3aaf5 "0", file=0xf47a80 "controller/src/vnsw/agent/oper/vrf.cc", line=314, function=0xf47c40 "bool VrfEntry::DeleteTimeout()")
    at assert.c:101
#4 0x0000000000969bad in VrfEntry::DeleteTimeout() ()
#5 0x0000000000f0e3e9 in Timer::TimerTask::Run() ()
#6 0x0000000000f05c30 in TaskImpl::execute() ()
#7 0x00007f5c91fb5b3a in ?? () from /usr/lib/libtbb.so.2
#8 0x00007f5c91fb1816 in ?? () from /usr/lib/libtbb.so.2
#9 0x00007f5c91fb0f4b in ?? () from /usr/lib/libtbb.so.2
#10 0x00007f5c91fad0ff in ?? () from /usr/lib/libtbb.so.2
#11 0x00007f5c91fad2f9 in ?? () from /usr/lib/libtbb.so.2
#12 0x00007f5c921d1182 in start_thread (arg=0x7f5c89830700) at pthread_create.c:312
#13 0x00007f5c914aa47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

root@cd-st-lnxserver-02:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-tor-agent-1 active
contrail-tor-agent-2 active
contrail-tor-agent-3 active
contrail-tor-agent-4 active
contrail-tor-agent-5 active
contrail-tor-agent-6 active
contrail-tor-agent-7 active
contrail-tor-agent-8 active
contrail-tor-agent-9 active
contrail-vrouter-agent initializing (Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down)
contrail-vrouter-nodemgr active

========Run time service failures=============
/var/crashes/core.contrail-tor-ag.23842.cd-st-lnxserver-02.1430529693
/var/crashes/core.contrail-tor-ag.19481.cd-st-lnxserver-02.1430531933
/var/crashes/core.contrail-tor-ag.28532.cd-st-lnxserver-02.1430848529
/var/crashes/core.contrail-tor-ag.18777.cd-st-lnxserver-02.1430531922
/var/crashes/core.contrail-tor-ag.30145.cd-st-lnxserver-02.1430932098
/var/crashes/core.contrail-tor-ag.29861.cd-st-lnxserver-02.1430932098
/var/crashes/core.contrail-tor-ag.29907.cd-st-lnxserver-02.1430932098
/var/crashes/core.contrail-tor-ag.25483.cd-st-lnxserver-02.1430531918
/var/crashes/core.contrail-tor-ag.10624.cd-st-lnxserver-02.1429562326
/var/crashes/core.contrail-tor-ag.17946.cd-st-lnxserver-02.1430531924
/var/crashes/core.contrail-tor-ag.26885.cd-st-lnxserver-02.1430848528
/var/crashes/core.contrail-tor-ag.7705.cd-st-lnxserver-02.1430529732
/var/crashes/core.contrail-tor-ag.22511.cd-st-lnxserver-02.1430509447
/var/crashes/core.contrail-tor-ag.26904.cd-st-lnxserver-02.1430529828
/var/crashes/core.contrail-tor-ag.24452.cd-st-lnxserver-02.1430529722
/var/crashes/core.contrail-tor-ag.29724.cd-st-lnxserver-02.1430932098
/var/crashes/core.contrail-tor-ag.7706.cd-st-lnxserver-02.1430529756
/var/crashes/core.contrail-vroute.8937.cd-st-lnxserver-02.1430531916
/var/crashes/core.contrail-tor-ag.4255.cd-st-lnxserver-02.1430848529
/var/crashes/core.contrail-vroute.3306.cd-st-lnxserver-02.1430848529
/var/crashes/core.contrail-tor-ag.29890.cd-st-lnxserver-02.1430932098
/var/crashes/core.contrail-vroute.1651.cd-st-lnxserver-02.1430932098
/var/crashes/core.contrail-tor-ag.7708.cd-st-lnxserver-02.1430529945
/var/crashes/core.contrail-tor-ag.25862.cd-st-lnxserver-02.1430531923
/var/crashes/core.contrail-tor-ag.27584.cd-st-lnxserver-02.1430509530
/var/crashes/core.contrail-tor-ag.10516.cd-st-lnxserver-02.1429562275
/var/crashes/core.contrail-tor-ag.10570.cd-st-lnxserver-02.1429562298
/var/crashes/core.contrail-tor-ag.30242.cd-st-lnxserver-02.1430932098
/var/crashes/core.contrail-tor-ag.23803.cd-st-lnxserver-02.1430530060
/var/crashes/core.contrail-tor-ag.7305.cd-st-lnxserver-02.1430529844
/var/crashes/core.contrail-tor-ag.6761.cd-st-lnxserver-02.1430848529
/var/crashes/core.contrail-tor-ag.14299.cd-st-lnxserver-02.1429564027
/var/crashes/core.contrail-tor-ag.4781.cd-st-lnxserver-02.1430848528
/var/crashes/core.contrail-tor-ag.19475.cd-st-lnxserver-02.1429564061
/var/crashes/core.contrail-tor-ag.30812.cd-st-lnxserver-02.1430932098
/var/crashes/core.contrail-tor-ag.25245.cd-st-lnxserver-02.1430509659
/var/crashes/core.contrail-tor-ag.3229.cd-st-lnxserver-02.1430848528
/var/crashes/core.contrail-tor-ag.10931.cd-st-lnxserver-02.1429562349
/var/crashes/core.contrail-tor-ag.3588.cd-st-lnxserver-02.1430848529
/var/crashes/core.contrail-vroute.2527.cd-st-lnxserver-02.1429773800
/var/crashes/core.contrail-tor-ag.20478.cd-st-lnxserver-02.1430531926
/var/crashes/core.contrail-tor-ag.2051.cd-st-lnxserver-02.1430860197
/var/crashes/core.contrail-tor-ag.7707.cd-st-lnxserver-02.1430529896
/var/crashes/core.contrail-tor-ag.30146.cd-st-lnxserver-02.1430932105
/var/crashes/core.contrail-tor-ag.11299.cd-st-lnxserver-02.1429562883
root@cd-st-lnxserver-02:~#

root@cd-st-lnxserver-02:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-fabric-utils 2.11-55 55
contrail-install-packages 2.11-55~icehouse 55
contrail-lib 2.11-55 55
contrail-nodemgr 2.11-55 55
contrail-nova-vif 2.11-55 55
contrail-openstack-vrouter 2.10-40 55
contrail-setup 2.11-55 55
contrail-utils 2.11-55 55
contrail-vrouter-agent 2.11-55 55
contrail-vrouter-common 2.11-55 55
contrail-vrouter-dkms 2.11-55 55
contrail-vrouter-init 2.11-55 55
contrail-vrouter-utils 2.11-55 55
nova-common 1:2014.1.3-0ubuntu1~cloud0.2contrail55
nova-compute 1:2014.1.3-0ubuntu1~cloud0.2contrail55
nova-compute-kvm 1:2014.1.3-0ubuntu1~cloud0.2contrail55
nova-compute-libvirt 1:2014.1.3-0ubuntu1~cloud0.2contrail55
python-contrail 2.11-55 55
python-contrail-vrouter-api 2.11-55 55
python-neutronclient 2:2.3.4-0ubuntu1.2contrail 55
python-nova 1:2014.1.3-0ubuntu1~cloud0.2contrail55
python-opencontrail-vrouter-netns 2.11-55 55
root@cd-st-lnxserver-02:~#

[Hari Prasad Killi (haripk)]

This issue was fixed as part of the following:
https://bugs.launchpad.net/juniperopenstack/+bug/1433757
https://bugs.launchpad.net/juniperopenstack/+bug/1440990

One more remaining is tracked as https://bugs.launchpad.net/juniperopenstack/+bug/1452362

The following issues were also seen here:
https://bugs.launchpad.net/juniperopenstack/+bug/1443720
https://bugs.launchpad.net/juniperopenstack/+bug/1438527