[3.1.2.0-62] Tor Agent Crash @ VrfEntry::DeleteTimeout()

[chhandak]

Crash happened on scale setup of QFX team.

Backtrace
-----------
root@NTTC-TSN-1:/var/crashes# gdb /usr/bin/contrail-tor-agent /var/crashes/core.contrail-tor-ag.2734.NTTC-TSN-1.1493092801
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/contrail-tor-agent...(no debugging symbols found)...done.

warning: core file may not match specified executable file.
[New LWP 3439]
[New LWP 7515]
[New LWP 7518]
[New LWP 3089]
[New LWP 2734]
[New LWP 13499]
[New LWP 3088]
[New LWP 14876]
[New LWP 14875]
[New LWP 3084]
[New LWP 3083]
[New LWP 3440]
[New LWP 13498]
[New LWP 7517]
[New LWP 7516]
[New LWP 3086]
[New LWP 3085]
[New LWP 3090]
[New LWP 3087]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-tor-agent --config_file /etc/contrail/contrail-tor-agent-2.co'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007ff777977c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ff777977c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77797b028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ff777970bf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ff777970ca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000a52942 in VrfEntry::DeleteTimeout() ()
#5 0x000000000104f049 in Timer::TimerTask::Run() ()
#6 0x0000000001047dff in TaskImpl::execute() ()
#7 0x00007ff778546b3a in ?? () from /usr/lib/libtbb.so.2
#8 0x00007ff778542816 in ?? () from /usr/lib/libtbb.so.2
#9 0x00007ff778541f4b in ?? () from /usr/lib/libtbb.so.2
#10 0x00007ff77853e0ff in ?? () from /usr/lib/libtbb.so.2
#11 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#12 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) thr info
No symbol "info" in current context.
(gdb) th info
Ambiguous command "th info": thbreak, thread.
(gdb) thread apply all bt

Thread 19 (Thread 0x7ff76bbfe700 (LWP 3087)):
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 18 (Thread 0x7ff76affb700 (LWP 3090)):
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 17 (Thread 0x7ff7707f9700 (LWP 3085)):
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 16 (Thread 0x7ff76bfff700 (LWP 3086)):
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 15 (Thread 0x7ff7696f7700 (LWP 7516)):
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 14 (Thread 0x7ff7692f6700 (LWP 7517)):
#0 0x00007ff777a0c2a7 in sched_yield () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff778546567 in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff7785427f3 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778541f4b in ?? () from /usr/lib/libtbb.so.2
#4 0x00007ff77853e0ff in ?? () from /usr/lib/libtbb.so.2
#5 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#6 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 13 (Thread 0x7ff768af4700 (LWP 13498)):
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 12 (Thread 0x7ff76a6f9700 (LWP 3440)):
---Type <return> to continue, or q <return> to quit---
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 11 (Thread 0x7ff770ffb700 (LWP 3083)):
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 10 (Thread 0x7ff770bfa700 (LWP 3084)):
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 9 (Thread 0x7ff743fff700 (LWP 14875)):
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 8 (Thread 0x7ff743bfe700 (LWP 14876)):
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 7 (Thread 0x7ff76b7fd700 (LWP 3088)):
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 6 (Thread 0x7ff7686f3700 (LWP 13499)):
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 5 (Thread 0x7ff77a6df8c0 (LWP 2734)):
#0 0x00007ff777a3ba13 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x000000000097ab05 in boost::asio::detail::epoll_reactor::run(bool, boost::asio::detail::op_queue<boost::asio::detail::task_io_service_operation>&) ()
#2 0x000000000097be02 in boost::asio::detail::task_io_service::run(boost::system::error_code&) ()
#3 0x0000000000f2a0b1 in EventManager::Run() ()
#4 0x0000000000791453 in main ()
---Type <return> to continue, or q <return> to quit---

Thread 4 (Thread 0x7ff76b3fc700 (LWP 3089)):
#0 0x00007ff777a35709 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77853e2db in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 3 (Thread 0x7ff768ef5700 (LWP 7518)):
#0 0x00007ff777a0c2a7 in sched_yield () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff778546567 in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff7785427f3 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778541f4b in ?? () from /usr/lib/libtbb.so.2
#4 0x00007ff77853e0ff in ?? () from /usr/lib/libtbb.so.2
#5 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#6 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 2 (Thread 0x7ff769af8700 (LWP 7515)):
#0 0x00007ff777a0c2a7 in sched_yield () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff778546567 in ?? () from /usr/lib/libtbb.so.2
#2 0x00007ff7785427f3 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007ff778541f4b in ?? () from /usr/lib/libtbb.so.2
#4 0x00007ff77853e0ff in ?? () from /usr/lib/libtbb.so.2
#5 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#6 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 1 (Thread 0x7ff76aafa700 (LWP 3439)):
#0 0x00007ff777977c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff77797b028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ff777970bf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ff777970ca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000a52942 in VrfEntry::DeleteTimeout() ()
#5 0x000000000104f049 in Timer::TimerTask::Run() ()
#6 0x0000000001047dff in TaskImpl::execute() ()
#7 0x00007ff778546b3a in ?? () from /usr/lib/libtbb.so.2
#8 0x00007ff778542816 in ?? () from /usr/lib/libtbb.so.2
#9 0x00007ff778541f4b in ?? () from /usr/lib/libtbb.so.2
#10 0x00007ff77853e0ff in ?? () from /usr/lib/libtbb.so.2
#11 0x00007ff77853e2f9 in ?? () from /usr/lib/libtbb.so.2
#12 0x00007ff778762184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007ff777a3b37d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)
(gdb)
(gdb)
(gdb) quit
root@NTTC-TSN-1:/var/crashes# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-fabric-utils 3.1.2.0-62 62
contrail-install-packages 3.1.2.0-62~mitaka 62
contrail-lib 3.1.2.0-62 62
contrail-nodemgr 3.1.2.0-62 62
contrail-nova-vif 3.1.2.0-62 62
contrail-setup 3.1.2.0-62 62
contrail-utils 3.1.2.0-62 62
contrail-vrouter-3.13.0-100-generic 3.1.2.0-62 62
contrail-vrouter-agent 3.1.2.0-62 62
contrail-vrouter-common 3.1.2.0-62 62
contrail-vrouter-init 3.1.2.0-62 62

[chhandak (chhandak)]

Core and binary copied to /auto/cores/1687833

[Manish Singh (manishs)]

Multicast address has not been withdrawn by QFX.

  name_ = "06304b58-128a-48b1-8d97-3d46bf93c776",
  device_name_ = "vdc-vcf-l1",
  physical_switch_ = (boost::intrusive_ptr<KSyncEntry>) 0x7f2d50b2e4c0,
  local_mac_ref_ = (boost::intrusive_ptr<KSyncEntry>) 0x7f2d50fcc380,
  pl_create_ref_ = (boost::intrusive_ptr<KSyncEntry>) 0x0,
  vxlan_id_ = 21056,
  mcast_local_row_list_ = std::set with 1 elements = {
    [0] = 0x7f2d4e09bea0
  },

For tracking in gdb, use:
p (('OVSDB::LogicalSwitchEntry' *) 0x7f2d50fcc380)->vxlan_id_
p ((‘OVSDB::LogicalSwitchEntry' *) 0x7f2d50fcc380)->vxlan_id_

VN holding vrf reference: $20 = (VnEntry *) 0x7f2d6beb0290

[Manish Singh (manishs)]

Not contrail issue.