[ubuntu-icehouse-R1.10-#34] vizd cored at operator()

[Prashant Shetty]

collector crashed during flow table query.

Crash-Decode:

Core was generated by `/usr/bin/contrail-collector'.
Program terminated with signal 6, Aborted.
#0 0x00007f4720646425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007f4720646425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4720649b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f472063f0ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f472063f192 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x000000000058e295 in operator() (a1=0x246a350, a0=..., this=0x7f471efa8af0) at build/include/boost/function/function_template.hpp:760
#5 SandeshReader::OnRead (this=0x24234e0, buffer=...) at tools/sandesh/library/cpp/sandesh_session.cc:547
#6 0x000000000058c2da in SandeshSession::OnRead (this=0x246a350, buffer=...) at tools/sandesh/library/cpp/sandesh_session.cc:357
#7 0x000000000053c778 in operator() (a1=..., p=<optimized out>, this=<optimized out>) at build/include/boost/bind/mem_fn_template.hpp:165
#8 operator()<boost::_mfi::mf1<void, TcpSession, boost::asio::const_buffer>, boost::_bi::list1<boost::asio::const_buffer&> > (a=<synthetic pointer>, f=...,
    this=<optimized out>) at build/include/boost/bind/bind.hpp:313
#9 operator()<boost::asio::const_buffer> (a1=..., this=<optimized out>) at build/include/boost/bind/bind_template.hpp:32
#10 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, TcpSession, boost::asio::const_buffer>, boost::_bi::list2<boost::_bi::value<TcpSession*>, boost::arg<1> > >, void, boost::asio::const_buffer>::invoke (function_obj_ptr=..., a0=...) at build/include/boost/function/function_template.hpp:153
#11 0x000000000053d4a4 in operator() (a0=..., this=0x2465b20) at build/include/boost/function/function_template.hpp:763
#12 TcpSession::Reader::Run (this=0x2465af0) at controller/src/io/tcp_session.cc:36
#13 0x00000000006b8bc0 in TaskImpl::execute (this=0x2434840) at controller/src/base/task.cc:224
#14 0x00007f4721ff2ece in ?? () from /usr/lib/libtbb_debug.so.2
#15 0x00007f4721fe9e0b in ?? () from /usr/lib/libtbb_debug.so.2
#16 0x00007f4721fe86f2 in ?? () from /usr/lib/libtbb_debug.so.2
#17 0x00007f4721fe33ce in ?? () from /usr/lib/libtbb_debug.so.2
#18 0x00007f4721fe3270 in ?? () from /usr/lib/libtbb_debug.so.2
#19 0x00007f4721b3ae9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#20 0x00007f4720703ccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#21 0x0000000000000000 in ?? ()
(gdb)

[Prashant Shetty (shettyp)]

All logs/cores will be at /cs-shared/test_runs/nodea35/2014_09_08_19_13_06 on nodeb10.englab.juniper.net

Mayamruga is down currently, will copy cores to : http://mayamruga.englab.juniper.net/bugs/<bug-ID>

[Megh Bhatt (meghb)]

I'm not able to decode the core.

root@a6s23:/var/tmp# gdb /var/tmp/vizd nodea35_core.contrail-collec.19119.nodea35.1410186621
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /var/tmp/vizd...done.

warning: core file may not match specified executable file.
[New LWP 19146]
[New LWP 19119]
[New LWP 19144]
[New LWP 19147]
[New LWP 19145]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff58da2000
Core was generated by `/usr/bin/contrail-collector'.
Program terminated with signal 6, Aborted.
#0 0x00007f4720646425 in sigandset () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007f4720646425 in sigandset () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4720649b8b in qsort_r () from /lib/x86_64-linux-gnu/libc.so.6
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) info threads
  Id Target Id Frame
  5 Thread 0x7f471f3aa700 (LWP 19145) 0x00007f47206ffee9 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
  4 Thread 0x7f471eba8700 (LWP 19147) 0x00007f47206ffee9 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
  3 Thread 0x7f471f7ab700 (LWP 19144) 0x00007f47206ffee9 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
  2 Thread 0x7f4722d48740 (LWP 19119) 0x00007f4720704363 in recv () from /lib/x86_64-linux-gnu/libc.so.6
* 1 Thread 0x7f471efa9700 (LWP 19146) 0x00007f4720646425 in sigandset () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) thread apply all bt

Thread 5 (Thread 0x7f471f3aa700 (LWP 19145)):
#0 0x00007f47206ffee9 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4721fd0d32 in ?? () from /usr/lib/libtbb_debug.so.2
#2 0x00007f4721fd108d in ?? () from /usr/lib/libtbb_debug.so.2
#3 0x00007f4721fe3d9f in rml::internal::thread_monitor::commit_wait(rml::internal::thread_monitor::cookie&) () from /usr/lib/libtbb_debug.so.2
#4 0x00007f4721fe343d in ?? () from /usr/lib/libtbb_debug.so.2
#5 0x00007f4721fe3270 in ?? () from /usr/lib/libtbb_debug.so.2
#6 0x00007f4721b3ae9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007f4720703ccd in personality () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f471eba8700 (LWP 19147)):
#0 0x00007f47206ffee9 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4721fd0d32 in ?? () from /usr/lib/libtbb_debug.so.2
#2 0x00007f4721fd108d in ?? () from /usr/lib/libtbb_debug.so.2
#3 0x00007f4721fe3d9f in rml::internal::thread_monitor::commit_wait(rml::internal::thread_monitor::cookie&) () from /usr/lib/libtbb_debug.s...

[Megh Bhatt (meghb)]

Removed blocker tag in absence of core file that enables to decode/debug

[Megh Bhatt (meghb)]

Fixed via https://github.com/Juniper/contrail-sandesh/commit/fc865df0e682e5b63c852fdf7b4710035000e518