Bug #1456853 “contrail-collector core tor qfx bms testing” : Series r2.20 : Bugs : Juniper Openstack

Revision history for this message

Megh Bhatt (meghb) wrote on 2015-05-20:

#1

Download full text (14.7 KiB)

It seems that we cannot get the proper bt of the assert from the contrail-collector core - core.contrail-collec.2768.ubuntu-cn.1431399756

(gdb) info threads
  Id Target Id Frame
  9 Thread 0x7f55c1f5e700 (LWP 3226) 0x00007f55c83bcf45 in readdir64 () from /lib/x86_64-linux-gnu/libc.so.6
  8 Thread 0x7f55c235f700 (LWP 3220) 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
  7 Thread 0x7f55c1b5d700 (LWP 3227) 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
  6 Thread 0x7f55c2760700 (LWP 3219) 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
  5 Thread 0x7f55c135b700 (LWP 3229) 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
  4 Thread 0x7f55c2b61700 (LWP 3218) 0x00007f55c9ae43bd in read () from /lib/x86_64-linux-gnu/libpthread.so.0
  3 Thread 0x7f55c175c700 (LWP 3228) 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
  2 Thread 0x7f55c2f62700 (LWP 3217) 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
* 1 Thread 0x7f55cb41d100 (LWP 2768) 0x00007f55c8335cc9 in __open_catalog () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) thread apply all bt

Thread 9 (Thread 0x7f55c1f5e700 (LWP 3226)):
#0 0x00007f55c83bcf45 in readdir64 () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f55c83bcc96 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f55c1f5cd70 in ?? ()
#3 0x00007f55a80010a0 in ?? ()
#4 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f55c235f700 (LWP 3220)):
#0 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f55c1b5d700 (LWP 3227)):
#0 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f55c2760700 (LWP 3219)):
#0 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f55c135b700 (LWP 3229)):
#0 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f55c2b61700 (LWP 3218)):
#0 0x00007f55c9ae43bd in read () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000073ee10 in read (__nbytes=16384, __buf=0x7f55c2b5b230, __fd=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/unistd.h:44
#2 redisBufferRead (c=c@entry=0x7f55bc01a480) at build/third_party/hiredis/src/hiredis.c:1078
#3 0x000000000073f265 in redisGetReply (c=0x7f55bc01a480, reply=0x7f55c2b5f2a0) at build/third_party/hiredis/src/hiredis.c:1164
#4 0x000000000073f94f in __redisBlockForReply (c=0x7f55bc01a480) at build/third_party/hiredis/src/hiredis.c:1259
#5 redisvCommand (ap=0x7f55c2b5f2a8, format=<optimized out>, c=0x7f55bc01a480) at build/third_party/hiredis/src/hiredis.c:1269
#6 redisCommand (c=0x7f55bc01a480, format=<optimized out>) at build/third_party/hiredis/src/hiredis.c:1276
#7 0x00000000004e3d50 in RedisProcessorExec::SyncGetSeq (redis_ip=..., redis_port=<optimized out>, redis_password=..., source=..., node_type=..., module=..., instance_id=..., seqReply=...)...

It seems that we cannot get the proper bt of the assert from the contrail-collector core - core.contrail-collec.2768.ubuntu-cn.1431399756

(gdb) info threads
  Id   Target Id         Frame
  9    Thread 0x7f55c1f5e700 (LWP 3226) 0x00007f55c83bcf45 in readdir64 () from /lib/x86_64-linux-gnu/libc.so.6
  8    Thread 0x7f55c235f700 (LWP 3220) 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
  7    Thread 0x7f55c1b5d700 (LWP 3227) 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
  6    Thread 0x7f55c2760700 (LWP 3219) 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
  5    Thread 0x7f55c135b700 (LWP 3229) 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
  4    Thread 0x7f55c2b61700 (LWP 3218) 0x00007f55c9ae43bd in read () from /lib/x86_64-linux-gnu/libpthread.so.0
  3    Thread 0x7f55c175c700 (LWP 3228) 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
  2    Thread 0x7f55c2f62700 (LWP 3217) 0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
* 1    Thread 0x7f55cb41d100 (LWP 2768) 0x00007f55c8335cc9 in __open_catalog () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) thread apply all bt

Thread 9 (Thread 0x7f55c1f5e700 (LWP 3226)):
#0  0x00007f55c83bcf45 in readdir64 () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f55c83bcc96 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f55c1f5cd70 in ?? ()
#3  0x00007f55a80010a0 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f55c235f700 (LWP 3220)):
#0  0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f55c1b5d700 (LWP 3227)):
#0  0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f55c2760700 (LWP 3219)):
#0  0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f55c135b700 (LWP 3229)):
#0  0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f55c2b61700 (LWP 3218)):
#0  0x00007f55c9ae43bd in read () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x000000000073ee10 in read (__nbytes=16384, __buf=0x7f55c2b5b230, __fd=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/unistd.h:44
#2  redisBufferRead (c=c@entry=0x7f55bc01a480) at build/third_party/hiredis/src/hiredis.c:1078
#3  0x000000000073f265 in redisGetReply (c=0x7f55bc01a480, reply=0x7f55c2b5f2a0) at build/third_party/hiredis/src/hiredis.c:1164
#4  0x000000000073f94f in __redisBlockForReply (c=0x7f55bc01a480) at build/third_party/hiredis/src/hiredis.c:1259
#5  redisvCommand (ap=0x7f55c2b5f2a8, format=<optimized out>, c=0x7f55bc01a480) at build/third_party/hiredis/src/hiredis.c:1269
#6  redisCommand (c=0x7f55bc01a480, format=<optimized out>) at build/third_party/hiredis/src/hiredis.c:1276
#7  0x00000000004e3d50 in RedisProcessorExec::SyncGetSeq (redis_ip=..., redis_port=<optimized out>, redis_password=..., source=..., node_type=..., module=..., instance_id=..., seqReply=...) at controller/src/analytics/redis_processor_vizd.cc:169
#8  0x0000000000521de7 in OpServerProxy::GetSeq (this=0x28ea4d0, source=..., node_type=..., module=..., instance_id=..., seqReply=...) at controller/src/analytics/OpServerProxy.cc:719
#9  0x00000000004bb107 in Collector::ReceiveSandeshCtrlMsg (this=0x28f3c80, state_machine=<optimized out>, session=<optimized out>, sandesh=<optimized out>) at controller/src/analytics/collector.cc:283
#10 0x0000000000612a50 in SandeshServerConnection::ProcessSandeshCtrlMessage (this=this@entry=0x2941ae0, msg=..., header=..., sandesh_name=..., header_offset=header_offset@entry=735) at tools/sandesh/library/cpp/sandesh_connection.cc:169
#11 0x0000000000611a6e in ssm::ServerInit::react (this=this@entry=0x7f55ac001f40, event=...) at tools/sandesh/library/cpp/sandesh_state_machine.cc:303
#12 0x00000000006121d5 in react<ssm::ServerInit, boost::statechart::event_base, void const*> (eventType=<synthetic pointer>, evt=..., stt=...) at /usr/include/boost/statechart/custom_reaction.hpp:42
#13 local_react_impl<boost::mpl::list3<boost::statechart::custom_reaction<ssm::EvSandeshCtrlMessageRecv>, boost::statechart::custom_reaction<ssm::EvSandeshMessageRecv>, boost::statechart::in_state_reaction<ssm::EvTcpDeleteSession, SandeshStateMachine, &SandeshStateMachine::DeleteTcpSession> >, boost::statechart::simple_state<ssm::ServerInit, SandeshStateMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0> > (eventType=0xaca7a0 <boost::statechart::detail::id_holder<ssm::EvSandeshCtrlMessageRecv>::idProvider_>, evt=..., stt=...) at /usr/include/boost/statechart/simple_state.hpp:816
#14 local_react<boost::mpl::list3<boost::statechart::custom_reaction<ssm::EvSandeshCtrlMessageRecv>, boost::statechart::custom_reaction<ssm::EvSandeshMessageRecv>, boost::statechart::in_state_reaction<ssm::EvTcpDeleteSession, SandeshStateMachine, &SandeshStateMachine::DeleteTcpSession> > > (eventType=0xaca7a0 <boost::statechart::detail::id_holder<ssm::EvSandeshCtrlMessageRecv>::idProvider_>, evt=..., this=0x7f55ac001f40) at /usr/include/boost/statechart/simple_state.hpp:851
#15 local_react_impl<boost::mpl::list4<boost::statechart::custom_reaction<ssm::EvTcpClose>, boost::statechart::custom_reaction<ssm::EvSandeshCtrlMessageRecv>, boost::statechart::custom_reaction<ssm::EvSandeshMessageRecv>, boost::statechart::in_state_reaction<ssm::EvTcpDeleteSession, SandeshStateMachine, &SandeshStateMachine::DeleteTcpSession> >, boost::statechart::simple_state<ssm::ServerInit, SandeshStateMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0> > (eventType=<optimized out>, evt=..., stt=...) at /usr/include/boost/statechart/simple_state.hpp:820
#16 local_react<boost::mpl::list4<boost::statechart::custom_reaction<ssm::EvTcpClose>, boost::statechart::custom_reaction<ssm::EvSandeshCtrlMessageRecv>, boost::statechart::custom_reaction<ssm::EvSandeshMessageRecv>, boost::statechart::in_state_reaction<ssm::EvTcpDeleteSession, SandeshStateMachine, &SandeshStateMachine::DeleteTcpSession> > > (eventType=<optimized out>, evt=..., this=<optimized out>) at /usr/include/boost/statechart/simple_state.hpp:851
#17 local_react_impl<boost::mpl::list<boost::statechart::transition<ssm::EvStop, ssm::Idle, SandeshStateMachine, &SandeshStateMachine::OnIdle>, boost::statechart::custom_reaction<ssm::EvTcpClose>, boost::statechart::custom_reaction<ssm::EvSandeshCtrlMessageRecv>, boost::statechart::custom_reaction<ssm::EvSandeshMessageRecv>, boost::statechart::in_state_reaction<ssm::EvTcpDeleteSession, SandeshStateMachine, &SandeshStateMachine::DeleteTcpSession> >, boost::statechart::simple_state<ssm::ServerInit, SandeshStateMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0> > (
    eventType=0xaca7a0 <boost::statechart::detail::id_holder<ssm::EvSandeshCtrlMessageRecv>::idProvider_>, evt=..., stt=...) at /usr/include/boost/statechart/simple_state.hpp:820
#18 local_react<boost::mpl::list<boost::statechart::transition<ssm::EvStop, ssm::Idle, SandeshStateMachine, &SandeshStateMachine::OnIdle>, boost::statechart::custom_reaction<ssm::EvTcpClose>, boost::statechart::custom_reaction<ssm::EvSandeshCtrlMessageRecv>, boost::statechart::custom_reaction<ssm::EvSandeshMessageRecv>, boost::statechart::in_state_reaction<ssm::EvTcpDeleteSession, SandeshStateMachine, &SandeshStateMachine::DeleteTcpSession> > > (
    eventType=0xaca7a0 <boost::statechart::detail::id_holder<ssm::EvSandeshCtrlMessageRecv>::idProvider_>, evt=..., this=0x7f55ac001f40) at /usr/include/boost/statechart/simple_state.hpp:851
#19 boost::statechart::simple_state<ssm::ServerInit, SandeshStateMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl (this=0x7f55ac001f40, evt=..., eventType=0xaca7a0 <boost::statechart::detail::id_holder<ssm::EvSandeshCtrlMessageRecv>::idProvider_>) at /usr/include/boost/statechart/simple_state.hpp:489
#20 0x000000000060fd6b in operator() (this=<synthetic pointer>) at /usr/include/boost/statechart/state_machine.hpp:87
#21 operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<SandeshStateMachine, ssm::Idle>::exception_event_handler> (this=0x2946c68, action=...) at /usr/include/boost/statechart/null_exception_translator.hpp:33
#22 boost::statechart::state_machine<SandeshStateMachine, ssm::Idle, std::allocator<void>, boost::statechart::null_exception_translator>::send_event (this=0x2946c10, evt=...) at /usr/include/boost/statechart/state_machine.hpp:889
#23 0x0000000000608325 in process_event (evt=..., this=0x2946c10) at /usr/include/boost/statechart/state_machine.hpp:275
---Type <return> to continue, or q <return> to quit---
#24 SandeshStateMachine::DequeueEvent (this=0x2946c10, ec=...) at tools/sandesh/library/cpp/sandesh_state_machine.cc:719
#25 0x000000000060f217 in operator() (a0=<error reading variable: access outside bounds of object referenced via synthetic pointer>, this=0x7f55c2b60130) at /usr/include/boost/function/function_template.hpp:767
#26 QueueTaskRunner<SandeshStateMachine::EventContainer, WorkQueue<SandeshStateMachine::EventContainer> >::RunQueue (this=0x7f55bc0018c0) at controller/src/base/queue_task.h:74
#27 0x00000000006ff880 in TaskImpl::execute (this=0x7f55c333f940) at controller/src/base/task.cc:232
#28 0x00007f55c98c1b3a in ?? () from /usr/lib/libtbb.so.2
#29 0x00007f55c98bd816 in ?? () from /usr/lib/libtbb.so.2
#30 0x00007f55c98bcf4b in ?? () from /usr/lib/libtbb.so.2
#31 0x00007f55c98b90ff in ?? () from /usr/lib/libtbb.so.2
#32 0x00007f55c98b92f9 in ?? () from /usr/lib/libtbb.so.2
#33 0x00007f55c9add182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#34 0x00007f55c83fa00d in qfcvt_r () from /lib/x86_64-linux-gnu/libc.so.6
#35 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f55c175c700 (LWP 3228)):
#0  0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f55c2f62700 (LWP 3217)):
#0  0x00007f55c83f43e9 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f55cb41d100 (LWP 2768)):
#0  0x00007f55c8335cc9 in __open_catalog () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x79745f65756c6176 in ?? ()
#2  0x554e5f434c5f6570 in ?? ()
#3  0x5d305b434952454d in ?? ()
#4  0x0000000000002929 in ?? ()
#5  0x7328203c20746e63 in ?? ()
#6  0x5f2820666f657a69 in ?? ()
#7  0x65756c61765f6c6e in ?? ()
#8  0x434c5f657079745f in ?? ()
#9  0x2f2029454d49545f in ?? ()
#10 0x20666f657a697320 in ?? ()
#11 0x6c61765f6c6e5f28 in ?? ()
#12 0x5f657079745f6575 in ?? ()
#13 0x5b454d49545f434c in ?? ()
#14 0x0000000029295d30 in ?? ()
#15 0x7328203c20746e63 in ?? ()
#16 0x5f2820666f657a69 in ?? ()
#17 0x65756c61765f6c6e in ?? ()
#18 0x434c5f657079745f in ?? ()
#19 0x4554414c4c4f435f in ?? ()
#20 0x657a6973202f2029 in ?? ()
#21 0x5f6c6e5f2820666f in ?? ()
#22 0x79745f65756c6176 in ?? ()
#23 0x4f435f434c5f6570 in ?? ()
#24 0x5d305b4554414c4c in ?? ()
#25 0x0000000000002929 in ?? ()
#26 0x7328203c20746e63 in ?? ()
#27 0x5f2820666f657a69 in ?? ()
#28 0x65756c61765f6c6e in ?? ()
#29 0x434c5f657079745f in ?? ()
#30 0x524154454e4f4d5f in ?? ()
#31 0x7a6973202f202959 in ?? ()
#32 0x6c6e5f2820666f65 in ?? ()
#33 0x745f65756c61765f in ?? ()
#34 0x4d5f434c5f657079 in ?? ()
#35 0x5b59524154454e4f in ?? ()
#36 0x0000000029295d30 in ?? ()
#37 0x7328203c20746e63 in ?? ()
#38 0x5f2820666f657a69 in ?? ()
#39 0x65756c61765f6c6e in ?? ()
---Type <return> to continue, or q <return> to quit---
#40 0x434c5f657079745f in ?? ()
#41 0x4547415353454d5f in ?? ()
#42 0x7a6973202f202953 in ?? ()
#43 0x6c6e5f2820666f65 in ?? ()
#44 0x745f65756c61765f in ?? ()
#45 0x4d5f434c5f657079 in ?? ()
#46 0x5b53454741535345 in ?? ()
#47 0x0000000029295d30 in ?? ()
#48 0x7328203c20746e63 in ?? ()
#49 0x5f2820666f657a69 in ?? ()
#50 0x65756c61765f6c6e in ?? ()
#51 0x434c5f657079745f in ?? ()
#52 0x202952455041505f in ?? ()
#53 0x666f657a6973202f in ?? ()
#54 0x61765f6c6e5f2820 in ?? ()
#55 0x657079745f65756c in ?? ()
#56 0x455041505f434c5f in ?? ()
#57 0x000029295d305b52 in ?? ()
#58 0x7328203c20746e63 in ?? ()
#59 0x5f2820666f657a69 in ?? ()
#60 0x65756c61765f6c6e in ?? ()
#61 0x434c5f657079745f in ?? ()
#62 0x2f2029454d414e5f in ?? ()
#63 0x20666f657a697320 in ?? ()
#64 0x6c61765f6c6e5f28 in ?? ()
#65 0x5f657079745f6575 in ?? ()
#66 0x5b454d414e5f434c in ?? ()
#67 0x0000000029295d30 in ?? ()
#68 0x7328203c20746e63 in ?? ()
#69 0x5f2820666f657a69 in ?? ()
#70 0x65756c61765f6c6e in ?? ()
#71 0x434c5f657079745f in ?? ()
#72 0x535345524444415f in ?? ()
#73 0x657a6973202f2029 in ?? ()
#74 0x5f6c6e5f2820666f in ?? ()
#75 0x79745f65756c6176 in ?? ()
#76 0x44415f434c5f6570 in ?? ()
#77 0x5d305b5353455244 in ?? ()
#78 0x0000000000002929 in ?? ()
#79 0x7328203c20746e63 in ?? ()
#80 0x5f2820666f657a69 in ?? ()
#81 0x65756c61765f6c6e in ?? ()
#82 0x434c5f657079745f in ?? ()
#83 0x4f4850454c45545f in ?? ()
#84 0x6973202f2029454e in ?? ()
#85 0x6e5f2820666f657a in ?? ()
#86 0x5f65756c61765f6c in ?? ()
#87 0x5f434c5f65707974 in ?? ()
#88 0x4e4f4850454c4554 in ?? ()
#89 0x000029295d305b45 in ?? ()
#90 0x7328203c20746e63 in ?? ()
#91 0x5f2820666f657a69 in ?? ()
#92 0x65756c61765f6c6e in ?? ()
#93 0x434c5f657079745f in ?? ()
#94 0x4552555341454d5f in ?? ()
#95 0x202f2029544e454d in ?? ()
#96 0x2820666f657a6973 in ?? ()
#97 0x756c61765f6c6e5f in ?? ()
#98 0x4c5f657079745f65 in ?? ()
#99 0x52555341454d5f43 in ?? ()
#100 0x5d305b544e454d45 in ?? ()
#101 0x0000000000002929 in ?? ()
---Type <return> to continue, or q <return> to quit---
#102 0x7328203c20746e63 in ?? ()
#103 0x5f2820666f657a69 in ?? ()
#104 0x65756c61765f6c6e in ?? ()
#105 0x434c5f657079745f in ?? ()
#106 0x4649544e4544495f in ?? ()
#107 0x294e4f4954414349 in ?? ()
#108 0x6f657a6973202f20 in ?? ()
#109 0x765f6c6e5f282066 in ?? ()
#110 0x7079745f65756c61 in ?? ()
#111 0x4544495f434c5f65 in ?? ()
#112 0x544143494649544e in ?? ()
#113 0x29295d305b4e4f49 in ?? ()
#114 0x0000000000000000 in ?? ()
(gdb)

Changed in juniperopenstack:
assignee:	nobody → Megh Bhatt (meghb)
importance:	Undecided → High

Revision history for this message

Megh Bhatt (meghb) wrote on 2015-05-20:

#2

Anoop can you please upload the other contrail-collector core and also the /var/log/contrail-collector.log*

Changed in juniperopenstack:
assignee:	Megh Bhatt (meghb) → Anoop Kumar Sahu (anoops)
milestone:	none → r2.30-fcs
status:	New → Incomplete

Revision history for this message

Megh Bhatt (meghb) wrote on 2015-05-20:

#3

Download full text (6.8 KiB)

Provided access to the box.

root@ubuntu-cn:~# gdb /var/tmp/vizd /var/crashes/core.contrail-collec.2768.ubuntu-cn.1431399756
GNU gdb (Ubuntu 7.7-0ubuntu3) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/tmp/vizd...done.

warning: core file may not match specified executable file.
[New LWP 2768]
[New LWP 3217]
[New LWP 3228]
[New LWP 3218]
[New LWP 3229]
[New LWP 3219]
[New LWP 3227]
[New LWP 3220]
[New LWP 3226]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-collector'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f55c8335cc9 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007f55c8335cc9 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f55c83390d8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f55c832eb86 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f55c832ec32 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000529aea in OpServerProxy::OpServerImpl::toConnectCallbackProcess (this=<optimized out>, c=<optimized out>, r=<optimized out>, privdata=<optimized out>)
at controller/src/analytics/OpServerProxy.cc:242
#5 0x00000000004d707a in operator() (a2=0x0, a1=0x2947050, a0=0x294b030, this=0x7fffe1727120) at /usr/include/boost/function/function_template.hpp:767
#6 RedisAsyncConnection::RAC_AsyncCmdCallback (c=0x294b030, r=0x2947050, privdata=0x0) at controller/src/analytics/redis_connection.cc:239
#7 0x00000000007423f3 in __redisRunCallback (cb=0x7fffe1727240, cb=0x7fffe1727240, reply=<optimized out>, ac=0x294b030) at build/third_party/hiredis/src/async.c:219
#8 redisProcessCallbacks (ac=0x294b030) at build/third_party/hiredis/src/async.c:417
#9 0x0000000000743819 in redisBoostClient::handle_read (this=0x294afe0, ec=...) at build/third_party/hiredis/hiredis-boostasio-adapter/boostasio.cpp:62
#10 0x0000000000743ee4 in call<boost::shared_ptr<redisBoostClient>, boost::system::error_code> (b1=<synthetic pointer>, u=..., this=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:156
#11 operator()<boost::shared_ptr<redisBoostClient> > (a1=..., u=..., this=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:171
#12 operator()<boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<const boost::system::error_code&, long unsigned int const&> > (a=<synthetic pointer>, f=...,
this=<optimized out>) at /usr/include/boost/bind/bind.hpp...

Provided access to the box.

root@ubuntu-cn:~# gdb /var/tmp/vizd /var/crashes/core.contrail-collec.2768.ubuntu-cn.1431399756
GNU gdb (Ubuntu 7.7-0ubuntu3) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/tmp/vizd...done.

warning: core file may not match specified executable file.
[New LWP 2768]
[New LWP 3217]
[New LWP 3228]
[New LWP 3218]
[New LWP 3229]
[New LWP 3219]
[New LWP 3227]
[New LWP 3220]
[New LWP 3226]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-collector'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f55c8335cc9 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007f55c8335cc9 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f55c83390d8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f55c832eb86 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f55c832ec32 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x0000000000529aea in OpServerProxy::OpServerImpl::toConnectCallbackProcess (this=<optimized out>, c=<optimized out>, r=<optimized out>, privdata=<optimized out>)
    at controller/src/analytics/OpServerProxy.cc:242
#5  0x00000000004d707a in operator() (a2=0x0, a1=0x2947050, a0=0x294b030, this=0x7fffe1727120) at /usr/include/boost/function/function_template.hpp:767
#6  RedisAsyncConnection::RAC_AsyncCmdCallback (c=0x294b030, r=0x2947050, privdata=0x0) at controller/src/analytics/redis_connection.cc:239
#7  0x00000000007423f3 in __redisRunCallback (cb=0x7fffe1727240, cb=0x7fffe1727240, reply=<optimized out>, ac=0x294b030) at build/third_party/hiredis/src/async.c:219
#8  redisProcessCallbacks (ac=0x294b030) at build/third_party/hiredis/src/async.c:417
#9  0x0000000000743819 in redisBoostClient::handle_read (this=0x294afe0, ec=...) at build/third_party/hiredis/hiredis-boostasio-adapter/boostasio.cpp:62
#10 0x0000000000743ee4 in call<boost::shared_ptr<redisBoostClient>, boost::system::error_code> (b1=<synthetic pointer>, u=..., this=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:156
#11 operator()<boost::shared_ptr<redisBoostClient> > (a1=..., u=..., this=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:171
#12 operator()<boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<const boost::system::error_code&, long unsigned int const&> > (a=<synthetic pointer>, f=...,
    this=<optimized out>) at /usr/include/boost/bind/bind.hpp:313
#13 operator()<boost::system::error_code, long unsigned int> (a2=<optimized out>, a1=..., this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:102
#14 operator() (this=<optimized out>) at /usr/include/boost/asio/detail/bind_handler.hpp:127
#15 asio_handler_invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long> > (function=...) at /usr/include/boost/asio/handler_invoke_hook.hpp:64
#16 invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> > > (context=..., function=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#17 boost::asio::detail::reactive_null_buffers_op<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> > >::do_complete (owner=<optimized out>, base=<optimized out>) at /usr/include/boost/asio/detail/reactive_null_buffers_op.hpp:75
#18 0x0000000000584a6f in complete (bytes_transferred=0, ec=..., owner=..., this=<optimized out>) at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
#19 boost::asio::detail::epoll_reactor::descriptor_state::do_complete (owner=0x28d91f0, base=0x28f37d0, ec=..., bytes_transferred=<optimized out>)
    at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:651
#20 0x000000000055bc27 in complete (bytes_transferred=5, ec=..., owner=..., this=0x28f37d0) at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
#21 do_run_one (ec=..., this_thread=..., lock=..., this=0x28d91f0) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:384
#22 boost::asio::detail::task_io_service::run (this=0x28d91f0, ec=...) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
#23 0x00000000005772b8 in run (this=0x28bbf50, ec=...) at /usr/include/boost/asio/impl/io_service.ipp:66
#24 EventManager::Run (this=0x28bbf50) at controller/src/io/event_manager.cc:32
#25 0x0000000000421062 in main (argc=<optimized out>, argv=<optimized out>) at controller/src/analytics/main.cc:407
(gdb) f 4
#4  0x0000000000529aea in OpServerProxy::OpServerImpl::toConnectCallbackProcess (this=<optimized out>, c=<optimized out>, r=<optimized out>, privdata=<optimized out>)
    at controller/src/analytics/OpServerProxy.cc:242
242	controller/src/analytics/OpServerProxy.cc: No such file or directory.
(gdb) p reply
$1 = <optimized out>
(gdb) p reply.type
$2 = <optimized out>
(gdb) p r
$3 = <optimized out>
(gdb)  6
Undefined command: "6".  Try "help".
(gdb) f 6
#6  RedisAsyncConnection::RAC_AsyncCmdCallback (c=0x294b030, r=0x2947050, privdata=0x0) at controller/src/analytics/redis_connection.cc:239
239	controller/src/analytics/redis_connection.cc: No such file or directory.
(gdb) p r
$4 = (void *) 0x2947050
(gdb) p (redisReply*)r
$5 = (redisReply *) 0x2947050
(gdb) p *$5
$6 = {type = 6, integer = 0, len = 46, str = 0x28f2dc0 "LOADING Redis is loading the dataset in memory", elements = 0, element = 0x0}
(gdb)

Looks like redis is initializing and hence returning the above error leading to assert. We should retry. Also need to check why is redis loading the database in memory

Megh Bhatt (meghb) on 2015-05-20

Changed in juniperopenstack:
status:	Incomplete → New
assignee:	Anoop Kumar Sahu (anoops) → Megh Bhatt (meghb)

OpenContrail Admin (ci-admin-f) on 2015-05-20

tags:

added: blocker

OpenContrail Admin (ci-admin-f) on 2015-05-20

information type:

Proprietary → Public

Revision history for this message

Sundaresan Rajangam (srajanga) wrote on 2015-05-21:

#4

By default, snapshot is enabled in redis.conf

On a loaded setup, upon redis-server restart, loading of dump.rdb in memory takes more time and this causes the issue [Logs below]
Should we consider disabling the snapshot in redis.conf?

[5141] 21 May 10:44:18.907 # Server started, Redis version 2.8.4
[5141] 21 May 10:44:18.907 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
[5141] 21 May 10:52:46.450 * DB loaded from disk: 507.543 seconds <<<<<<
.....
.....
[7348] 21 May 13:32:48.114 # Server started, Redis version 2.8.4
[7348] 21 May 13:32:48.114 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
[7348] 21 May 13:36:36.040 * DB loaded from disk: 227.926 seconds <<<<<<

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-05-28: R2.20

#5

Review in progress for https://review.opencontrail.org/11012
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-05-29: A change has been merged

#7

Reviewed: https://review.opencontrail.org/11012
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/532a93bdbae8929a5d35ed50d23287f0612a517d
Submitter: Zuul
Branch: R2.20

commit 532a93bdbae8929a5d35ed50d23287f0612a517d
Author: Megh Bhatt <email address hidden>
Date: Thu May 28 16:05:53 2015 -0700

Disable redis server persistence since that is not used by analytics
or webui and causes issues with slow redis bootup and subsequent
collector core. Refactor redis server setup code and create a
setup_redis_server_node task.
Closes-Bug: #1456853

Change-Id: Ida4e0374f3e8360e64cb3ac4ba6edadb57388523

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-05-30: master

#8

Review in progress for https://review.opencontrail.org/11075
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-06-01: A change has been merged

#10

Reviewed: https://review.opencontrail.org/11075
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/4d83d527fb9f0508769bf0ee1998c3504c524f6a
Submitter: Zuul
Branch: master

commit 4d83d527fb9f0508769bf0ee1998c3504c524f6a
Author: Megh Bhatt <email address hidden>
Date: Thu May 28 16:05:53 2015 -0700

Disable redis server persistence since that is not used by analytics
or webui and causes issues with slow redis bootup and subsequent
collector core. Refactor redis server setup code and create a
setup_redis_server_node task.
Closes-Bug: #1456853

Change-Id: Ida4e0374f3e8360e64cb3ac4ba6edadb57388523
(cherry picked from commit 532a93bdbae8929a5d35ed50d23287f0612a517d)

Changed in juniperopenstack:
status:	New → Fix Committed

Juniper Openstack

contrail-collector core tor qfx bms testing

Bug Description

Duplicates of this bug

Other bug subscribers

Remote bug watches

Affects		Status	Importance	Assigned to	Milestone
	Juniper Openstack	Fix Committed	High	Megh Bhatt	Juniper Openstack r3.0-fcs
	R2.20	Fix Committed	High	Megh Bhatt	Juniper Openstack r2.20-fcs