Vrouter agent corruption

Bug #1475617 reported by Hari Prasad Killi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
New
High
Hari Prasad Killi
Trunk
New
High
Hari Prasad Killi

Bug Description

#0 0x00007f48e22e2cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f48e22e60d8 in __GI_abort () at abort.c:89
#2 0x00007f48e231f394 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7f48e242db28 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007f48e232b66e in malloc_printerr (ptr=<optimized out>, str=0x7f48e242dc10 "double free or corruption (!prev)", action=1) at malloc.c:4996
#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5 0x0000000000fa469b in deallocate (this=0x317ee18, __p=<optimized out>) at /usr/include/c++/4.8/ext/new_allocator.h:110
#6 _M_deallocate (this=0x317ee18, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/4.8/bits/stl_vector.h:174
#7 std::vector<TaskGroup*, std::allocator<TaskGroup*> >::_M_fill_insert (this=this@entry=0x317ee18, __position=..., __n=<optimized out>,
    __x=@0x7f48db566290: 0x0) at /usr/include/c++/4.8/bits/vector.tcc:519
#8 0x0000000000fa04a2 in insert (__x=@0x7f48db566290: 0x0, __n=<optimized out>, __position=..., this=0x317ee18)
    at /usr/include/c++/4.8/bits/stl_vector.h:1024
#9 resize (__x=0x0, __new_size=37, this=0x317ee18) at /usr/include/c++/4.8/bits/stl_vector.h:707
#10 TaskScheduler::GetTaskGroup (this=0x317edd0, task_id=21) at controller/src/base/task.cc:354
#11 0x0000000000fa2b5b in TaskScheduler::EnqueueUnLocked (this=0x317edd0, t=0x7f48cc068d20) at controller/src/base/task.cc:433
#12 0x0000000000fa2ce1 in TaskScheduler::Enqueue (this=0x317edd0, t=0x7f48cc068d20) at controller/src/base/task.cc:425
#13 0x0000000000f0edfb in WorkQueue<SandeshClientSMImpl::EventContainer>::MayBeStartRunner (this=0x7f48d40754e8) at controller/src/base/queue_task.h:313
#14 0x0000000000f0f7fa in EnqueueInternal (entry=..., this=0x7f48d40754e8) at controller/src/base/queue_task.h:486
#15 WorkQueue<SandeshClientSMImpl::EventContainer>::Enqueue (this=this@entry=0x7f48d40754e8, entry=...) at controller/src/base/queue_task.h:280
#16 0x0000000000f10dc4 in SandeshClientSMImpl::Enqueue<scm::EvSandeshSend> (this=0x7f48d4075380, event=...)
    at tools/sandesh/library/cpp/sandesh_client_sm.cc:917
#17 0x0000000000f07f0a in SandeshClientSMImpl::SendSandeshUVE (this=<optimized out>, snh=<optimized out>)
    at tools/sandesh/library/cpp/sandesh_client_sm.cc:778
#18 0x0000000000ef09d0 in SendSandeshUVE (snh_uve=0x7f48cc607010, this=<optimized out>) at tools/sandesh/library/cpp/sandesh_client.h:69
#19 SandeshUVE::Dispatch (this=0x7f48cc607010, sconn=<optimized out>) at tools/sandesh/library/cpp/sandesh.cc:647
#20 0x0000000000f568b9 in SandeshModuleClientTrace::Send (data=..., seq=seq@entry=false, seqno=seqno@entry=0, ctx=..., more=more@entry=false)
    at build/production/tools/sandesh/library/common/sandesh_uve_types.cpp:5807
#21 0x0000000000f06494 in SandeshClientSMImpl::StatisticsTimerExpired (this=0x7f48d4075380) at tools/sandesh/library/cpp/sandesh_client_sm.cc:1028
#22 0x0000000000fa9529 in operator() (this=<optimized out>) at /usr/include/boost/function/function_template.hpp:767
#23 Timer::TimerTask::Run (this=0x31a88b0) at controller/src/base/timer.cc:42
#24 0x0000000000fa2f20 in TaskImpl::execute (this=0x7f48dbb3b740) at controller/src/base/task.cc:238

#3 0x00007f44a378a66e in malloc_printerr (ptr=<optimized out>, str=0x7f44a388cc10 "double free or corruption (!prev)", action=1) at malloc.c:4996
#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5 0x0000000000fa469b in deallocate (this=0x336de18, __p=<optimized out>) at /usr/include/c++/4.8/ext/new_allocator.h:110
#6 _M_deallocate (this=0x336de18, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/4.8/bits/stl_vector.h:174
#7 std::vector<TaskGroup*, std::allocator<TaskGroup*> >::_M_fill_insert (this=this@entry=0x336de18, __position=..., __n=<optimized out>,
    __x=@0x7f449c9c5340: 0x0) at /usr/include/c++/4.8/bits/vector.tcc:519
#8 0x0000000000fa04a2 in insert (__x=@0x7f449c9c5340: 0x0, __n=<optimized out>, __position=..., this=0x336de18)
    at /usr/include/c++/4.8/bits/stl_vector.h:1024
#9 resize (__x=0x0, __new_size=36, this=0x336de18) at /usr/include/c++/4.8/bits/stl_vector.h:707
#10 TaskScheduler::GetTaskGroup (this=0x336ddd0, task_id=20) at controller/src/base/task.cc:354
#11 0x0000000000fa2b5b in TaskScheduler::EnqueueUnLocked (this=0x336ddd0, t=0x7f44907b2060) at controller/src/base/task.cc:433
#12 0x0000000000fa2ce1 in TaskScheduler::Enqueue (this=0x336ddd0, t=0x7f44907b2060) at controller/src/base/task.cc:425
#13 0x0000000000b933eb in WorkQueue<IoContext*>::MayBeStartRunner (this=0x7f44900548f0) at controller/src/base/queue_task.h:313
#14 0x0000000000b8e787 in EnqueueInternal (entry=0x7f4490924d50, this=0x7f44900548f0) at controller/src/base/queue_task.h:486
#15 Enqueue (entry=<optimized out>, this=0x7f44900548f0) at controller/src/base/queue_task.h:280
#16 KSyncSock::SendAsync (this=<optimized out>, entry=entry@entry=0x7f4490214380, msg_len=msg_len@entry=225, msg=msg@entry=0x7f449035f9d0 "",
    event=event@entry=KSyncEntry::ADD_ACK) at controller/src/ksync/ksync_sock.cc:772
#17 0x0000000000b8a366 in KSyncNetlinkEntry::Add (this=0x7f4490214380) at controller/src/ksync/ksync_netlink.cc:46
#18 0x0000000000c2a3fd in KSyncSM_Add (entry=0x7f4490214380, obj=0x7f4490006080) at controller/src/ksync/ksync_object.cc:646
#19 KSyncSM_Init (obj=0x7f4490006080, entry=0x7f4490214380, event=<optimized out>) at controller/src/ksync/ksync_object.cc:706
#20 0x0000000000c2abc5 in KSyncObject::NotifyEvent (this=0x7f4490006080, entry=0x7f4490214380, event=KSyncEntry::ADD_CHANGE_REQ)
    at controller/src/ksync/ksync_object.cc:1118
#21 0x0000000000c2bd45 in KSyncObject::Create (this=this@entry=0x7f4490006080, key=key@entry=0x7f449c9c55b0) at controller/src/ksync/ksync_object.cc:143
#22 0x0000000000ae56c2 in FlowEntry::UpdateKSync (this=this@entry=0x7f449083fcf0, table=table@entry=0x7f449003b870)
    at controller/src/vnsw/agent/pkt/flow_table.cc:852
#23 0x0000000000ae5c28 in FlowTable::ResyncAFlow (this=0x7f449003b870, fe=0x7f449083fcf0) at controller/src/vnsw/agent/pkt/flow_table.cc:3144
#24 0x0000000000ae5cce in FlowTable::Add (this=0x7f449003b870, flow=flow@entry=0x7f4490994010, rflow=rflow@entry=0x7f449083fcf0)
    at controller/src/vnsw/agent/pkt/flow_table.cc:946
#25 0x0000000000afc014 in PktFlowInfo::Add (this=this@entry=0x7f449c9c5940, pkt=0x3390870, in=in@entry=0x7f449c9c58c0, out=out@entry=0x7f449c9c5900)
    at controller/src/vnsw/agent/pkt/pkt_flow_info.cc:1409
#26 0x0000000000b06f8b in FlowHandler::Run (this=0x7f4490373250) at controller/src/vnsw/agent/pkt/flow_handler.cc:80
#27 0x0000000000b03c56 in Proto::ProcessProto (this=<optimized out>, msg_info=...) at controller/src/vnsw/agent/pkt/proto.cc:44

#3 0x00007ffa3f72d66e in malloc_printerr (ptr=<optimized out>, str=0x7ffa3f82fc10 "double free or corruption (!prev)", action=1) at malloc.c:4996
#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5 0x0000000000fa469b in deallocate (this=0x2d8ce18, __p=<optimized out>) at /usr/include/c++/4.8/ext/new_allocator.h:110
#6 _M_deallocate (this=0x2d8ce18, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/4.8/bits/stl_vector.h:174
#7 std::vector<TaskGroup*, std::allocator<TaskGroup*> >::_M_fill_insert (this=this@entry=0x2d8ce18, __position=..., __n=<optimized out>,
    __x=@0x7ffa38d69ce0: 0x0) at /usr/include/c++/4.8/bits/vector.tcc:519
#8 0x0000000000fa04a2 in insert (__x=@0x7ffa38d69ce0: 0x0, __n=<optimized out>, __position=..., this=0x2d8ce18)
    at /usr/include/c++/4.8/bits/stl_vector.h:1024
#9 resize (__x=0x0, __new_size=38, this=0x2d8ce18) at /usr/include/c++/4.8/bits/stl_vector.h:707
#10 TaskScheduler::GetTaskGroup (this=0x2d8cdd0, task_id=22) at controller/src/base/task.cc:354
#11 0x0000000000fa2d86 in TaskScheduler::OnTaskExit (this=0x2d8cdd0, t=0x7ffa340088c0) at controller/src/base/task.cc:528
#12 0x0000000000fa2e83 in TaskImpl::~TaskImpl (this=0x7ffa38f43940, __in_chrg=<optimized out>) at controller/src/base/task.cc:270

#3 0x00007f6519c3d66e in malloc_printerr (ptr=<optimized out>, str=0x7f6519d3fc10 "double free or corruption (!prev)", action=1) at malloc.c:4996
#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5 0x0000000000fa469b in deallocate (this=0x2cdfe18, __p=<optimized out>) at /usr/include/c++/4.8/ext/new_allocator.h:110
#6 _M_deallocate (this=0x2cdfe18, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/4.8/bits/stl_vector.h:174
#7 std::vector<TaskGroup*, std::allocator<TaskGroup*> >::_M_fill_insert (this=this@entry=0x2cdfe18, __position=..., __n=<optimized out>,
    __x=@0x7f6512e78480: 0x0) at /usr/include/c++/4.8/bits/vector.tcc:519
#8 0x0000000000fa04a2 in insert (__x=@0x7f6512e78480: 0x0, __n=<optimized out>, __position=..., this=0x2cdfe18)
    at /usr/include/c++/4.8/bits/stl_vector.h:1024
#9 resize (__x=0x0, __new_size=37, this=0x2cdfe18) at /usr/include/c++/4.8/bits/stl_vector.h:707
#10 TaskScheduler::GetTaskGroup (this=0x2cdfdd0, task_id=21) at controller/src/base/task.cc:354
#11 0x0000000000fa2b5b in TaskScheduler::EnqueueUnLocked (this=0x2cdfdd0, t=0x7f650c0021b0) at controller/src/base/task.cc:433
#12 0x0000000000fa2ce1 in TaskScheduler::Enqueue (this=0x2cdfdd0, t=0x7f650c0021b0) at controller/src/base/task.cc:425
#13 0x0000000000f0edfb in WorkQueue<SandeshClientSMImpl::EventContainer>::MayBeStartRunner (this=0x7f650c0754f8) at controller/src/base/queue_task.h:313
#14 0x0000000000f0f7fa in EnqueueInternal (entry=..., this=0x7f650c0754f8) at controller/src/base/queue_task.h:486
#15 WorkQueue<SandeshClientSMImpl::EventContainer>::Enqueue (this=this@entry=0x7f650c0754f8, entry=...) at controller/src/base/queue_task.h:280
#16 0x0000000000f10dc4 in SandeshClientSMImpl::Enqueue<scm::EvSandeshSend> (this=0x7f650c075390, event=...)
    at tools/sandesh/library/cpp/sandesh_client_sm.cc:917
#17 0x0000000000f07f4a in SandeshClientSMImpl::SendSandesh (this=<optimized out>, snh=<optimized out>) at tools/sandesh/library/cpp/sandesh_client_sm.cc:783
#18 0x0000000000ef0686 in Sandesh::SendEnqueue (this=0x7f650c1b11a0) at tools/sandesh/library/cpp/sandesh.cc:591
#19 0x0000000000ad75ea in Send (flowdata=..., level=SandeshLevel::SYS_DEBUG, category=...) at build/production/sandesh/common/flow_types.h:603
#20 FlowTable::DispatchFlowMsg (this=<optimized out>, level=SandeshLevel::SYS_DEBUG, flow=...) at controller/src/vnsw/agent/pkt/flow_table.cc:3653
#21 0x0000000000ada733 in FlowTable::FlowExport (this=this@entry=0x7f650c039500, flow=flow@entry=0x7f6504086f40, diff_bytes=diff_bytes@entry=144,
    diff_pkts=diff_pkts@entry=2) at controller/src/vnsw/agent/pkt/flow_table.cc:3639
#22 0x0000000000b0808e in FlowStatsCollector::Run (this=0x7f650c005c10) at controller/src/vnsw/agent/vrouter/flow_stats/flow_stats_collector.cc:318
#23 0x0000000000b0880a in StatsCollector::TimerExpiry (this=<optimized out>) at controller/src/vnsw/agent/uve/stats_collector.h:67
#24 0x0000000000fa9529 in operator() (this=<optimized out>) at /usr/include/boost/function/function_template.hpp:767
#25 Timer::TimerTask::Run (this=0x2d01b70) at controller/src/base/timer.cc:42
#26 0x0000000000fa2f20 in TaskImpl::execute (this=0x7f6513473b40) at controller/src/base/task.cc:238

Tags: quench vrouter
Revision history for this message
Hari Prasad Killi (haripk) wrote :

cores at /data/bugs/1475617 on mayamruga.englab.juniper.net

Changed in juniperopenstack:
assignee: nobody → Hari Prasad Killi (haripk)
tags: added: quench
Revision history for this message
Hari Prasad Killi (haripk) wrote :

Another thread was exiting and these double free issues were due to that. An exception is thrown, resulting in agent exit from event manager (same as bug 1476613).

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.