[R2.20 21] TOR Scale: TOR Agent crash @ contrail::sandesh::transport::TMemoryBuffer::initCommon

[chhandak]

Trgger
-------------
Restarted supervisor-vrouter

Backtrace
-----------------
(gdb) bt
#0 0x00007fb1f35218fd in malloc_consolidate (av=av@entry=0x7fb1b4000020) at malloc.c:4157
#1 0x00007fb1f3522df8 in _int_malloc (av=0x7fb1b4000020, bytes=2048) at malloc.c:3423
#2 0x00007fb1f35257b0 in __GI___libc_malloc (bytes=2048) at malloc.c:2891
#3 0x0000000001832bbc in contrail::sandesh::transport::TMemoryBuffer::initCommon (this=0x7fb1b48d9560, buf=0x0, size=2048,
    owner=true, wPos=0) at build/include/sandesh/transport/TBufferTransports.h:199
#4 0x0000000001845692 in contrail::sandesh::transport::TMemoryBuffer::TMemoryBuffer (this=0x7fb1b48d9560, sz=2048)
    at build/include/sandesh/transport/TBufferTransports.h:265
#5 0x00000000018738b5 in SandeshWriter::SendMsg (this=0x7fb1b8000c90, sandesh=0x7fb191060080, more=false)
    at tools/sandesh/library/cpp/sandesh_session.cc:79
#6 0x0000000001875e99 in SandeshSession::SendMsg (this=0x7fb1b8001ba0, sandesh=0x7fb191060080)
    at tools/sandesh/library/cpp/sandesh_session.cc:366
#7 0x000000000187e972 in boost::_mfi::mf1<bool, SandeshSession, Sandesh*>::operator() (this=0x7fb1e5ff6a78, p=0x7fb1b8001ba0,
    a1=0x7fb191060080) at /usr/include/boost/bind/mem_fn_template.hpp:165
#8 0x000000000187daac in boost::_bi::list2<boost::_bi::value<SandeshSession*>, boost::arg<1> >::operator()<bool, boost::_mfi::mf1<bool, SandeshSession, Sandesh*>, boost::_bi::list1<Sandesh*&> > (this=0x7fb1e5ff6a88, f=..., a=...)
    at /usr/include/boost/bind/bind.hpp:303
#9 0x000000000187cac6 in boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, SandeshSession, Sandesh*>, boost::_bi::list2<boost::_bi::value<SandeshSession*>, boost::arg<1> > >::operator()<Sandesh*> (this=0x7fb1e5ff6a78, a1=@0x7fb1e5ff69e0: 0x7fb191060080)
    at /usr/include/boost/bind/bind_template.hpp:32
#10 0x000000000187bda1 in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, SandeshSession, Sandesh*>, boost::_bi::list2<boost::_bi::value<SandeshSession*>, boost::arg<1> > >, bool, Sandesh*>::invoke (
    function_obj_ptr=..., a0=0x7fb191060080) at /usr/include/boost/function/function_template.hpp:132
#11 0x00000000018410e7 in boost::function1<bool, Sandesh*>::operator() (this=0x7fb1e5ff6a70, a0=0x7fb191060080)
    at /usr/include/boost/function/function_template.hpp:767
#12 0x000000000183ed16 in QueueTaskRunner<Sandesh*, WorkQueue<Sandesh*> >::RunQueue (this=0x7fb190667a80)
    at controller/src/base/queue_task.h:74
#13 0x000000000183df20 in QueueTaskRunner<Sandesh*, WorkQueue<Sandesh*> >::Run (this=0x7fb190667a80)
    at controller/src/base/queue_task.h:57
#14 0x000000000191fed8 in TaskImpl::execute (this=0x7fb1ecccaa40) at controller/src/base/task.cc:232
#15 0x00007fb1f40a8b3a in ?? () from /usr/lib/libtbb.so.2
#16 0x00007fb1f40a4816 in ?? () from /usr/lib/libtbb.so.2
#17 0x00007fb1f40a3f4b in ?? () from /usr/lib/libtbb.so.2
#18 0x00007fb1f40a00ff in ?? () from /usr/lib/libtbb.so.2
---Type <return> to continue, or q <return> to quit---
#19 0x00007fb1f40a02f9 in ?? () from /usr/lib/libtbb.so.2
#20 0x00007fb1f42c4182 in start_thread (arg=0x7fb1e5ff7700) at pthread_create.c:312
#21 0x00007fb1f359d47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Setup
----------
env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6],
    'cfgm': [host1, host2, host3],
    'openstack': [host1, host2, host3],
    'webui': [host2],
    'control': [host1, host3],
    'compute': [host4, host5, host6],
    'tsn': [host4, host5],
    'toragent': [host4, host5],
    'collector': [host1, host3],
    'database': [host1, host2, host3],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodei6', 'nodei7', 'nodei8', 'nodei9', 'nodei10', 'nodei19']
}

[Hari Prasad Killi (haripk)]

Memory corruption is seen here. Access to freed memory is seen in a few cases and have been fixed with the following commits.

https://github.com/Juniper/contrail-controller/commit/ba144cc5153a099b12d08e7e32a57ab433c4f697
https://github.com/Juniper/contrail-controller/commit/9056ce9349ba7f46fb0d7a8f88af68a9e537944b
https://github.com/Juniper/contrail-controller/commit/791439aafdf7f836b9579dcffb1299859826bb66

Also, valgrind reported issues are fixed with the following:
https://github.com/Juniper/contrail-controller/commit/a134faf39280de10a88342a430368b2ff76451be
https://bugs.launchpad.net/juniperopenstack/+bug/1460518

[Hari Prasad Killi (haripk)]

Following issues were found with valgrind:

https://bugs.launchpad.net/bugs/1461910
https://bugs.launchpad.net/bugs/1461062
https://bugs.launchpad.net/bugs/1461159

[Hari Prasad Killi (haripk)]

Will keep under observation - all the above mentioned fixes are merged.