The issue was caused due to the Sandesh send queue getting stuck with ready_to_send_ being set to false due to a race condition. This issue was introduced as part of recent checkin https://github.com/Juniper/contrail-sandesh/commit/fc865df0e682e5b63c852fdf7b4710035000e518 which removed mutex from SandeshWriter and replaced it by using tbb::atomic to update ready_to_send_. However the mutex is needed since TcpSession::Send and TcpSession::WriteReady callback can happen in parallel and hence the mutex is needed to serialize the execution of both and updating of the ready_to_send_. Following is the gdb analysis of the gcore of contrail-vrouter-agent: [root@nodec60 ~]# gdb /var/tmp/contrail-vrouter-agent /var/tmp/core.6504 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /var/tmp/contrail-vrouter-agent...done. warning: core file may not match specified executable file. [New Thread 7013] [New Thread 7014] [New Thread 7015] [New Thread 7016] [New Thread 11208] [New Thread 6504] Missing separate debuginfo for Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b5/d86fbcf0ccb03331e6c7c73897b96845e0a4eb Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libxml2.so.2 Reading symbols from /usr/lib/libthriftasio.so.0...done. Loaded symbols for /usr/lib/libthriftasio.so.0 Reading symbols from /usr/lib/libthrift-0.8.0.so...done. Loaded symbols for /usr/lib/libthrift-0.8.0.so Reading symbols from /usr/lib/libboost_filesystem.so.1.48.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libboost_filesystem.so.1.48.0 Reading symbols from /usr/lib/libboost_program_options.so.1.48.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libboost_program_options.so.1.48.0 Reading symbols from /usr/lib/libboost_regex.so.1.48.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libboost_regex.so.1.48.0 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /usr/lib/libboost_system.so.1.48.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libboost_system.so.1.48.0 Reading symbols from /usr/lib/liblog4cplus-1.1.so.7...done. Loaded symbols for /usr/lib/liblog4cplus-1.1.so.7 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /usr/lib/libtbb_debug.so.2...done. Loaded symbols for /usr/lib/libtbb_debug.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl.so.10 Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcrypto.so.10 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_dns.so.2 Core was generated by `/usr/bin/contrail-vrouter-agent'. #0 0x00002ac506fa03c9 in syscall () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install contrail-lib-2.0-2427.el6.x86_64 glibc-2.12-1.132.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6.x86_64 libcom_err-1.41.12-14.el6.x86_64 libgcc-4.4.7-4.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 libstdc++-4.4.7-3.el6.x86_64 libxml2-2.7.6-8.el6_3.4.x86_64 openssl-1.0.1e-16.el6_5.7.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00002ac506fa03c9 in syscall () from /lib64/libc.so.6 #1 0x00002ac5064d5620 in tbb::internal::futex_wait (futex=0x3970d2c, comparand=2) at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/include/tbb/machine/linux_common.h:67 #2 0x00002ac5064d597b in tbb::internal::binary_semaphore::P (this=0x3970d2c) at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/semaphore.h:212 #3 0x00002ac5064e8e95 in rml::internal::thread_monitor::commit_wait (this=0x3970d20, c=...) at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/rml/include/../server/thread_monitor.h:209 #4 0x00002ac5064e852b in tbb::internal::rml::private_worker::run (this=0x3970d00) at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/private_server.cpp:270 #5 0x00002ac5064e8362 in tbb::internal::rml::private_worker::thread_routine (arg=0x3970d00) at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/private_server.cpp:231 #6 0x00002ac5062af9d1 in start_thread () from /lib64/libpthread.so.0 #7 0x00002ac506fa3b6d in clone () from /lib64/libc.so.6 (gdb) info threads 6 Thread 0x2ac508e5d1e0 (LWP 6504) 0x00002ac506fa4163 in epoll_wait () from /lib64/libc.so.6 5 Thread 0x2ac522280700 (LWP 11208) 0x00002ac5062b35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 4 Thread 0x2ac514801700 (LWP 7016) 0x00002ac506fa03c9 in syscall () from /lib64/libc.so.6 3 Thread 0x2ac514400700 (LWP 7015) 0x00002ac506fa03c9 in syscall () from /lib64/libc.so.6 2 Thread 0x2ac509667700 (LWP 7014) 0x00002ac506fa03c9 in syscall () from /lib64/libc.so.6 * 1 Thread 0x2ac509266700 (LWP 7013) 0x00002ac506fa03c9 in syscall () from /lib64/libc.so.6 (gdb) t 6 [Switching to thread 6 (Thread 0x2ac508e5d1e0 (LWP 6504))]#0 0x00002ac506fa4163 in epoll_wait () from /lib64/libc.so.6 (gdb) bt #0 0x00002ac506fa4163 in epoll_wait () from /lib64/libc.so.6 #1 0x00000000011227b2 in boost::asio::detail::epoll_reactor::run (this=0x397ae80, block=true, ops=...) at build/include/boost/asio/detail/impl/epoll_reactor.ipp:401 #2 0x000000000112319d in boost::asio::detail::task_io_service::do_run_one (this=0x39783b0, lock=..., this_thread=..., private_op_queue=..., ec=...) at build/include/boost/asio/detail/impl/task_io_service.ipp:350 #3 0x0000000001122e8e in boost::asio::detail::task_io_service::run (this=0x39783b0, ec=...) at build/include/boost/asio/detail/impl/task_io_service.ipp:146 #4 0x00000000011235b7 in boost::asio::io_service::run (this=0x3915160, ec=...) at build/include/boost/asio/impl/io_service.ipp:68 #5 0x0000000001882db4 in EventManager::RunWithExceptionHandling (this=0x3915160) at controller/src/io/event_manager.cc:51 #6 0x0000000001217b42 in main (argc=1, argv=0x7fff962c4808) at controller/src/vnsw/agent/contrail/main.cc:111 (gdb) p Sandesh::client_ $1 = (SandeshClient *) 0x2ac510069700 (gdb) p *Sandesh::client_ $2 = { = {_vptr.TcpServer = 0x1ab4dd0, stats_ = { read_calls = {> = {> = {rep = { value = 1}}, }, }, read_bytes = {> = {> = {rep = { value = 979}}, }, }, read_errors = {> = {> = {rep = { value = 0}}, }, }, write_calls = {> = {> = {rep = { value = 153}}, }, }, write_bytes = {> = {> = {rep = { value = 860748}}, }, }, write_errors = {> = {> = {rep = { value = 0}}, }, }, write_blocked = {> = {> = {rep = { value = 16}}, }, }, write_blocked_duration_usecs = {> = {> = {rep = { value = 83305}}, }, }}, evm_ = 0x3915160, mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = { __data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , __align = 0}}, cond_var_ = { = { = {}, }, my_cv = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' , __align = 0}}, session_ref_ = std::set with 1 elements = {[0] = {px = 0x2ac51c004320}}, session_map_ = std::multimap with 1 elements = {[{impl_ = {data_ = {base = {sa_family = 2, sa_data = "\037\226\300\250\n\006\000\000\000\000\000\000\000"}, v4 = {sin_family = 2, sin_port = 38431, sin_addr = {s_addr = 101361856}, sin_zero = "\000\000\000\000\000\000\000"}, v6 = {sin6_family = 2, sin6_port = 38431, sin6_flowinfo = 101361856, sin6_addr = { __in6_u = {__u6_addr8 = '\000' , __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}}] = 0x2ac51c004320}, so_accept_ = { _M_ptr = 0x0}, acceptor_ = {px = 0x0}, refcount_ = {> = {> = {rep = { value = 2}}, }, }, name_ = "", socket_open_failure_ = false}, = {_vptr.Mgr = 0x1ab4e50}, static kSMTaskInstance = 0, static kSMTask = "sandesh::SandeshClientSM", static kSessionTaskInstance = -1, static kSessionWriterTask = "sandesh::SandeshClientSession", static kSessionReaderTask = "io::ReaderTask", static kSessionWaterMarkInfo = {, std::allocator > >> = { _M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, _M_start = 0x38fc190, _M_finish = 0x38fc1d0, _M_end_of_storage = 0x38fc1d0}}, }, sm_task_instance_ = 0, sm_task_id_ = 17, session_task_instance_ = -1, session_writer_task_id_ = 18, session_reader_task_id_ = 6, primary_ = {impl_ = {data_ = {base = {sa_family = 2, sa_data = '\000' }, v4 = {sin_family = 2, sin_port = 0, sin_addr = { s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, v6 = {sin6_family = 2, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' , __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}}, secondary_ = {impl_ = {data_ = {base = {sa_family = 2, sa_data = '\000' }, v4 = {sin_family = 2, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, v6 = {sin6_family = 2, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = { __u6_addr8 = '\000' , __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}}, csf_ = {, std::allocator >, unsigned char, boost::function >)> >> = { = {vtable = 0x1a42e61, functor = {obj_ptr = 0x16d5eec, type = {type = 0x16d5eec, const_qualified = false, volatile_qualified = false}, func_ptr = 0x16d5eec , bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)( boost::detail::function::X *, int)) 0x16d5eec , obj_ptr = 0x2ac51004f1e0}, obj_ref = { ---Type to continue, or q to quit--- obj_ptr = 0x16d5eec, is_const_qualified = false, is_volatile_qualified = false}, data = -20 '\354'}}, static args = , static arity = }, }, sm_ = {px = 0x2ac5100638c0}, session_wm_info_ = std::vector of length 4, capacity 4 = { { > >> = {head = 50000, tail = {head = SandeshLevel::SYS_ERR, tail = { head = true}}}, }, { > >> = {head = 10000, tail = {head = SandeshLevel::SYS_DEBUG, tail = {head = true}}}, }, { > >> = {head = 10000, tail = {head = SandeshLevel::SYS_DEBUG, tail = {head = false}}}, }, { > >> = { head = 2500, tail = {head = SandeshLevel::INVALID, tail = {head = false}}}, }}, static task_policy_set_ = true} (gdb) p *Sandesh::client_->se secondary_ session_map_ session_ref_ session_wm_info_ set_socket_open_failure session session_reader_task_id_ session_task_instance_ session_writer_task_id_ (gdb) p *Sandesh::client_->se secondary_ session_map_ session_ref_ session_wm_info_ set_socket_open_failure session session_reader_task_id_ session_task_instance_ session_writer_task_id_ (gdb) p *Sandesh::client_->se secondary_ session_map_ session_ref_ session_wm_info_ set_socket_open_failure session session_reader_task_id_ session_task_instance_ session_writer_task_id_ (gdb) p *Sandesh::client_->session Attempt to take contents of a non-pointer value. (gdb) p *Sandesh::client_->sm_.ptx There is no member or method named ptx. (gdb) p *Sandesh::client_->sm_.px $3 = {_vptr.SandeshClientSM = 0x1abd6b0, mgr_ = 0x2ac510069830, state_ = {> = {rep = {value = 4}}, }, mtex_ = { static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , __align = 0}}, session_ = {> = {> = {rep = { value = 47026066703136}}, }, }, server_ = {impl_ = {data_ = {base = {sa_family = 2, sa_data = "\037\226\300\250\n\006\000\000\000\000\000\000\000"}, v4 = { sin_family = 2, sin_port = 38431, sin_addr = {s_addr = 101361856}, sin_zero = "\000\000\000\000\000\000\000"}, v6 = {sin6_family = 2, sin6_port = 38431, sin6_flowinfo = 101361856, sin6_addr = {__in6_u = {__u6_addr8 = '\000' , __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}}} (gdb) p *Sandesh::client_->sm_.px->session_ No symbol "operator*" in current context. (gdb) p Sandesh::client_->sm_.px->session_ $4 = {> = {> = {rep = { value = 47026066703136}}, }, } (gdb) p *(SandeshSession*)47026066703136 $5 = { = {_vptr.TcpSession = 0x1ac20b0, static kDefaultBufferSize = 4096, static reader_task_id_ = 6, server_ = {px = 0x2ac510069700}, socket_ = {px = 0x2ac51c001130}, read_on_connect_ = true, buffer_size_ = 4096, mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , __align = 0}}, established_ = true, closed_ = false, remote_ = {impl_ = {data_ = {base = {sa_family = 2, sa_data = "\037\226\300\250\n\006\000\000\000\000\000\000\000"}, v4 = {sin_family = 2, sin_port = 38431, sin_addr = {s_addr = 101361856}, sin_zero = "\000\000\000\000\000\000\000"}, v6 = {sin6_family = 2, sin6_port = 38431, sin6_flowinfo = 101361856, sin6_addr = {__in6_u = {__u6_addr8 = '\000' , __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}}, direction_ = TcpSession::ACTIVE, buffer_queue_ = std::list = {[0] = { data_ = 0x2ac51c023480, size_ = 4096}}, obs_mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , __align = 0}}, observer_ = {> = { = {vtable = 0x1ac06a1, functor = {obj_ptr = 0x18d6a60, type = {type = 0x18d6a60, const_qualified = false, volatile_qualified = false}, func_ptr = 0x18d6a60 , bound_memfunc_ptr = {memfunc_ptr = ( void (boost::detail::function::X::*)(boost::detail::function::X *, int)) 0x18d6a60 , obj_ptr = 0x2ac5100638c0}, obj_ref = { obj_ptr = 0x18d6a60, is_const_qualified = false, is_volatile_qualified = false}, data = 96 '`'}}, > = {}, static args = , static arity = }, }, stats_ = { read_calls = {> = {> = {rep = { value = 1}}, }, }, read_bytes = {> = {> = {rep = { value = 979}}, }, }, read_errors = {> = {> = {rep = { value = 0}}, }, }, write_calls = {> = {> = {rep = { value = 153}}, }, }, write_bytes = {> = {> = {rep = { value = 860748}}, }, }, write_errors = {> = {> = {rep = { value = 0}}, }, }, write_blocked = {> = {> = {rep = { value = 16}}, }, }, write_blocked_duration_usecs = {> = {> = {rep = { value = 83305}}, }, }}, writer_ = {px = 0x2ac51c0049c0}, refcount_ = {> = {> = {rep = {value = 2}}, }, }, name_ = "192.168.10.2:57354::192.168.10.6:8086"}, static kSessionKeepaliveIdleTime = 45, static kSessionKeepaliveInterval = 3, static kSessionKeepaliveProbes = 5, instance_ = -1, writer_ = { px = 0x2ac51c001070}, reader_ = {px = 0x2ac51c004d70}, send_queue_ = {px = 0x2ac51c0040c0}, send_buffer_queue_ = {px = 0x0}, connection_ = 0x0, conn_mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , __align = 0}}, send_mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = { __data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , __align = 0}}, keepalive_idle_time_ = 45, keepalive_interval_ = 3, keepalive_probes_ = 5, reader_task_id_ = 6, sstats_ = {_vptr.SandeshSessionStats = 0x1ab4f30, static ascii_fingerprint = 0x1ad4e88 "25BF2AA50A62EB4BBFE2781F980284B4", static binary_fingerprint = 0x0, num_recv_msg = 0, num_recv_msg_fail = 0, num_recv_fail = 0, num_send_msg = 654, num_send_msg_fail = 0, num_send_buffer_fail = 0, num_wait_msgq_enqueue = 0, num_wait_msgq_dequeue = 0, num_write_ready_cb_error = 0, __isset = {num_recv_msg = false, num_recv_msg_fail = false, num_recv_fail = false, num_send_msg = false, num_send_msg_fail = false, num_send_buffer_fail = false, num_wait_msgq_enqueue = false, num_wait_msgq_dequeue = false, num_write_ready_cb_error = false}}} (gdb) p ((SandeshSession*)47026066703136)->send_queue_.px $6 = (WorkQueue *) 0x2ac51c0040c0 (gdb) p *((SandeshSession*)47026066703136)->send_queue_.px $7 = {static kMaxSize = , static kMaxIterations = , queue_ = {> = { = {_vptr.concurrent_queue_page_allocator = 0x1ac21b0}, my_rep = 0x2ac51c004580}, my_allocator = {}}, count_ = {> = {> = {rep = { value = 18446}}, }, }, mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , __align = 0}}, running_ = false, taskId_ = 18, taskInstance_ = -1, callback_ = {> = { = {vtable = 0x1ac27c1, functor = {obj_ptr = 0x18f3092, type = {type = 0x18f3092, const_qualified = false, volatile_qualified = false}, func_ptr = 0x18f3092 , bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)( boost::detail::function::X *, int)) 0x18f3092 , obj_ptr = 0x2ac51c004320}, obj_ref = {obj_ptr = 0x18f3092, is_const_qualified = false, is_volatile_qualified = false}, data = -110 '\222'}}, > = {}, static args = , static arity = }, }, on_entry_cb_ = {> = { = {vtable = 0x0, functor = {obj_ptr = 0x0, type = {type = 0x0, const_qualified = false, volatile_qualified = false}, func_ptr = 0, bound_memfunc_ptr = {memfunc_ptr = NULL, obj_ptr = 0x0}, obj_ref = {obj_ptr = 0x0, is_const_qualified = false, is_volatile_qualified = false}, data = 0 '\000'}}, static args = , static arity = }, }, on_exit_cb_ = {> = { = {vtable = 0x0, functor = {obj_ptr = 0x0, type = {type = 0x0, const_qualified = false, volatile_qualified = false}, func_ptr = 0, bound_memfunc_ptr = {memfunc_ptr = NULL, obj_ptr = 0x0}, obj_ref = {obj_ptr = 0x0, is_const_qualified = false, is_volatile_qualified = false}, data = 0 '\000'}}, > = {}, static args = , static arity = }, }, start_runner_ = {> = { = {vtable = 0x1ac27e1, functor = {obj_ptr = 0x18f2c4a, type = {type = 0x18f2c4a, const_qualified = false, volatile_qualified = false}, func_ptr = 0x18f2c4a , bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)( boost::detail::function::X *, int)) 0x18f2c4a , obj_ptr = 0x2ac51c004320}, obj_ref = {obj_ptr = 0x18f2c4a, is_const_qualified = false, is_volatile_qualified = false}, data = 74 'J'}}, static args = , static arity = }, }, current_runner_ = 0x0, on_entry_defer_count_ = 0, disabled_ = false, deleted_ = false, enqueues_ = 19100, dequeues_ = 654, drops_ = 0, max_iterations_ = 32, size_ = 1024, bounded_ = false, shutdown_scheduled_ = false, delete_entries_on_shutdown_ = true, high_water_ = std::vector of length 2, capacity 2 = {{count_ = 50000, cb_ = {> = { = { vtable = 0x1ac27b1, functor = {obj_ptr = 0x18b5402, type = {type = 0x18b5402, const_qualified = 3, volatile_qualified = false}, func_ptr = 0x18b5402 , bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)(boost::detail::function::X *, int)) 0x18b5402 , this adjustment 3, obj_ptr = 0x10a0d32}, obj_ref = {obj_ptr = 0x18b5402, is_const_qualified = 3, is_volatile_qualified = false}, data = 2 '\002'}}, > = {}, static args = , static arity = }, }}, {count_ = 10000, cb_ = {> = { = {vtable = 0x1ac27b1, functor = { obj_ptr = 0x18b5402, type = {type = 0x18b5402, const_qualified = 7, volatile_qualified = false}, func_ptr = 0x18b5402 , bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)(boost::detail::function::X *, int)) 0x18b5402 , this adjustment 7, obj_ptr = 0x10a0d32}, obj_ref = {obj_ptr = 0x18b5402, is_const_qualified = 7, is_volatile_qualified = false}, data = 2 '\002'}}, > = {}, static args = , static arity = }, }}}, low_water_ = std::vector of length 2, capacity 2 = {{count_ = 10000, cb_ = {> = { = {vtable = 0x1ac27b1, functor = {obj_ptr = 0x18b5402, type = {type = 0x18b5402, const_qualified = 7, volatile_qualified = false}, func_ptr = 0x18b5402 , bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)( boost::detail::function::X *, int)) 0x18b5402 , this adjustment 7, obj_ptr = 0x10a0d32}, obj_ref = {obj_ptr = 0x18b5402, is_const_qualified = 7, is_volatile_qualified = false}, data = 2 '\002'}}, > = {}, static args = , static arity = }, }}, {count_ = 2500, cb_ = {> = { = {vtable = 0x1ac27b1, functor = { obj_ptr = 0x18b5402, type = {type = 0x18b5402, const_qualified = 255, volatile_qualified = 255}, func_ptr = 0x18b5402 , bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)(boost::detail::function::X *, int)) 0x18b5402 , this adjustment 2147483647, obj_ptr = 0x10a0d32}, obj_ref = {obj_ptr = 0x18b5402, is_const_qualified = 255, is_volatile_qualified = 255}, data = 2 '\002'}}, > = {}, static args = , static arity = }, }}}, hwater_mutex_ = {static is_rw_mutex = true, static is_recursive_mutex = false, static is_fair_mutex = false, static WRITER = 1, ---Type to continue, or q to quit--- static WRITER_PENDING = 2, static READERS = -4, static ONE_READER = 4, static BUSY = -3, state = 0}, lwater_mutex_ = {static is_rw_mutex = true, static is_recursive_mutex = false, static is_fair_mutex = false, static WRITER = 1, static WRITER_PENDING = 2, static READERS = -4, static ONE_READER = 4, static BUSY = -3, state = 0}} (gdb) p *((SandeshSession*)47026066703136) $8 = { = {_vptr.TcpSession = 0x1ac20b0, static kDefaultBufferSize = 4096, static reader_task_id_ = 6, server_ = {px = 0x2ac510069700}, socket_ = {px = 0x2ac51c001130}, read_on_connect_ = true, buffer_size_ = 4096, mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , __align = 0}}, established_ = true, closed_ = false, remote_ = {impl_ = {data_ = {base = {sa_family = 2, sa_data = "\037\226\300\250\n\006\000\000\000\000\000\000\000"}, v4 = {sin_family = 2, sin_port = 38431, sin_addr = {s_addr = 101361856}, sin_zero = "\000\000\000\000\000\000\000"}, v6 = {sin6_family = 2, sin6_port = 38431, sin6_flowinfo = 101361856, sin6_addr = {__in6_u = {__u6_addr8 = 0x2ac51c004320 "\260 \254\001", __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}}, direction_ = TcpSession::ACTIVE, buffer_queue_ = std::list = {[0] = { data_ = 0x2ac51c023480, size_ = 4096}}, obs_mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , __align = 0}}, observer_ = {> = { = {vtable = 0x1ac06a1, functor = {obj_ptr = 0x18d6a60, type = {type = 0x18d6a60, const_qualified = false, volatile_qualified = false}, func_ptr = 0x18d6a60 , bound_memfunc_ptr = {memfunc_ptr = ( void (boost::detail::function::X::*)(boost::detail::function::X *, int)) 0x18d6a60 , obj_ptr = 0x2ac5100638c0}, obj_ref = { obj_ptr = 0x18d6a60, is_const_qualified = false, is_volatile_qualified = false}, data = 96 '`'}}, > = {}, static args = , static arity = }, }, stats_ = { read_calls = {> = {> = {rep = { value = 1}}, }, }, read_bytes = {> = {> = {rep = { value = 979}}, }, }, read_errors = {> = {> = {rep = { value = 0}}, }, }, write_calls = {> = {> = {rep = { value = 153}}, }, }, write_bytes = {> = {> = {rep = { value = 860748}}, }, }, write_errors = {> = {> = {rep = { value = 0}}, }, }, write_blocked = {> = {> = {rep = { value = 16}}, }, }, write_blocked_duration_usecs = {> = {> = {rep = { value = 83305}}, }, }}, writer_ = {px = 0x2ac51c0049c0}, refcount_ = {> = {> = {rep = {value = 2}}, }, }, name_ = "192.168.10.2:57354::192.168.10.6:8086"}, static kSessionKeepaliveIdleTime = 45, static kSessionKeepaliveInterval = 3, static kSessionKeepaliveProbes = 5, instance_ = -1, writer_ = { px = 0x2ac51c001070}, reader_ = {px = 0x2ac51c004d70}, send_queue_ = {px = 0x2ac51c0040c0}, send_buffer_queue_ = {px = 0x0}, connection_ = 0x0, conn_mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , __align = 0}}, send_mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = { __data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , __align = 0}}, keepalive_idle_time_ = 45, keepalive_interval_ = 3, keepalive_probes_ = 5, reader_task_id_ = 6, sstats_ = {_vptr.SandeshSessionStats = 0x1ab4f30, static ascii_fingerprint = 0x1ad4e88 "25BF2AA50A62EB4BBFE2781F980284B4", static binary_fingerprint = 0x0, num_recv_msg = 0, num_recv_msg_fail = 0, num_recv_fail = 0, num_send_msg = 654, num_send_msg_fail = 0, num_send_buffer_fail = 0, num_wait_msgq_enqueue = 0, num_wait_msgq_dequeue = 0, num_write_ready_cb_error = 0, __isset = {num_recv_msg = false, num_recv_msg_fail = false, num_recv_fail = false, num_send_msg = false, num_send_msg_fail = false, num_send_buffer_fail = false, num_wait_msgq_enqueue = false, num_wait_msgq_dequeue = false, num_write_ready_cb_error = false}}} (gdb) p *((SandeshSession*)47026066703136)->writer_.px $9 = {static kEncodeBufferSize = 2048, static kDefaultSendSize = 16384, static sandesh_open_ = "", static sandesh_open_attr_length_ = "> = {rep = {value = 0 '\000'}}, }, send_buf_ = 0x2ac51c0054f0 "1414774337687479co"..., send_buf_offset_ = 0} (gdb) Write failed: Broken pipe centos-build02@tmp$