Juniper Openstack

VM UVE showing empty data during sanity tests

Bug #1388221 reported by Vedamurthy Joshi on 2014-10-31

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Juniper Openstack	Fix Committed	High	Vedamurthy Joshi

Bug Description

2.0 Build 2427 Centos Havana multi-node multi-interface testbed

This has happened during multiple tests in the sanity run :
http://10.204.216.50/Docs/logs/2427_2014_10_31_19_46_43/test_report.html

Logs and cassandra logs are in /cs-shared/test_runs/nodec21/2014_10_31_19_46_43 on nodeb10.englab.juniper.net(use juniper id)

In the testcase test_svc_mirroring, the case because it didnt find the VM uve for vn2_vm2

2014-10-31 22:43:06,877 - WARNING - Failed to get VM vn2_vm2, ID 55b6e5b0-e75d-4114-809e-c3e9c7e1a517 info from Opserver^M
2014-10-31 22:43:08,879 - INFO - Verifying the vm in opserver^M
2014-10-31 22:43:08,879 - INFO - Verifying in collector 10.204.217.6 ...^M
Key Error^M
2014-10-31 22:43:08,886 - WARNING - Failed to get VM vn2_vm2, ID 55b6e5b0-e75d-4114-809e-c3e9c7e1a517 info from Opserver^M
2014-10-31 22:43:10,888 - INFO - Verifying the vm in opserver^M
2014-10-31 22:43:10,889 - INFO - Verifying in collector 10.204.217.6 ...^M
Key Error^M

Tags:

Revision history for this message

Megh Bhatt (meghb) wrote on 2014-11-05:

Download full text (39.1 KiB)

The issue was caused due to the Sandesh send queue getting stuck with ready_to_send_ being set to false due to a race condition. This issue was introduced as part of recent checkin https://github.com/Juniper/contrail-sandesh/commit/fc865df0e682e5b63c852fdf7b4710035000e518 which removed mutex from SandeshWriter and replaced it by using tbb::atomic to update ready_to_send_. However the mutex is needed since TcpSession::Send and TcpSession::WriteReady callback can happen in parallel and hence the mutex is needed to serialize the execution of both and updating of the ready_to_send_.

Following is the gdb analysis of the gcore of contrail-vrouter-agent:

[root@nodec60 ~]# gdb /var/tmp/contrail-vrouter-agent /var/tmp/core.6504
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /var/tmp/contrail-vrouter-agent...done.

Following is the gdb analysis of the gcore of contrail-vrouter-agent:

[root@nodec60 ~]# gdb /var/tmp/contrail-vrouter-agent /var/tmp/core.6504
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /var/tmp/contrail-vrouter-agent...done.

warning: core file may not match specified executable file.
[New Thread 7013]
[New Thread 7014]
[New Thread 7015]
[New Thread 7016]
[New Thread 11208]
[New Thread 6504]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b5/d86fbcf0ccb03331e6c7c73897b96845e0a4eb
Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libxml2.so.2
Reading symbols from /usr/lib/libthriftasio.so.0...done.
Loaded symbols for /usr/lib/libthriftasio.so.0
Reading symbols from /usr/lib/libthrift-0.8.0.so...done.
Loaded symbols for /usr/lib/libthrift-0.8.0.so
Reading symbols from /usr/lib/libboost_filesystem.so.1.48.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libboost_filesystem.so.1.48.0
Reading symbols from /usr/lib/libboost_program_options.so.1.48.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libboost_program_options.so.1.48.0
Reading symbols from /usr/lib/libboost_regex.so.1.48.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libboost_regex.so.1.48.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib/libboost_system.so.1.48.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libboost_system.so.1.48.0
Reading symbols from /usr/lib/liblog4cplus-1.1.so.7...done.
Loaded symbols for /usr/lib/liblog4cplus-1.1.so.7
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib/libtbb_debug.so.2...done.
Loaded symbols for /usr/lib/libtbb_debug.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Core was generated by `/usr/bin/contrail-vrouter-agent'.
#0  0x00002ac506fa03c9 in syscall () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install contrail-lib-2.0-2427.el6.x86_64 glibc-2.12-1.132.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6.x86_64 libcom_err-1.41.12-14.el6.x86_64 libgcc-4.4.7-4.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 libstdc++-4.4.7-3.el6.x86_64 libxml2-2.7.6-8.el6_3.4.x86_64 openssl-1.0.1e-16.el6_5.7.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00002ac506fa03c9 in syscall () from /lib64/libc.so.6
#1  0x00002ac5064d5620 in tbb::internal::futex_wait (futex=0x3970d2c, comparand=2) at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/include/tbb/machine/linux_common.h:67
#2  0x00002ac5064d597b in tbb::internal::binary_semaphore::P (this=0x3970d2c) at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/semaphore.h:212
#3  0x00002ac5064e8e95 in rml::internal::thread_monitor::commit_wait (this=0x3970d20, c=...) at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/rml/include/../server/thread_monitor.h:209
#4  0x00002ac5064e852b in tbb::internal::rml::private_worker::run (this=0x3970d00) at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/private_server.cpp:270
#5  0x00002ac5064e8362 in tbb::internal::rml::private_worker::thread_routine (arg=0x3970d00) at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/private_server.cpp:231
#6  0x00002ac5062af9d1 in start_thread () from /lib64/libpthread.so.0
#7  0x00002ac506fa3b6d in clone () from /lib64/libc.so.6
(gdb) info threads
  6 Thread 0x2ac508e5d1e0 (LWP 6504)  0x00002ac506fa4163 in epoll_wait () from /lib64/libc.so.6
  5 Thread 0x2ac522280700 (LWP 11208)  0x00002ac5062b35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4 Thread 0x2ac514801700 (LWP 7016)  0x00002ac506fa03c9 in syscall () from /lib64/libc.so.6
  3 Thread 0x2ac514400700 (LWP 7015)  0x00002ac506fa03c9 in syscall () from /lib64/libc.so.6
  2 Thread 0x2ac509667700 (LWP 7014)  0x00002ac506fa03c9 in syscall () from /lib64/libc.so.6
* 1 Thread 0x2ac509266700 (LWP 7013)  0x00002ac506fa03c9 in syscall () from /lib64/libc.so.6
(gdb) t 6
[Switching to thread 6 (Thread 0x2ac508e5d1e0 (LWP 6504))]#0  0x00002ac506fa4163 in epoll_wait () from /lib64/libc.so.6
(gdb) bt
#0  0x00002ac506fa4163 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000011227b2 in boost::asio::detail::epoll_reactor::run (this=0x397ae80, block=true, ops=...) at build/include/boost/asio/detail/impl/epoll_reactor.ipp:401
#2  0x000000000112319d in boost::asio::detail::task_io_service::do_run_one (this=0x39783b0, lock=..., this_thread=..., private_op_queue=..., ec=...)
    at build/include/boost/asio/detail/impl/task_io_service.ipp:350
#3  0x0000000001122e8e in boost::asio::detail::task_io_service::run (this=0x39783b0, ec=...) at build/include/boost/asio/detail/impl/task_io_service.ipp:146
#4  0x00000000011235b7 in boost::asio::io_service::run (this=0x3915160, ec=...) at build/include/boost/asio/impl/io_service.ipp:68
#5  0x0000000001882db4 in EventManager::RunWithExceptionHandling (this=0x3915160) at controller/src/io/event_manager.cc:51
#6  0x0000000001217b42 in main (argc=1, argv=0x7fff962c4808) at controller/src/vnsw/agent/contrail/main.cc:111
(gdb) p Sandesh::client_
$1 = (SandeshClient *) 0x2ac510069700
(gdb) p *Sandesh::client_
$2 = {<TcpServer> = {_vptr.TcpServer = 0x1ab4dd0, stats_ = {
      read_calls = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 1}}, <No data fields>}, <No data fields>},
      read_bytes = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 979}}, <No data fields>}, <No data fields>},
      read_errors = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 0}}, <No data fields>}, <No data fields>},
      write_calls = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 153}}, <No data fields>}, <No data fields>},
      write_bytes = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 860748}}, <No data fields>}, <No data fields>},
      write_errors = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 0}}, <No data fields>}, <No data fields>},
      write_blocked = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 16}}, <No data fields>}, <No data fields>},
      write_blocked_duration_usecs = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 83305}}, <No data fields>}, <No data fields>}}, evm_ = 0x3915160, mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {
        __data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}},
    cond_var_ = {<tbb::internal::no_copy> = {<tbb::internal::no_assign> = {<No data fields>}, <No data fields>}, my_cv = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0,
          __woken_seq = 0, __mutex = 0x0, __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align = 0}}, session_ref_ = std::set with 1 elements = {[0] = {px =
    0x2ac51c004320}}, session_map_ = std::multimap with 1 elements = {[{impl_ = {data_ = {base = {sa_family = 2, sa_data = "\037\226\300\250\n\006\000\000\000\000\000\000\000"}, v4 = {sin_family = 2,
              sin_port = 38431, sin_addr = {s_addr = 101361856}, sin_zero = "\000\000\000\000\000\000\000"}, v6 = {sin6_family = 2, sin6_port = 38431, sin6_flowinfo = 101361856, sin6_addr = {
                __in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}}] = 0x2ac51c004320}, so_accept_ = {
      _M_ptr = 0x0}, acceptor_ = {px = 0x0}, refcount_ = {<tbb::internal::atomic_impl_with_arithmetic<int, int, char>> = {<tbb::internal::atomic_impl<int>> = {rep = {
            value = 2}}, <No data fields>}, <No data fields>}, name_ = "", socket_open_failure_ = false}, <SandeshClientSM::Mgr> = {_vptr.Mgr = 0x1ab4e50}, static kSMTaskInstance = 0, static kSMTask =
    "sandesh::SandeshClientSM", static kSessionTaskInstance = -1, static kSessionWriterTask = "sandesh::SandeshClientSession", static kSessionReaderTask = "io::ReaderTask",
  static kSessionWaterMarkInfo = {<std::_Vector_base<boost::tuples::tuple<unsigned long, SandeshLevel::type, bool, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type>, std::allocator<boost::tuples::tuple<unsigned long, SandeshLevel::type, bool, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> > >> = {
      _M_impl = {<std::allocator<boost::tuples::tuple<unsigned long, SandeshLevel::type, bool, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> >> = {<__gnu_cxx::new_allocator<boost::tuples::tuple<unsigned long, SandeshLevel::type, bool, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> >> = {<No data fields>}, <No data fields>},
        _M_start = 0x38fc190, _M_finish = 0x38fc1d0, _M_end_of_storage = 0x38fc1d0}}, <No data fields>}, sm_task_instance_ = 0, sm_task_id_ = 17, session_task_instance_ = -1,
  session_writer_task_id_ = 18, session_reader_task_id_ = 6, primary_ = {impl_ = {data_ = {base = {sa_family = 2, sa_data = '\000' <repeats 13 times>}, v4 = {sin_family = 2, sin_port = 0, sin_addr = {
            s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, v6 = {sin6_family = 2, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
              __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}}, secondary_ = {impl_ = {data_ = {base = {sa_family = 2, sa_data = '\000' <repeats 13 times>},
        v4 = {sin_family = 2, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, v6 = {sin6_family = 2, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {
              __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}},
  csf_ = {<boost::function3<void, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned char, boost::function<void(std::vector<DSResponse, std::allocator<DSResponse> >)> >> = {<boost::function_base> = {vtable = 0x1a42e61, functor = {obj_ptr = 0x16d5eec, type = {type = 0x16d5eec, const_qualified = false, volatile_qualified = false}, func_ptr = 0x16d5eec
     <DiscoveryServiceClient::Subscribe(std::string, uint8_t, DiscoveryServiceClient::ServiceHandler)>, bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)(
    boost::detail::function::X *, int)) 0x16d5eec <DiscoveryServiceClient::Subscribe(std::string, uint8_t, DiscoveryServiceClient::ServiceHandler)>, obj_ptr = 0x2ac51004f1e0}, obj_ref = {
---Type <return> to continue, or q <return> to quit---
            obj_ptr = 0x16d5eec, is_const_qualified = false, is_volatile_qualified = false}, data = -20 '\354'}}, static args = <optimized out>, static arity = <optimized out>}, <No data fields>},
  sm_ = {px = 0x2ac5100638c0}, session_wm_info_ = std::vector of length 4, capacity 4 = {
    {<boost::tuples::cons<unsigned long, boost::tuples::cons<SandeshLevel::type, boost::tuples::cons<bool, boost::tuples::null_type> > >> = {head = 50000, tail = {head = SandeshLevel::SYS_ERR, tail = {
            head = true}}}, <No data fields>}, {<boost::tuples::cons<unsigned long, boost::tuples::cons<SandeshLevel::type, boost::tuples::cons<bool, boost::tuples::null_type> > >> = {head = 10000,
        tail = {head = SandeshLevel::SYS_DEBUG, tail = {head = true}}}, <No data fields>},
    {<boost::tuples::cons<unsigned long, boost::tuples::cons<SandeshLevel::type, boost::tuples::cons<bool, boost::tuples::null_type> > >> = {head = 10000, tail = {head = SandeshLevel::SYS_DEBUG,
          tail = {head = false}}}, <No data fields>}, {<boost::tuples::cons<unsigned long, boost::tuples::cons<SandeshLevel::type, boost::tuples::cons<bool, boost::tuples::null_type> > >> = {
        head = 2500, tail = {head = SandeshLevel::INVALID, tail = {head = false}}}, <No data fields>}}, static task_policy_set_ = true}
(gdb) p *Sandesh::client_->se
secondary_               session_map_             session_ref_             session_wm_info_         set_socket_open_failure
session                  session_reader_task_id_  session_task_instance_   session_writer_task_id_
(gdb) p *Sandesh::client_->se
secondary_               session_map_             session_ref_             session_wm_info_         set_socket_open_failure
session                  session_reader_task_id_  session_task_instance_   session_writer_task_id_
(gdb) p *Sandesh::client_->se
secondary_               session_map_             session_ref_             session_wm_info_         set_socket_open_failure
session                  session_reader_task_id_  session_task_instance_   session_writer_task_id_
(gdb) p *Sandesh::client_->session
Attempt to take contents of a non-pointer value.
(gdb) p *Sandesh::client_->sm_.ptx
There is no member or method named ptx.
(gdb) p *Sandesh::client_->sm_.px
$3 = {_vptr.SandeshClientSM = 0x1abd6b0, mgr_ = 0x2ac510069830, state_ = {<tbb::internal::atomic_impl<SandeshClientSM::State>> = {rep = {value = 4}}, <No data fields>}, mtex_ = {
    static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0,
        __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}},
  session_ = {<tbb::internal::atomic_impl_with_arithmetic<SandeshSession*, long, SandeshSession>> = {<tbb::internal::atomic_impl<SandeshSession*>> = {rep = {
          value = 47026066703136}}, <No data fields>}, <No data fields>}, server_ = {impl_ = {data_ = {base = {sa_family = 2, sa_data = "\037\226\300\250\n\006\000\000\000\000\000\000\000"}, v4 = {
          sin_family = 2, sin_port = 38431, sin_addr = {s_addr = 101361856}, sin_zero = "\000\000\000\000\000\000\000"}, v6 = {sin6_family = 2, sin6_port = 38431, sin6_flowinfo = 101361856,
          sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}}}
(gdb) p *Sandesh::client_->sm_.px->session_
No symbol "operator*" in current context.
(gdb) p Sandesh::client_->sm_.px->session_

$4 = {<tbb::internal::atomic_impl_with_arithmetic<SandeshSession*, long, SandeshSession>> = {<tbb::internal::atomic_impl<SandeshSession*>> = {rep = {
        value = 47026066703136}}, <No data fields>}, <No data fields>}
(gdb) p *(SandeshSession*)47026066703136
$5 = {<TcpSession> = {_vptr.TcpSession = 0x1ac20b0, static kDefaultBufferSize = 4096, static reader_task_id_ = 6, server_ = {px = 0x2ac510069700}, socket_ = {px = 0x2ac51c001130},
    read_on_connect_ = true, buffer_size_ = 4096, mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0,
          __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, established_ = true, closed_ = false,
    remote_ = {impl_ = {data_ = {base = {sa_family = 2, sa_data = "\037\226\300\250\n\006\000\000\000\000\000\000\000"}, v4 = {sin_family = 2, sin_port = 38431, sin_addr = {s_addr = 101361856},
            sin_zero = "\000\000\000\000\000\000\000"}, v6 = {sin6_family = 2, sin6_port = 38431, sin6_flowinfo = 101361856, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
                __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}}, direction_ = TcpSession::ACTIVE, buffer_queue_ = std::list = {[0] = {
        data_ = 0x2ac51c023480, size_ = 4096}}, obs_mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0,
          __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}},
    observer_ = {<boost::function2<void, TcpSession*, TcpSession::Event>> = {<boost::function_base> = {vtable = 0x1ac06a1, functor = {obj_ptr = 0x18d6a60, type = {type = 0x18d6a60,
              const_qualified = false, volatile_qualified = false}, func_ptr = 0x18d6a60 <SandeshClientSMImpl::OnSessionEvent(TcpSession*, TcpSession::Event)>, bound_memfunc_ptr = {memfunc_ptr = (
    void (boost::detail::function::X::*)(boost::detail::function::X *, int)) 0x18d6a60 <SandeshClientSMImpl::OnSessionEvent(TcpSession*, TcpSession::Event)>, obj_ptr = 0x2ac5100638c0}, obj_ref = {
              obj_ptr = 0x18d6a60, is_const_qualified = false, is_volatile_qualified = false}, data = 96 '`'}}, <std::binary_function<TcpSession*, TcpSession::Event, void>> = {<No data fields>},
        static args = <optimized out>, static arity = <optimized out>}, <No data fields>}, stats_ = {
      read_calls = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 1}}, <No data fields>}, <No data fields>},
      read_bytes = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 979}}, <No data fields>}, <No data fields>},
      read_errors = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 0}}, <No data fields>}, <No data fields>},
      write_calls = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 153}}, <No data fields>}, <No data fields>},
      write_bytes = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 860748}}, <No data fields>}, <No data fields>},
      write_errors = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 0}}, <No data fields>}, <No data fields>},
      write_blocked = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 16}}, <No data fields>}, <No data fields>},
      write_blocked_duration_usecs = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 83305}}, <No data fields>}, <No data fields>}}, writer_ = {px = 0x2ac51c0049c0},
    refcount_ = {<tbb::internal::atomic_impl_with_arithmetic<int, int, char>> = {<tbb::internal::atomic_impl<int>> = {rep = {value = 2}}, <No data fields>}, <No data fields>}, name_ =
    "192.168.10.2:57354::192.168.10.6:8086"}, static kSessionKeepaliveIdleTime = 45, static kSessionKeepaliveInterval = 3, static kSessionKeepaliveProbes = 5, instance_ = -1, writer_ = {
    px = 0x2ac51c001070}, reader_ = {px = 0x2ac51c004d70}, send_queue_ = {px = 0x2ac51c0040c0}, send_buffer_queue_ = {px = 0x0}, connection_ = 0x0, conn_mutex_ = {static is_rw_mutex = false,
    static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0,
          __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, send_mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {
      __data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}},
  keepalive_idle_time_ = 45, keepalive_interval_ = 3, keepalive_probes_ = 5, reader_task_id_ = 6, sstats_ = {_vptr.SandeshSessionStats = 0x1ab4f30, static ascii_fingerprint =
    0x1ad4e88 "25BF2AA50A62EB4BBFE2781F980284B4", static binary_fingerprint = 0x0, num_recv_msg = 0, num_recv_msg_fail = 0, num_recv_fail = 0, num_send_msg = 654, num_send_msg_fail = 0,
    num_send_buffer_fail = 0, num_wait_msgq_enqueue = 0, num_wait_msgq_dequeue = 0, num_write_ready_cb_error = 0, __isset = {num_recv_msg = false, num_recv_msg_fail = false, num_recv_fail = false,
      num_send_msg = false, num_send_msg_fail = false, num_send_buffer_fail = false, num_wait_msgq_enqueue = false, num_wait_msgq_dequeue = false, num_write_ready_cb_error = false}}}
(gdb) p ((SandeshSession*)47026066703136)->send_queue_.px

$6 = (WorkQueue<Sandesh*> *) 0x2ac51c0040c0
(gdb) p *((SandeshSession*)47026066703136)->send_queue_.px
$7 = {static kMaxSize = <optimized out>, static kMaxIterations = <optimized out>,
  queue_ = {<tbb::strict_ppl::internal::concurrent_queue_base_v3<Sandesh*>> = {<tbb::strict_ppl::internal::concurrent_queue_page_allocator> = {_vptr.concurrent_queue_page_allocator = 0x1ac21b0},
      my_rep = 0x2ac51c004580}, my_allocator = {<No data fields>}},
  count_ = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
          value = 18446}}, <No data fields>}, <No data fields>}, mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0,
        __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, running_ = false, taskId_ = 18,
  taskInstance_ = -1, callback_ = {<boost::function1<bool, Sandesh*>> = {<boost::function_base> = {vtable = 0x1ac27c1, functor = {obj_ptr = 0x18f3092, type = {type = 0x18f3092,
            const_qualified = false, volatile_qualified = false}, func_ptr = 0x18f3092 <SandeshSession::SendMsg(Sandesh*)>, bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)(
    boost::detail::function::X *, int)) 0x18f3092 <SandeshSession::SendMsg(Sandesh*)>, obj_ptr = 0x2ac51c004320}, obj_ref = {obj_ptr = 0x18f3092, is_const_qualified = false,
            is_volatile_qualified = false}, data = -110 '\222'}}, <std::unary_function<Sandesh*, bool>> = {<No data fields>}, static args = <optimized out>,
      static arity = <optimized out>}, <No data fields>}, on_entry_cb_ = {<boost::function0<bool>> = {<boost::function_base> = {vtable = 0x0, functor = {obj_ptr = 0x0, type = {type = 0x0,
            const_qualified = false, volatile_qualified = false}, func_ptr = 0, bound_memfunc_ptr = {memfunc_ptr = NULL, obj_ptr = 0x0}, obj_ref = {obj_ptr = 0x0, is_const_qualified = false,
            is_volatile_qualified = false}, data = 0 '\000'}}, static args = <optimized out>, static arity = <optimized out>}, <No data fields>},
  on_exit_cb_ = {<boost::function1<void, bool>> = {<boost::function_base> = {vtable = 0x0, functor = {obj_ptr = 0x0, type = {type = 0x0, const_qualified = false, volatile_qualified = false},
          func_ptr = 0, bound_memfunc_ptr = {memfunc_ptr = NULL, obj_ptr = 0x0}, obj_ref = {obj_ptr = 0x0, is_const_qualified = false, is_volatile_qualified = false},
          data = 0 '\000'}}, <std::unary_function<bool, void>> = {<No data fields>}, static args = <optimized out>, static arity = <optimized out>}, <No data fields>},
  start_runner_ = {<boost::function0<bool>> = {<boost::function_base> = {vtable = 0x1ac27e1, functor = {obj_ptr = 0x18f2c4a, type = {type = 0x18f2c4a, const_qualified = false,
            volatile_qualified = false}, func_ptr = 0x18f2c4a <SandeshSession::SessionSendReady()>, bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)(
    boost::detail::function::X *, int)) 0x18f2c4a <SandeshSession::SessionSendReady()>, obj_ptr = 0x2ac51c004320}, obj_ref = {obj_ptr = 0x18f2c4a, is_const_qualified = false,
            is_volatile_qualified = false}, data = 74 'J'}}, static args = <optimized out>, static arity = <optimized out>}, <No data fields>}, current_runner_ = 0x0, on_entry_defer_count_ = 0,
  disabled_ = false, deleted_ = false, enqueues_ = 19100, dequeues_ = 654, drops_ = 0, max_iterations_ = 32, size_ = 1024, bounded_ = false, shutdown_scheduled_ = false,
  delete_entries_on_shutdown_ = true, high_water_ = std::vector of length 2, capacity 2 = {{count_ = 50000, cb_ = {<boost::function1<void, unsigned long>> = {<boost::function_base> = {
            vtable = 0x1ac27b1, functor = {obj_ptr = 0x18b5402, type = {type = 0x18b5402, const_qualified = 3, volatile_qualified = false},
              func_ptr = 0x18b5402 <Sandesh::SetSendingLevel(size_t, SandeshLevel::type)>, bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)(boost::detail::function::X *,
    int)) 0x18b5402 <Sandesh::SetSendingLevel(size_t, SandeshLevel::type)>, this adjustment 3, obj_ptr = 0x10a0d32}, obj_ref = {obj_ptr = 0x18b5402, is_const_qualified = 3,
                is_volatile_qualified = false}, data = 2 '\002'}}, <std::unary_function<unsigned long, void>> = {<No data fields>}, static args = <optimized out>,
          static arity = <optimized out>}, <No data fields>}}, {count_ = 10000, cb_ = {<boost::function1<void, unsigned long>> = {<boost::function_base> = {vtable = 0x1ac27b1, functor = {
              obj_ptr = 0x18b5402, type = {type = 0x18b5402, const_qualified = 7, volatile_qualified = false}, func_ptr = 0x18b5402 <Sandesh::SetSendingLevel(size_t, SandeshLevel::type)>,
              bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)(boost::detail::function::X *,
    int)) 0x18b5402 <Sandesh::SetSendingLevel(size_t, SandeshLevel::type)>, this adjustment 7, obj_ptr = 0x10a0d32}, obj_ref = {obj_ptr = 0x18b5402, is_const_qualified = 7,
                is_volatile_qualified = false}, data = 2 '\002'}}, <std::unary_function<unsigned long, void>> = {<No data fields>}, static args = <optimized out>,
          static arity = <optimized out>}, <No data fields>}}}, low_water_ = std::vector of length 2, capacity 2 = {{count_ = 10000,
      cb_ = {<boost::function1<void, unsigned long>> = {<boost::function_base> = {vtable = 0x1ac27b1, functor = {obj_ptr = 0x18b5402, type = {type = 0x18b5402, const_qualified = 7,
                volatile_qualified = false}, func_ptr = 0x18b5402 <Sandesh::SetSendingLevel(size_t, SandeshLevel::type)>, bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)(
    boost::detail::function::X *, int)) 0x18b5402 <Sandesh::SetSendingLevel(size_t, SandeshLevel::type)>, this adjustment 7, obj_ptr = 0x10a0d32}, obj_ref = {obj_ptr = 0x18b5402,
                is_const_qualified = 7, is_volatile_qualified = false}, data = 2 '\002'}}, <std::unary_function<unsigned long, void>> = {<No data fields>}, static args = <optimized out>,
          static arity = <optimized out>}, <No data fields>}}, {count_ = 2500, cb_ = {<boost::function1<void, unsigned long>> = {<boost::function_base> = {vtable = 0x1ac27b1, functor = {
              obj_ptr = 0x18b5402, type = {type = 0x18b5402, const_qualified = 255, volatile_qualified = 255}, func_ptr = 0x18b5402 <Sandesh::SetSendingLevel(size_t, SandeshLevel::type)>,
              bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)(boost::detail::function::X *,
    int)) 0x18b5402 <Sandesh::SetSendingLevel(size_t, SandeshLevel::type)>, this adjustment 2147483647, obj_ptr = 0x10a0d32}, obj_ref = {obj_ptr = 0x18b5402, is_const_qualified = 255,
                is_volatile_qualified = 255}, data = 2 '\002'}}, <std::unary_function<unsigned long, void>> = {<No data fields>}, static args = <optimized out>,

static arity = <optimized out>}, <No data fields>}}}, hwater_mutex_ = {static is_rw_mutex = true, static is_recursive_mutex = false, static is_fair_mutex = false, static WRITER = 1,
---Type <return> to continue, or q <return> to quit---
    static WRITER_PENDING = 2, static READERS = -4, static ONE_READER = 4, static BUSY = -3, state = 0}, lwater_mutex_ = {static is_rw_mutex = true, static is_recursive_mutex = false,
    static is_fair_mutex = false, static WRITER = 1, static WRITER_PENDING = 2, static READERS = -4, static ONE_READER = 4, static BUSY = -3, state = 0}}
(gdb) p *((SandeshSession*)47026066703136)
$8 = {<TcpSession> = {_vptr.TcpSession = 0x1ac20b0, static kDefaultBufferSize = 4096, static reader_task_id_ = 6, server_ = {px = 0x2ac510069700}, socket_ = {px = 0x2ac51c001130},
    read_on_connect_ = true, buffer_size_ = 4096, mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0,
          __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, established_ = true, closed_ = false,
    remote_ = {impl_ = {data_ = {base = {sa_family = 2, sa_data = "\037\226\300\250\n\006\000\000\000\000\000\000\000"}, v4 = {sin_family = 2, sin_port = 38431, sin_addr = {s_addr = 101361856},
            sin_zero = "\000\000\000\000\000\000\000"}, v6 = {sin6_family = 2, sin6_port = 38431, sin6_flowinfo = 101361856, sin6_addr = {__in6_u = {__u6_addr8 = 0x2ac51c004320 "\260 \254\001",
                __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}}, direction_ = TcpSession::ACTIVE, buffer_queue_ = std::list = {[0] = {
        data_ = 0x2ac51c023480, size_ = 4096}}, obs_mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0,
          __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}},
    observer_ = {<boost::function2<void, TcpSession*, TcpSession::Event>> = {<boost::function_base> = {vtable = 0x1ac06a1, functor = {obj_ptr = 0x18d6a60, type = {type = 0x18d6a60,
              const_qualified = false, volatile_qualified = false}, func_ptr = 0x18d6a60 <SandeshClientSMImpl::OnSessionEvent(TcpSession*, TcpSession::Event)>, bound_memfunc_ptr = {memfunc_ptr = (
    void (boost::detail::function::X::*)(boost::detail::function::X *, int)) 0x18d6a60 <SandeshClientSMImpl::OnSessionEvent(TcpSession*, TcpSession::Event)>, obj_ptr = 0x2ac5100638c0}, obj_ref = {
              obj_ptr = 0x18d6a60, is_const_qualified = false, is_volatile_qualified = false}, data = 96 '`'}}, <std::binary_function<TcpSession*, TcpSession::Event, void>> = {<No data fields>},
        static args = <optimized out>, static arity = <optimized out>}, <No data fields>}, stats_ = {
      read_calls = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 1}}, <No data fields>}, <No data fields>},
      read_bytes = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 979}}, <No data fields>}, <No data fields>},
      read_errors = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 0}}, <No data fields>}, <No data fields>},
      write_calls = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 153}}, <No data fields>}, <No data fields>},
      write_bytes = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 860748}}, <No data fields>}, <No data fields>},
      write_errors = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 0}}, <No data fields>}, <No data fields>},
      write_blocked = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 16}}, <No data fields>}, <No data fields>},
      write_blocked_duration_usecs = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {rep = {
              value = 83305}}, <No data fields>}, <No data fields>}}, writer_ = {px = 0x2ac51c0049c0},
    refcount_ = {<tbb::internal::atomic_impl_with_arithmetic<int, int, char>> = {<tbb::internal::atomic_impl<int>> = {rep = {value = 2}}, <No data fields>}, <No data fields>}, name_ =
    "192.168.10.2:57354::192.168.10.6:8086"}, static kSessionKeepaliveIdleTime = 45, static kSessionKeepaliveInterval = 3, static kSessionKeepaliveProbes = 5, instance_ = -1, writer_ = {
    px = 0x2ac51c001070}, reader_ = {px = 0x2ac51c004d70}, send_queue_ = {px = 0x2ac51c0040c0}, send_buffer_queue_ = {px = 0x0}, connection_ = 0x0, conn_mutex_ = {static is_rw_mutex = false,
    static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0,
          __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, send_mutex_ = {static is_rw_mutex = false, static is_recursive_mutex = false, static is_fair_mutex = false, impl = {
      __data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}},
  keepalive_idle_time_ = 45, keepalive_interval_ = 3, keepalive_probes_ = 5, reader_task_id_ = 6, sstats_ = {_vptr.SandeshSessionStats = 0x1ab4f30,
    static ascii_fingerprint = 0x1ad4e88 "25BF2AA50A62EB4BBFE2781F980284B4", static binary_fingerprint = 0x0, num_recv_msg = 0, num_recv_msg_fail = 0, num_recv_fail = 0, num_send_msg = 654,
    num_send_msg_fail = 0, num_send_buffer_fail = 0, num_wait_msgq_enqueue = 0, num_wait_msgq_dequeue = 0, num_write_ready_cb_error = 0, __isset = {num_recv_msg = false, num_recv_msg_fail = false,
      num_recv_fail = false, num_send_msg = false, num_send_msg_fail = false, num_send_buffer_fail = false, num_wait_msgq_enqueue = false, num_wait_msgq_dequeue = false,
      num_write_ready_cb_error = false}}}
(gdb) p *((SandeshSession*)47026066703136)->writer_.px
$9 = {static kEncodeBufferSize = 2048, static kDefaultSendSize = 16384, static sandesh_open_ = "<sandesh length=\"0000000000\">", static sandesh_open_attr_length_ = "<sandesh length=\"",
  static sandesh_close_ = "</sandesh>", session_ = 0x2ac51c004320, ready_to_send_ = {<tbb::internal::atomic_impl<bool>> = {rep = {value = 0 '\000'}}, <No data fields>},
  send_buf_ = 0x2ac51c0054f0 "<sandesh length=\"0000001487\"><SandeshHeader><Namespace type=\"string\" identifier=\"1\"></Namespace><Timestamp type=\"i64\" identifier=\"2\">1414774337687479</Timestamp><Module type=\"string\" identifier=\"3\">co"..., send_buf_offset_ = 0}
(gdb) Write failed: Broken pipe
centos-build02@tmp$

Changed in juniperopenstack:
assignee:	Raj Reddy (rajreddy) → Megh Bhatt (meghb)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2014-11-06: A change has been merged

Reviewed: https://review.opencontrail.org/4322
Committed: http://github.org/Juniper/contrail-sandesh/commit/421e705ef085d6f2b225bf1d3b27beb5c423f8da
Submitter: Zuul
Branch: master

commit 421e705ef085d6f2b225bf1d3b27beb5c423f8da
Author: Megh Bhatt <email address hidden>
Date: Tue Nov 4 18:14:34 2014 -0800

TcpSession::Send() and TcpSession::WriteReady() can be called from
concurrent threads and hence just having an atomic bool to update
the send status is not enough to guarantee that the correct send
status is finally set. A recent change in SandeshWriter removed
the mutex acquired before called TcpSession::Send() and updating the
send status and used to update the send status in the
TcpSession::WriteReady() callback. This change reverts back to
using the mutex in SandeshWriter.
Closes-Bug: #1388221

Change-Id: Ibb560021d6a0d8963db6284f1d722a974cb45be2

Megh Bhatt (meghb) on 2014-11-10

Changed in juniperopenstack:
assignee:	Megh Bhatt (meghb) → Vedamurthy Joshi (vedujoshi)
status:	New → Fix Committed

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.