HandlerSocket leaks thread objects

[Laurynas Biveinis]

PS 5.6 trunk on Valgrind (with dlclose() commented out in the core server to get usable stacktraces):

main.handler_socket [ pass ] 23754
worker[1] Valgrind report from /home/laurynas/obj-percona-5.6-valgrind/mysql-test/var/log/mysqld.1.err after tests:
 main.handler_socket
--------------------------------------------------------------------------
HEAP SUMMARY:
    in use at exit: 71,111,989 bytes in 1,134 blocks
  total heap usage: 12,358 allocs, 11,224 frees, 107,333,476 bytes allocated

95 bytes in 1 blocks are still reachable in loss record 180 of 345
   at 0x4C2DB8F: malloc (vg_replace_malloc.c:299)
   by 0x401CD89: strdup (strdup.c:42)
   by 0x40089D3: expand_dynamic_string_token (dl-load.c:376)
   by 0x40089D3: _dl_map_object (dl-load.c:2409)
   by 0x40153A6: dl_open_worker (dl-open.c:237)
   by 0x4010393: _dl_catch_error (dl-error.c:187)
   by 0x4014BD8: _dl_open (dl-open.c:660)
   by 0x5696F08: dlopen_doit (dlopen.c:66)
   by 0x4010393: _dl_catch_error (dl-error.c:187)
   by 0x5697570: _dlerror_run (dlerror.c:163)
   by 0x5696FA0: dlopen@@GLIBC_2.2.5 (dlopen.c:87)
   by 0x826496: plugin_dl_add(st_mysql_lex_string const*, int) (sql_plugin.cc:494)
   by 0x8277DC: plugin_add(st_mem_root*, st_mysql_lex_string const*, st_mysql_lex_string const*, int*, char**, int) (sql_plugin.cc:856)
   by 0x82A8BC: mysql_install_plugin(THD*, st_mysql_lex_string const*, st_mysql_lex_string const*) (sql_plugin.cc:1896)
   by 0x81B4E6: mysql_execute_command(THD*) (sql_parse.cc:5422)
   by 0x81F444: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:6972)
   by 0x81092E: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1441)

2,560 bytes in 16 blocks are indirectly lost in loss record 275 of 345
   at 0x4C2E0EF: operator new(unsigned long) (vg_replace_malloc.c:333)
   by 0x148734C2: dena::database::create_context(bool) volatile (database.cpp:200)
   by 0x148819D0: dena::hstcpsvr_worker::hstcpsvr_worker(dena::hstcpsvr_worker_arg const&) (hstcpsvr_worker.cpp:287)
   by 0x14884D47: dena::hstcpsvr_worker_i::create(dena::hstcpsvr_worker_arg const&) (hstcpsvr_worker.cpp:943)
   by 0x1488C741: dena::worker_throbj::worker_throbj(dena::hstcpsvr_worker_arg const&) (hstcpsvr.cpp:30)
   by 0x1488CD1A: dena::thread<dena::worker_throbj>::thread<dena::hstcpsvr_worker_arg>(dena::hstcpsvr_worker_arg const&, unsigned long) (thread.hpp:22)
   by 0x1488C142: dena::hstcpsvr::start_listen[abi:cxx11]() (hstcpsvr.cpp:122)
   by 0x1487FB9A: daemon_handlersocket_init(void*) (handlersocket.cpp:84)
   by 0x828808: plugin_initialize(st_plugin_int*) (sql_plugin.cc:1159)
   by 0x82A9A3: mysql_install_plugin(THD*, st_mysql_lex_string const*, st_mysql_lex_string const*) (sql_plugin.cc:1912)
   by 0x81B4E6: mysql_execute_command(THD*) (sql_parse.cc:5422)
   by 0x81F444: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:6972)
   by 0x81092E: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1441)
   by 0x80F7A4: do_command(THD*) (sql_parse.cc:1053)
   by 0x7D27F1: do_handle_one_connection(THD*) (sql_connect.cc:1541)
   by 0x7D2289: handle_one_connection (sql_connect.cc:1444)

2,944 bytes in 16 blocks are indirectly lost in loss record 276 of 345
   at 0x4C2E0EF: operator new(unsigned long) (vg_replace_malloc.c:333)
   by 0x14884D35: dena::hstcpsvr_worker_i::create(dena::hstcpsvr_worker_arg const&) (hstcpsvr_worker.cpp:943)
   by 0x1488C741: dena::worker_throbj::worker_throbj(dena::hstcpsvr_worker_arg const&) (hstcpsvr.cpp:30)
   by 0x1488CD1A: dena::thread<dena::worker_throbj>::thread<dena::hstcpsvr_worker_arg>(dena::hstcpsvr_worker_arg const&, unsigned long) (thread.hpp:22)
   by 0x1488C142: dena::hstcpsvr::start_listen[abi:cxx11]() (hstcpsvr.cpp:122)
   by 0x1487FB9A: daemon_handlersocket_init(void*) (handlersocket.cpp:84)
   by 0x828808: plugin_initialize(st_plugin_int*) (sql_plugin.cc:1159)
   by 0x82A9A3: mysql_install_plugin(THD*, st_mysql_lex_string const*, st_mysql_lex_string const*) (sql_plugin.cc:1912)
   by 0x81B4E6: mysql_execute_command(THD*) (sql_parse.cc:5422)
   by 0x81F444: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:6972)
   by 0x81092E: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1441)
   by 0x80F7A4: do_command(THD*) (sql_parse.cc:1053)
   by 0x7D27F1: do_handle_one_connection(THD*) (sql_connect.cc:1541)
   by 0x7D2289: handle_one_connection (sql_connect.cc:1444)
   by 0xB958CF: pfs_spawn_thread (pfs.cc:1860)
   by 0x58A16F9: start_thread (pthread_create.c:333)

12,800 bytes in 16 blocks are indirectly lost in loss record 289 of 345
   at 0x4C2E0EF: operator new(unsigned long) (vg_replace_malloc.c:333)
   by 0x14873CFD: dena::dbcontext::init_thread(void const*, int volatile&) (database.cpp:333)
   by 0x14881CBE: dena::(anonymous namespace)::thr_init::thr_init(std::auto_ptr<dena::dbcontext_i> const&, int volatile&) (hstcpsvr_worker.cpp:311)
   by 0x14881D2D: dena::hstcpsvr_worker::run() (hstcpsvr_worker.cpp:324)
   by 0x1488C7AA: dena::worker_throbj::operator()() (hstcpsvr.cpp:32)
   by 0x1488E050: dena::thread<dena::worker_throbj>::thread_main(void*) (thread.hpp:71)
   by 0x58A16F9: start_thread (pthread_create.c:333)
   by 0x645EB5C: clone (clone.S:109)

131,072 bytes in 16 blocks are indirectly lost in loss record 311 of 345
   at 0x4C2E0EF: operator new(unsigned long) (vg_replace_malloc.c:333)
   by 0x1487E4DF: __gnu_cxx::new_allocator<char>::allocate(unsigned long, void const*) (new_allocator.h:104)
   by 0x1487DC1B: __gnu_cxx::__alloc_traits<std::allocator<char> >::allocate(std::allocator<char>&, unsigned long) (alloc_traits.h:182)
   by 0x1487D301: std::_Vector_base<char, std::allocator<char> >::_M_allocate(unsigned long) (stl_vector.h:170)
   by 0x1487C0A1: std::vector<char, std::allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, unsigned long, char const&) (vector.tcc:491)
   by 0x1487AB61: std::vector<char, std::allocator<char> >::insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, unsigned long, char const&) (stl_vector.h:1073)
   by 0x14879C44: std::vector<char, std::allocator<char> >::resize(unsigned long, char) (stl_vector.h:716)
   by 0x148736EC: dena::dbcontext::dbcontext(dena::database volatile*, bool) (database.cpp:226)
   by 0x148734D8: dena::database::create_context(bool) volatile (database.cpp:200)
   by 0x148819D0: dena::hstcpsvr_worker::hstcpsvr_worker(dena::hstcpsvr_worker_arg const&) (hstcpsvr_worker.cpp:287)
   by 0x14884D47: dena::hstcpsvr_worker_i::create(dena::hstcpsvr_worker_arg const&) (hstcpsvr_worker.cpp:943)
   by 0x1488C741: dena::worker_throbj::worker_throbj(dena::hstcpsvr_worker_arg const&) (hstcpsvr.cpp:30)
   by 0x1488CD1A: dena::thread<dena::worker_throbj>::thread<dena::hstcpsvr_worker_arg>(dena::hstcpsvr_worker_arg const&, unsigned long) (thread.hpp:22)
   by 0x1488C142: dena::hstcpsvr::start_listen[abi:cxx11]() (hstcpsvr.cpp:122)
   by 0x1487FB9A: daemon_handlersocket_init(void*) (handlersocket.cpp:84)
   by 0x828808: plugin_initialize(st_plugin_int*) (sql_plugin.cc:1159)

737,280 bytes in 6 blocks are indirectly lost in loss record 325 of 345
   at 0x4C2E0EF: operator new(unsigned long) (vg_replace_malloc.c:333)
   by 0x1488A980: __gnu_cxx::new_allocator<epoll_event>::allocate(unsigned long, void const*) (new_allocator.h:104)
   by 0x1488A267: __gnu_cxx::__alloc_traits<std::allocator<epoll_event> >::allocate(std::allocator<epoll_event>&, unsigned long) (alloc_traits.h:182)
   by 0x1488949F: std::_Vector_base<epoll_event, std::allocator<epoll_event> >::_M_allocate(unsigned long) (stl_vector.h:170)
   by 0x14887BE4: std::vector<epoll_event, std::allocator<epoll_event> >::_M_fill_insert(__gnu_cxx::__normal_iterator<epoll_event*, std::vector<epoll_event, std::allocator<epoll_event> > >, unsigned long, epoll_event const&) (vector.tcc:491)
   by 0x14886AA5: std::vector<epoll_event, std::allocator<epoll_event> >::insert(__gnu_cxx::__normal_iterator<epoll_event*, std::vector<epoll_event, std::allocator<epoll_event> > >, unsigned long, epoll_event const&) (stl_vector.h:1073)
   by 0x14885E42: std::vector<epoll_event, std::allocator<epoll_event> >::resize(unsigned long, epoll_event) (stl_vector.h:716)
   by 0x14881BF4: dena::hstcpsvr_worker::hstcpsvr_worker(dena::hstcpsvr_worker_arg const&) (hstcpsvr_worker.cpp:301)
   by 0x14884D47: dena::hstcpsvr_worker_i::create(dena::hstcpsvr_worker_arg const&) (hstcpsvr_worker.cpp:943)
   by 0x1488C741: dena::worker_throbj::worker_throbj(dena::hstcpsvr_worker_arg const&) (hstcpsvr.cpp:30)
   by 0x1488CD1A: dena::thread<dena::worker_throbj>::thread<dena::hstcpsvr_worker_arg>(dena::hstcpsvr_worker_arg const&, unsigned long) (thread.hpp:22)
   by 0x1488C142: dena::hstcpsvr::start_listen[abi:cxx11]() (hstcpsvr.cpp:122)
   by 0x1487FB9A: daemon_handlersocket_init(void*) (handlersocket.cpp:84)
   by 0x828808: plugin_initialize(st_plugin_int*) (sql_plugin.cc:1159)
   by 0x82A9A3: mysql_install_plugin(THD*, st_mysql_lex_string const*, st_mysql_lex_string const*) (sql_plugin.cc:1912)
   by 0x81B4E6: mysql_execute_command(THD*) (sql_parse.cc:5422)

887,168 (512 direct, 886,656 indirect) bytes in 16 blocks are definitely lost in loss record 326 of 345
   at 0x4C2E0EF: operator new(unsigned long) (vg_replace_malloc.c:333)
   by 0x1488C129: dena::hstcpsvr::start_listen[abi:cxx11]() (hstcpsvr.cpp:122)
   by 0x1487FB9A: daemon_handlersocket_init(void*) (handlersocket.cpp:84)
   by 0x828808: plugin_initialize(st_plugin_int*) (sql_plugin.cc:1159)
   by 0x82A9A3: mysql_install_plugin(THD*, st_mysql_lex_string const*, st_mysql_lex_string const*) (sql_plugin.cc:1912)
   by 0x81B4E6: mysql_execute_command(THD*) (sql_parse.cc:5422)
   by 0x81F444: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:6972)
   by 0x81092E: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1441)
   by 0x80F7A4: do_command(THD*) (sql_parse.cc:1053)
   by 0x7D27F1: do_handle_one_connection(THD*) (sql_connect.cc:1541)
   by 0x7D2289: handle_one_connection (sql_connect.cc:1444)
   by 0xB958CF: pfs_spawn_thread (pfs.cc:1860)
   by 0x58A16F9: start_thread (pthread_create.c:333)
   by 0x645EB5C: clone (clone.S:109)

1,228,800 bytes in 10 blocks are possibly lost in loss record 333 of 345
   at 0x4C2E0EF: operator new(unsigned long) (vg_replace_malloc.c:333)
   by 0x1488A980: __gnu_cxx::new_allocator<epoll_event>::allocate(unsigned long, void const*) (new_allocator.h:104)
   by 0x1488A267: __gnu_cxx::__alloc_traits<std::allocator<epoll_event> >::allocate(std::allocator<epoll_event>&, unsigned long) (alloc_traits.h:182)
   by 0x1488949F: std::_Vector_base<epoll_event, std::allocator<epoll_event> >::_M_allocate(unsigned long) (stl_vector.h:170)
   by 0x14887BE4: std::vector<epoll_event, std::allocator<epoll_event> >::_M_fill_insert(__gnu_cxx::__normal_iterator<epoll_event*, std::vector<epoll_event, std::allocator<epoll_event> > >, unsigned long, epoll_event const&) (vector.tcc:491)
   by 0x14886AA5: std::vector<epoll_event, std::allocator<epoll_event> >::insert(__gnu_cxx::__normal_iterator<epoll_event*, std::vector<epoll_event, std::allocator<epoll_event> > >, unsigned long, epoll_event const&) (stl_vector.h:1073)
   by 0x14885E42: std::vector<epoll_event, std::allocator<epoll_event> >::resize(unsigned long, epoll_event) (stl_vector.h:716)
   by 0x14881BF4: dena::hstcpsvr_worker::hstcpsvr_worker(dena::hstcpsvr_worker_arg const&) (hstcpsvr_worker.cpp:301)
   by 0x14884D47: dena::hstcpsvr_worker_i::create(dena::hstcpsvr_worker_arg const&) (hstcpsvr_worker.cpp:943)
   by 0x1488C741: dena::worker_throbj::worker_throbj(dena::hstcpsvr_worker_arg const&) (hstcpsvr.cpp:30)
   by 0x1488CD1A: dena::thread<dena::worker_throbj>::thread<dena::hstcpsvr_worker_arg>(dena::hstcpsvr_worker_arg const&, unsigned long) (thread.hpp:22)
   by 0x1488C142: dena::hstcpsvr::start_listen[abi:cxx11]() (hstcpsvr.cpp:122)
   by 0x1487FB9A: daemon_handlersocket_init(void*) (handlersocket.cpp:84)
   by 0x828808: plugin_initialize(st_plugin_int*) (sql_plugin.cc:1159)
   by 0x82A9A3: mysql_install_plugin(THD*, st_mysql_lex_string const*, st_mysql_lex_string const*) (sql_plugin.cc:1912)
   by 0x81B4E6: mysql_execute_command(THD*) (sql_parse.cc:5422)

LEAK SUMMARY:
   definitely lost: 512 bytes in 16 blocks
   indirectly lost: 886,656 bytes in 70 blocks
     possibly lost: 1,228,800 bytes in 10 blocks
   still reachable: 95 bytes in 1 blocks
        suppressed: 68,995,926 bytes in 1,037 blocks

This is caused by hstcpsvr::threads, which is auto_ptrcontainer<std::vector<worker_thread_type *> > (effectively std::vector<foo *>), having clear() method, which clears the container without deleting the member pointers, which are owned by the container.

[Laurynas Biveinis (laurynas-biveinis)]

     https://github.com/percona/percona-server/pull/937
     https://github.com/percona/percona-server/pull/938

[Laurynas Biveinis (laurynas-biveinis)]

Reported upstream as https://github.com/DeNA/HandlerSocket-Plugin-for-MySQL/issues/113

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3536