Percona server 5.6.12 started with thread_pool hangs on shutdown
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Percona Server moved to https://jira.percona.com/projects/PS |
Fix Released
|
High
|
Sergei Glushchenko | ||
5.1 |
Invalid
|
Undecided
|
Unassigned | ||
5.5 |
Invalid
|
Undecided
|
Unassigned | ||
5.6 |
Fix Released
|
High
|
Sergei Glushchenko |
Bug Description
to reproduce this issue one need to start server with thread_pool, make connection, start shutdown.
console #1:
[root@host] cd bin64_5612.rc60.4
[root@host] ./scripts/
[root@host] ./bin/mysqld --no-defaults --basedir=. --datadir=data --thread_
...
Version: '5.6.12-rc60.4' socket: '/tmp/mysql.sock' port: 3306 Percona Server with XtraDB (GPL), Release rc60.4, Revision 393
console #2:
[root@host] mysql -uroot test
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.12-rc60.4 Percona Server with XtraDB (GPL), Release rc60.4, Revision 393
....
console #3:
mysqladmin shutdown -uroot -S /tmp/mysql.sock
<HANG>
console #1:
2013-07-15 21:41:45 1195 [Note] ./bin/mysqld: Normal shutdown
2013-07-15 21:41:45 1195 [Note] Giving 1 client threads a chance to die gracefully
2013-07-15 21:41:45 1195 [Note] Event Scheduler: Purging the queue. 0 events
2013-07-15 21:41:45 1195 [Note] Shutting down slave threads
2013-07-15 21:41:47 1195 [Note] Forcefully disconnecting 1 remaining clients
2013-07-15 21:41:47 1195 [Warning] ./bin/mysqld: Forcing close of thread 2 user: 'root'
<HANG>
backtrace
Thread 26 (Thread 0x7f994a209700 (LWP 1197)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_
at /mnt/workspace/
#2 os_aio_linux_handle (global_seg=0, message1=
at /mnt/workspace/
#3 0x0000000000a832aa in fil_aio_wait (segment=0) at /mnt/workspace/
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f9922388700 (LWP 1198)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_
at /mnt/workspace/
#2 os_aio_linux_handle (global_seg=1, message1=
at /mnt/workspace/
#3 0x0000000000a832aa in fil_aio_wait (segment=1) at /mnt/workspace/
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f9921987700 (LWP 1199)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_
at /mnt/workspace/
#2 os_aio_linux_handle (global_seg=2, message1=
at /mnt/workspace/
#3 0x0000000000a832aa in fil_aio_wait (segment=2) at /mnt/workspace/
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f9920f86700 (LWP 1200)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_
at /mnt/workspace/
#2 os_aio_linux_handle (global_seg=3, message1=
at /mnt/workspace/
#3 0x0000000000a832aa in fil_aio_wait (segment=3) at /mnt/workspace/
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
---Type <return> to continue, or q <return> to quit---
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f9920585700 (LWP 1201)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_
at /mnt/workspace/
#2 os_aio_linux_handle (global_seg=4, message1=
at /mnt/workspace/
#3 0x0000000000a832aa in fil_aio_wait (segment=4) at /mnt/workspace/
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f991fb84700 (LWP 1202)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_
at /mnt/workspace/
#2 os_aio_linux_handle (global_seg=5, message1=
at /mnt/workspace/
#3 0x0000000000a832aa in fil_aio_wait (segment=5) at /mnt/workspace/
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f991f183700 (LWP 1203)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_
at /mnt/workspace/
#2 os_aio_linux_handle (global_seg=6, message1=
at /mnt/workspace/
#3 0x0000000000a832aa in fil_aio_wait (segment=6) at /mnt/workspace/
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f991e782700 (LWP 1204)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_
at /mnt/workspace/
#2 os_aio_linux_handle (global_seg=7, message1=
at /mnt/workspace/
#3 0x0000000000a832aa in fil_aio_wait (segment=7) at /mnt/workspace/
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
at /mnt/workspace/
---Type <return> to continue, or q <return> to quit---
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f991dd81700 (LWP 1205)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_
at /mnt/workspace/
#2 os_aio_linux_handle (global_seg=8, message1=
at /mnt/workspace/
#3 0x0000000000a832aa in fil_aio_wait (segment=8) at /mnt/workspace/
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f991d380700 (LWP 1206)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_
at /mnt/workspace/
#2 os_aio_linux_handle (global_seg=9, message1=
at /mnt/workspace/
#3 0x0000000000a832aa in fil_aio_wait (segment=9) at /mnt/workspace/
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f991b8d2700 (LWP 1208)):
#0 0x00007f994d14f7bb in pthread_
#1 0x000000000097c92d in os_cond_wait_timed (event=0x1afe8c0, time_in_usec=<value optimized out>, reset_sig_count=1)
at /mnt/workspace/
#2 os_event_
at /mnt/workspace/
#3 0x000000000095e663 in lock_wait_
at /mnt/workspace/
#4 0x00007f994d14b851 in start_thread () from /lib64/
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f991aed1700 (LWP 1209)):
#0 0x00007f994d14f7bb in pthread_
#1 0x000000000097c92d in os_cond_wait_timed (event=0x173d2f0, time_in_usec=<value optimized out>, reset_sig_count=1)
at /mnt/workspace/
#2 os_event_
at /mnt/workspace/
#3 0x00000000009df3b5 in srv_error_
at /mnt/workspace/
#4 0x00007f994d14b851 in start_thread () from /lib64/
---Type <return> to continue, or q <return> to quit---
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f991a4d0700 (LWP 1210)):
#0 0x00007f994d14f7bb in pthread_
#1 0x000000000097c92d in os_cond_wait_timed (event=0x173d380, time_in_usec=<value optimized out>, reset_sig_count=1)
at /mnt/workspace/
#2 os_event_
at /mnt/workspace/
#3 0x00000000009e0ead in srv_monitor_thread (arg=<value optimized out>)
at /mnt/workspace/
#4 0x00007f994d14b851 in start_thread () from /lib64/
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f991c2d3700 (LWP 1211)):
#0 0x00007f994b9044f3 in select () from /lib64/libc.so.6
#1 0x000000000097cccf in os_thread_sleep (tm=<value optimized out>)
at /mnt/workspace/
#2 0x00000000009de5b3 in srv_master_sleep (arg=<value optimized out>)
at /mnt/workspace/
#3 srv_master_thread (arg=<value optimized out>) at /mnt/workspace/
#4 0x00007f994d14b851 in start_thread () from /lib64/
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f9919acf700 (LWP 1212)):
#0 0x00007f994d14f43c in pthread_
#1 0x000000000097cad2 in os_cond_wait (event=0x173d260, reset_sig_count=1)
at /mnt/workspace/
#2 os_event_wait_low (event=0x173d260, reset_sig_count=1)
at /mnt/workspace/
#3 0x00000000009e189d in srv_purge_
at /mnt/workspace/
#4 srv_purge_
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f99190ce700 (LWP 1213)):
#0 0x00007f994b9044f3 in select () from /lib64/libc.so.6
#1 0x000000000097cccf in os_thread_sleep (tm=<value optimized out>)
at /mnt/workspace/
#2 0x0000000000a475cb in page_cleaner_
at /mnt/workspace/
#3 buf_flush_
at /mnt/workspace/
#4 0x00007f994d14b851 in start_thread () from /lib64/
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 10 (Thread 0x7f99186cd700 (LWP 1214)):
#0 0x00007f994d14f43c in pthread_
#1 0x000000000097cad2 in os_cond_wait (event=0x173d410, reset_sig_count=1)
at /mnt/workspace/
#2 os_event_wait_low (event=0x173d410, reset_sig_count=1)
at /mnt/workspace/
#3 0x0000000000a42421 in buf_dump_thread (arg=<value optimized out>)
at /mnt/workspace/
#4 0x00007f994d14b851 in start_thread () from /lib64/
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f9917ccc700 (LWP 1215)):
#0 0x00007f994d14f7bb in pthread_
#1 0x000000000097c92d in os_cond_wait_timed (event=0x1b05640, time_in_usec=<value optimized out>, reset_sig_count=1)
at /mnt/workspace/
#2 os_event_
at /mnt/workspace/
#3 0x0000000000a75255 in dict_stats_thread (arg=<value optimized out>)
at /mnt/workspace/
#4 0x00007f994d14b851 in start_thread () from /lib64/
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f99172cb700 (LWP 1216)):
#0 0x00007f994d14f7bb in pthread_
#1 0x000000000097c92d in os_cond_wait_timed (event=0x1b35ff0, time_in_usec=<value optimized out>, reset_sig_count=1)
at /mnt/workspace/
#2 os_event_
at /mnt/workspace/
#3 0x0000000000a0ac04 in ib_wqueue_timedwait (wq=0x1b35e68, wait_in_
at /mnt/workspace/
#4 0x0000000000aa5721 in fts_optimize_thread (arg=<value optimized out>)
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f99168ca700 (LWP 1233)):
#0 0x00007f994d14f7bb in pthread_
#1 0x000000000079c96c in inline_
at /mnt/workspace/
#2 timer_thread (param=<value optimized out>) at /mnt/workspace/
#3 0x0000000000b3fa0b in pfs_spawn_thread (arg=<value optimized out>)
at /mnt/workspace/
#4 0x00007f994d14b851 in start_thread () from /lib64/
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f994859b700 (LWP 1234)):
#0 0x00007f994d1532a5 in sigwait () from /lib64/
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000583ec0 in signal_hand (arg=<value optimized out>) at /mnt/workspace/
#2 0x0000000000b3fa0b in pfs_spawn_thread (arg=<value optimized out>)
at /mnt/workspace/
#3 0x00007f994d14b851 in start_thread () from /lib64/
#4 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f994855a700 (LWP 1237)):
#0 0x00007f994b90bf03 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000079aee1 in io_poll_wait (pollfd=12, native_
at /mnt/workspace/
#2 0x000000000079bc1a in get_event (current_
at /mnt/workspace/
#3 0x000000000079c064 in worker_main (param=0x1316200) at /mnt/workspace/
#4 0x0000000000b3fa0b in pfs_spawn_thread (arg=<value optimized out>)
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f9948519700 (LWP 1240)):
#0 0x00007f994b90bf03 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000079aee1 in io_poll_wait (pollfd=13, native_
at /mnt/workspace/
#2 0x000000000079bc1a in get_event (current_
at /mnt/workspace/
#3 0x000000000079c064 in worker_main (param=0x1316400) at /mnt/workspace/
#4 0x0000000000b3fa0b in pfs_spawn_thread (arg=<value optimized out>)
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f99484d8700 (LWP 1243)):
#0 0x00007f994b90bf03 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000079aee1 in io_poll_wait (pollfd=14, native_
at /mnt/workspace/
#2 0x000000000079bc1a in get_event (current_
at /mnt/workspace/
#3 0x000000000079c064 in worker_main (param=0x1316600) at /mnt/workspace/
#4 0x0000000000b3fa0b in pfs_spawn_thread (arg=<value optimized out>)
at /mnt/workspace/
#5 0x00007f994d14b851 in start_thread () from /lib64/
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f9948497700 (LWP 1244)):
#0 0x00007f994d14f43c in pthread_
#1 0x000000000058ab54 in kill_server (arg=<value optimized out>)
at /mnt/workspace/
#2 kill_server_thread (arg=<value optimized out>) at /mnt/workspace/
---Type <return> to continue, or q <return> to quit---
#3 0x0000000000b3fa0b in pfs_spawn_thread (arg=<value optimized out>)
at /mnt/workspace/
#4 0x00007f994d14b851 in start_thread () from /lib64/
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f994d56a7e0 (LWP 1195)):
#0 0x00007f994d14f43c in pthread_
#1 0x00000000005895a8 in inline_
at /mnt/workspace/
#2 mysqld_main (argc=7, argv=0x1633de8) at /mnt/workspace/
#3 0x00007f994b841cdd in __libc_start_main () from /lib64/libc.so.6
#4 0x000000000057c799 in _start ()
Related branches
- Laurynas Biveinis (community): Approve
- Alexey Kopytov (community): Needs Fixing
- Vlad Lesin (community): Needs Fixing (g2)
-
Diff: 340 lines (+115/-42)13 files modifiedPercona-Server/include/violite.h (+7/-0)
Percona-Server/mysql-test/r/percona_bug1201681.result (+6/-0)
Percona-Server/mysql-test/t/percona_bug1201681-master.opt (+1/-0)
Percona-Server/mysql-test/t/percona_bug1201681.test (+27/-0)
Percona-Server/sql/mysqld.cc (+1/-1)
Percona-Server/sql/sql_class.cc (+2/-27)
Percona-Server/sql/threadpool_common.cc (+8/-2)
Percona-Server/sql/threadpool_unix.cc (+1/-1)
Percona-Server/vio/vio.c (+4/-0)
Percona-Server/vio/vio_priv.h (+2/-0)
Percona-Server/vio/viopipe.c (+14/-0)
Percona-Server/vio/vioshm.c (+16/-0)
Percona-Server/vio/viosocket.c (+26/-11)
Sergei, can you take a look?