threadpool UNIV_DEBUG | void remove_global_thread(THD*): Assertion `thd->release_resources_done()' failed

[Raghavendra D Prabhu]

# 2014-06-23T01:59:34 [6941] #0 0x00007f50eecbc69c in pthread_kill () from /lib64/libpthread.so.0
# 2014-06-23T01:59:34 [6941] #1 0x0000000000affeb6 in my_write_core (sig=6) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/mysys/stacktrace.c:422
# 2014-06-23T01:59:34 [6941] #2 0x0000000000762c3d in handle_fatal_signal (sig=6) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/sql/signal_handler.cc:254
# 2014-06-23T01:59:34 [6941] #3 <signal handler called>
# 2014-06-23T01:59:34 [6941] #4 0x00007f50ed1098e5 in raise () from /lib64/libc.so.6
# 2014-06-23T01:59:34 [6941] #5 0x00007f50ed10b0c5 in abort () from /lib64/libc.so.6
# 2014-06-23T01:59:34 [6941] #6 0x00007f50ed102a0e in __assert_fail_base () from /lib64/libc.so.6
# 2014-06-23T01:59:34 [6941] #7 0x00007f50ed102ad0 in __assert_fail () from /lib64/libc.so.6
# 2014-06-23T01:59:34 [6941] #8 0x0000000000636549 in remove_global_thread (thd=0x7f50c40009a0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/sql/mysqld.cc:916
# 2014-06-23T01:59:34 [6941] #9 0x000000000063f92e in start_wsrep_THD (arg=0x66139f) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/sql/mysqld.cc:5554
# 2014-06-23T01:59:34 [6941] #10 0x00007f50eecb7851 in start_thread () from /lib64/libpthread.so.0
# 2014-06-23T01:59:34 [6941] #11 0x00007f50ed1bf94d in clone () from /lib64/libc.so.6

==========================================================================================================

2014-06-23 01:59:31 10559 [Note] WSREP: gcomm: closing backend
2014-06-23 01:59:31 10559 [Note] WSREP: Received self-leave message.
2014-06-23 01:59:31 10559 [Note] WSREP: gcomm: closed
2014-06-23 01:59:31 10559 [Note] WSREP: Flow-control interval: [0, 0]
2014-06-23 01:59:31 10559 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2014-06-23 01:59:31 10559 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 45)
2014-06-23 01:59:31 10559 [Note] WSREP: RECV thread exiting 0: Success
2014-06-23 01:59:31 10559 [Note] WSREP: New cluster view: global state: 7c3e9610-fa9a-11e3-8734-7218126acbcf:45, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 3
2014-06-23 01:59:31 10559 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-06-23 01:59:31 10559 [Note] WSREP: applier thread exiting (code:0)
2014-06-23 01:59:31 10559 [Note] WSREP: recv_thread() joined.
2014-06-23 01:59:31 10559 [Note] WSREP: Closing replication queue.
2014-06-23 01:59:31 10559 [Note] WSREP: Closing slave action queue.
mysqld: /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/sql/mysqld.cc:916: void remove_global_thread(THD*): Assertion `thd->release_resources_done()' failed.
05:59:31 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=8
max_threads=100001
thread_count=2
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 38410895 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f50c40009a0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f50ef174da0 thread_stack 0x40000
/rqg/workspace/PXC-5.6-mrandgen/BTYPE/debug/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.19-25.6.803.Linux.x86_64/bin/mysqld(my_print_stacktrace+0x35)[0xaffe34]
/rqg/workspace/PXC-5.6-mrandgen/BTYPE/debug/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.19-25.6.803.Linux.x86_64/bin/mysqld(handle_fatal_signal+0x42e)[0x762a02]
/lib64/libpthread.so.0(+0xf500)[0x7f50eecbf500]
/lib64/libc.so.6(gsignal+0x35)[0x7f50ed1098e5]
/lib64/libc.so.6(abort+0x175)[0x7f50ed10b0c5]
/lib64/libc.so.6(+0x2ba0e)[0x7f50ed102a0e]
/lib64/libc.so.6(__assert_perror_fail+0x0)[0x7f50ed102ad0]
/rqg/workspace/PXC-5.6-mrandgen/BTYPE/debug/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.19-25.6.803.Linux.x86_64/bin/mysqld(_Z20remove_global_threadP3THD+0x8b)[0x636549]
/rqg/workspace/PXC-5.6-mrandgen/BTYPE/debug/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.19-25.6.803.Linux.x86_64/bin/mysqld(start_wsrep_THD+0x4d9)[0x63f92e]
/lib64/libpthread.so.0(+0x7851)[0x7f50eecb7851]
/lib64/libc.so.6(clone+0x6d)[0x7f50ed1bf94d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 1
Status: KILL_CONNECTION

[Raghavendra D Prabhu (raghavendra-prabhu)]

Logs.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Main RQG tests with threadpool crashed during shutdown - http://jenkins.percona.com/job/PXC-5.6-mrandgen/133/BTYPE=debug,Host=pxc-rqg/console - validating this issue.

Removing the threadpool for wsrep internal threads should fix this for long term.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1146