crash with signal 11 : handle_fatal_signal / libpthread.so.0

[jfr]

Hello

We've got recurent crash on a two nodes Galera cluster.

Here are the errors of the last two crashs :

Yesterday on node 2 :

140226 22:48:16 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 5.5.34-MariaDB-wsrep-log
key_buffer_size=16777216
read_buffer_size=1048576
max_used_connections=53
max_threads=202
thread_count=47
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 433752 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7f33e27e5000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f35158d7d68 thread_stack 0x48000
??:0(my_print_stacktrace)[0xaad6ee]
??:0(handle_fatal_signal)[0x6ecccb]
??:0(??)[0x384ea0f500]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f33df689018): is an invalid pointer
Connection ID (thread ID): 18926
Status: KILL_QUERY

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

resolved stack :

0xaad6ee my_print_stacktrace + 46
0x6ecccb handle_fatal_signal + 1035
0x384ea0f500 _end + 1291719368

Today on the other node (node 1) :

140227 16:04:22 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 5.5.34-MariaDB-wsrep-log
key_buffer_size=16777216
read_buffer_size=1048576
max_used_connections=88
max_threads=202
thread_count=43
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 433752 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7f0c30ad4000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f0d73fb1d68 thread_stack 0x48000
(my_addr_resolve failure: fork)
/usr/sbin/mysqld(my_print_stacktrace+0x2e) [0xaad6ee]
/usr/sbin/mysqld(handle_fatal_signal+0x40b) [0x6ecccb]
/lib64/libpthread.so.0() [0x348fa0f500]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f0c3af56018): is an invalid pointer
Connection ID (thread ID): 57062
Status: KILL_QUERY

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

resolved stack :
0xaad6ee my_print_stacktrace + 46
0x6ecccb handle_fatal_signal + 1035
0x348fa0f500 _end + -1912728888

Version : Server version: 5.5.34-MariaDB-wsrep-log MariaDB Server, wsrep_23.7.6
OS : CentOS release 6.4 (Final)

Thanks
Julien

[jfr (j-francoz)]

Hello

This bug is maybe related to https://bugs.launchpad.net/maria/+bug/1020645 : crashes (sig 11) with 5.3.7-MariaDB union query

Each time it occurs we have transactions like :
UPDATE users ....
SELECT ... from users UNION ALL SELECT ....
commit

in case of deadlocks between different nodes of the cluster, maybe during execution of the SELECT ... UNION ALL ..., we got the signal 11 crash.

We're now trying to reproduce the crash after update to 5.5.36.