Percona XtraDB crash on daily basis

[xamzab]

Hello,

I made a Percona XtraDB Cluster with 3 same-hardware nodes and having one node crashing each day...

Here is a few lines of log:

====

2015-03-25 09:22:38 4602 [Warning] Client failed to provide its character set. 'latin1' will be used as client character set.
2015-03-25 09:22:39 4602 [Warning] Client failed to provide its character set. 'latin1' will be used as client character set.
08:22:39 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=256
max_threads=10002
thread_count=57
connection_count=55
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4123730 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7ffbb4177c80
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ffbeeff9e50 thread_stack 0x1000000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7ffd8bc1d0a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x79429)[0x7ffd89e9e429]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7ffd89e9fa70]
/usr/sbin/mysqld(my_malloc+0x32)[0x8c8bc2]
/usr/sbin/mysqld(reset_root_defaults+0x67)[0x8c4aa7]
/usr/sbin/mysqld(_ZN3THD16init_for_queriesEP14Relay_log_info+0x43)[0x6c7b93]
/usr/sbin/mysqld(_Z28prepare_new_connection_stateP3THD+0xe3)[0x6d0ad3]
/usr/sbin/mysqld(_Z22thd_prepare_connectionP3THD+0x1a0)[0x6d0eb0]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x137)[0x6d1067]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6d1262]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0xb195d0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7ffd8bc14b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ffd89f0095d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 442848
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
150325 09:22:40 mysqld_safe Number of processes running now: 0
150325 09:22:40 mysqld_safe WSREP: not restarting wsrep node automatically
150325 09:22:40 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

===

Hopes to find the issue to get a secure DB Cluster soon.

Thanks in advance.

[Valerii Kravchuk (valerii-kravchuk)]

What exact version of Percona XtraDB Cluster, 5.x.y-..., do you use?

Please, send my.cnf file content and the output of pt-summary script from the crashing node.

[xamzab (xamzab)]

Here is some information:

dpkg -l|grep percona

ii percona-xtrabackup 2.2.9-5067-1.wheezy amd64 Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-56 5.6.22-25.8-978.wheezy amd64 Percona XtraDB Cluster with Galera
ii percona-xtradb-cluster-client-5.6 5.6.22-25.8-978.wheezy amd64 Percona XtraDB Cluster database client binaries
/my.cnf)
ii percona-xtradb-cluster-common-5.6 5.6.22-25.8-978.wheezy amd64 Percona XtraDB Cluster database common files (e.g. /etc/mysql
/my.cnf)
ii percona-xtradb-cluster-galera-3 3.9.3494.wheezy amd64 Metapackage for latest version of galera3.
ii percona-xtradb-cluster-galera-3.x 3.9.3494.wheezy amd64 Galera components of Percona XtraDB Cluster
ii percona-xtradb-cluster-server-5.6 5.6.22-25.8-978.wheezy amd64 Percona XtraDB Cluster database server binaries

uname -a

Linux sql1 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u1 x86_64 GNU/Linux

[xamzab (xamzab)]

And here is pt-summary's output.

[Valerii Kravchuk (valerii-kravchuk)]

I see at least two potential problems (for a server in VM with <6G of memory) in my.cnf:

max_connections = 10000
query_cache_size = 1G

Can you, please, check if with max_connections of, say, 500, and query_cache_size of 32M you still have crashes?

It would be useful to the output of SHOW GLOBAL STATUS after having this node running for some time. For now even pt-summary is provided while mysqld process is NOT running, so we can not even see how "close" to out of memory condition it may be under real load.

[xamzab (xamzab)]

Applied your recommendations and waiting for some hours to run again pt-summary.

[xamzab (xamzab)]

Here is some info you asked.

[xamzab (xamzab)]

Hi,

One node has crashed a few minutes ago. It looks like the VM has just stopped. Restarted it to see if this happens again. Will let you know about what I found.

[xamzab (xamzab)]

The Percona node has crashed again.

Here is what we got:

04:27:45 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=78
max_threads=502
thread_count=71
connection_count=69
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 331448 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f36ec29bde0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f36d9fece50 thread_stack 0x1000000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f37ec6890a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x7890c)[0x7f37ea90990c]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7f37ea90ba70]
/usr/sbin/mysqld(my_malloc+0x32)[0x8c8bc2]
/usr/sbin/mysqld(alloc_root+0x8c)[0x8c4b9c]
/usr/sbin/mysqld(_ZN9Sql_allocnwEm+0x9)[0x601939]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x2f6)[0x71e6d6]
/usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x175)[0x71e8f5]
/usr/sbin/mysqld[0x59a85f]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xb60)[0x6f8810]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x5c8)[0x6fd7f8]
/usr/sbin/mysqld[0x6fd8f2]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1076)[0x6fef36]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x202)[0x6ffbe2]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x2ad)[0x6d11dd]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6d1262]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0xb195d0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f37ec680b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f37ea96c95d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (38df1e0): [SQL QUERY REMOVED]
Connection ID (thread ID): 219313
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
150329 06:27:46 mysqld_safe Number of processes running now: 0
150329 06:27:46 mysqld_safe WSREP: not restarting wsrep node automatically
150329 06:27:46 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

[xamzab (xamzab)]

Here is another crash log from another crashed node.

13:01:14 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=119
max_threads=502
thread_count=72
connection_count=70
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 331448 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fc280387050
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fc276ff5e50 thread_stack 0x1000000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7fc35e42f0a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x7890c)[0x7fc35c6af90c]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7fc35c6b1a70]
/usr/sbin/mysqld(my_malloc+0x32)[0x8c8bc2]
/usr/sbin/mysqld(alloc_root+0x8c)[0x8c4b9c]
/usr/sbin/mysqld[0x82e8dd]
/usr/sbin/mysqld[0x82fa05]
/usr/sbin/mysqld[0x83a0bc]
/usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x60c)[0x83c0bc]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x21c)[0x71e5fc]
/usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x175)[0x71e8f5]
/usr/sbin/mysqld[0x59a85f]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xb60)[0x6f8810]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x5c8)[0x6fd7f8]
/usr/sbin/mysqld[0x6fd8f2]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1076)[0x6fef36]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x202)[0x6ffbe2]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x2ad)[0x6d11dd]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6d1262]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0xb195d0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7fc35e426b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc35c71295d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (48e6b20): [SQL QUERY REMOVED]
Connection ID (thread ID): 1372718
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
150329 15:01:15 mysqld_safe Number of processes running now: 0
150329 15:01:15 mysqld_safe WSREP: not restarting wsrep node automatically
150329 15:01:15 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.p...

[xamzab (xamzab)]

Another crash log ...

16:01:05 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=123
max_threads=502
thread_count=55
connection_count=53
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 331448 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f42dc2d0470
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f4316fe9e50 thread_stack 0x1000000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f440256d0a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x79429)[0x7f44007ee429]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7f44007efa70]
/usr/sbin/mysqld(my_malloc+0x32)[0x8c8bc2]
/usr/sbin/mysqld(reset_root_defaults+0x67)[0x8c4aa7]
/usr/sbin/mysqld(_ZN3THD16init_for_queriesEP14Relay_log_info+0x43)[0x6c7b93]
/usr/sbin/mysqld(_Z28prepare_new_connection_stateP3THD+0xe3)[0x6d0ad3]
/usr/sbin/mysqld(_Z22thd_prepare_connectionP3THD+0x1a0)[0x6d0eb0]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x137)[0x6d1067]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6d1262]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0xb195d0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f4402564b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f440085095d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 1018029
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
150331 18:01:06 mysqld_safe Number of processes running now: 0
150331 18:01:06 mysqld_safe WSREP: not restarting wsrep node automatically
150331 18:01:06 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

[xamzab (xamzab)]

Here is another crash log from "Bootstraped" PXC node.

02:20:08 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=135
max_threads=502
thread_count=70
connection_count=68
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 331448 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f51d43188d0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f5197ff6e50 thread_stack 0x1000000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f52cf7ce0a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x76137)[0x7f52cda4c137]
/lib/x86_64-linux-gnu/libc.so.6(+0x77518)[0x7f52cda4d518]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f52cda5098c]
/usr/sbin/mysqld(free_root+0x73)[0x8c4df3]
/usr/sbin/mysqld(_ZN3THDD1Ev+0x158)[0x6cb728]
/usr/sbin/mysqld(_ZN3THDD0Ev+0x11)[0x6cba91]
/usr/sbin/mysqld(_Z29one_thread_per_connection_endP3THDb+0xc8)[0x5c3ef8]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0xf0)[0x6d1020]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6d1262]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0xb195d0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f52cf7c5b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f52cdab195d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 1524272
Status: KILL_CONNECTION

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
150404 04:20:09 mysqld_safe Number of processes running now: 0
150404 04:20:09 mysqld_safe WSREP: not restarting wsrep node automatically
150404 04:20:09 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

[xamzab (xamzab)]

Just after restarted the Primary PXC node, another node crashed a few minutes after.

Here is crash log

15:30:35 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=150
max_threads=502
thread_count=59
connection_count=57
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 331448 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f82081da690
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f8228e8fe50 thread_stack 0x1000000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f833465c0a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x78a8e)[0x7f83328dca8e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7f83328dea70]
/usr/sbin/mysqld(my_malloc+0x32)[0x8c8bc2]
/usr/sbin/mysqld(alloc_root+0x8c)[0x8c4b9c]
/usr/sbin/mysqld[0x82e8dd]
/usr/sbin/mysqld[0x82fa05]
/usr/sbin/mysqld[0x83a0bc]
/usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x60c)[0x83c0bc]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x21c)[0x71e5fc]
/usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x175)[0x71e8f5]
/usr/sbin/mysqld[0x59a85f]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xb60)[0x6f8810]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x5c8)[0x6fd7f8]
/usr/sbin/mysqld[0x6fd8f2]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1076)[0x6fef36]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x202)[0x6ffbe2]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x2ad)[0x6d11dd]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6d1262]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0xb195d0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f8334653b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f833293f95d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f81e03104d0): is an invalid pointer
Connection ID (thread ID): 2459594
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
150404 17:30:35 mysqld_safe Number of processes running now: 0
150404 17:30:35 mysqld_safe WSREP: not restarting wsrep node automatically
150404 17:30:35...

[xamzab (xamzab)]

Here is another crash log:

02:41:01 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=149
max_threads=502
thread_count=64
connection_count=62
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 331448 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7ff25035cfd0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ff280ffbe50 thread_stack 0x1000000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7ff35fb610a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x79429)[0x7ff35dde2429]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7ff35dde3a70]
/usr/sbin/mysqld(my_malloc+0x32)[0x8c8bc2]
/usr/sbin/mysqld(alloc_root+0x8c)[0x8c4b9c]
/usr/sbin/mysqld[0x838ada]
/usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x60c)[0x83c0bc]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x21c)[0x71e5fc]
/usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x175)[0x71e8f5]
/usr/sbin/mysqld[0x59a85f]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xb60)[0x6f8810]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x5c8)[0x6fd7f8]
/usr/sbin/mysqld[0x6fd8f2]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1076)[0x6fef36]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x202)[0x6ffbe2]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x2ad)[0x6d11dd]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6d1262]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0xb195d0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7ff35fb58b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ff35de4495d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (60fdbb0): SELECT username, pptp_password, trial_mode FROM users WHERE pptp_status = 1 AND inscrit = 1
Connection ID (thread ID): 653286
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
150409 04:41:01 mysqld_safe Number of processes running now: 0
150409 04:41:01 mysqld_safe WSREP: not restarting wsrep node automatically
150409 04:41:01 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended...

[xamzab (xamzab)]

Here is another different crash log:

08:10:02 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=153
max_threads=502
thread_count=141
connection_count=139
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 331448 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f90d811f650
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f9129ff0e50 thread_stack 0x1000000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f9214c9c0a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x7890c)[0x7f9212f1c90c]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7f9212f1ea70]
/usr/sbin/mysqld(my_malloc+0x32)[0x8c8bc2]
/usr/sbin/mysqld(init_io_cache+0x25e)[0x8b6f1e]
/usr/sbin/mysqld(mi_extra+0x348)[0xac4798]
/usr/sbin/mysqld(_ZN9ha_myisam9extra_optE17ha_extra_functionm+0x1c)[0xaa96fc]
/usr/sbin/mysqld(_Z8filesortP3THDP5TABLEP8FilesortbPyS5_+0x1b5a)[0x7d5c8a]
/usr/sbin/mysqld(_ZN13st_join_table10sort_tableEv+0x11a)[0x6dc37a]
/usr/sbin/mysqld(_Z21join_init_read_recordP13st_join_table+0x29)[0x6dc589]
/usr/sbin/mysqld(_ZN13QEP_tmp_table8end_sendEv+0xd8)[0x6defb8]
/usr/sbin/mysqld(_Z13sub_select_opP4JOINP13st_join_tableb+0x37)[0x6da327]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x2fb)[0x6d8f0b]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x265)[0x71e645]
/usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x175)[0x71e8f5]
/usr/sbin/mysqld[0x59a85f]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xb60)[0x6f8810]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x5c8)[0x6fd7f8]
/usr/sbin/mysqld[0x6fd8f2]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1076)[0x6fef36]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x202)[0x6ffbe2]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x2ad)[0x6d11dd]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6d1262]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0xb195d0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f9214c93b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f9212f7f95d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (67d5780): SELECT u.email, u.title, u.url, u.location, u.signature, u.email_setting, u.use_pm, u.num_posts, u.registered, u.admin_note, p.id...

[xamzab (xamzab)]

Reinstalled all nodes from scratch and here is what I've got:

11:27:58 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=93
max_threads=502
thread_count=61
connection_count=59
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 331448 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f2b742af5b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f2b797f3e50 thread_stack 0x1000000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f2c492530a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x78a8e)[0x7f2c474d3a8e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7f2c474d5a70]
/usr/sbin/mysqld(my_malloc+0x32)[0x8c8bc2]
/usr/sbin/mysqld(alloc_root+0x8c)[0x8c4b9c]
/usr/sbin/mysqld[0x82e8dd]
/usr/sbin/mysqld[0x82fa05]
/usr/sbin/mysqld[0x83a0bc]
/usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x60c)[0x83c0bc]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB
_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x21c)[0x71e5fc]
/usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x175)[0x71e8f5]
/usr/sbin/mysqld[0x59a85f]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xb60)[0x6f8810]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x5c8)[0x6fd7f8]
/usr/sbin/mysqld[0x6fd8f2]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1076)[0x6fef36]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x202)[0x6ffbe2]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x2ad)[0x6d11dd]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6d1262]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0xb195d0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f2c4924ab50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f2c4753695d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (3ac8dc0): SELECT COUNT(*) AS var1 FROM my_table WHERE my_field = 'my_value'
Connection ID (thread ID): 16968
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
150412 13:27:58 mysqld_safe Number of processes running now: 0
150412 13:27:58 mysqld_safe WSREP: not restarting wsrep node automatically
150412 13:27:58 mysq...

[xamzab (xamzab)]

Another crash log:

13:00:14 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=148
max_threads=502
thread_count=49
connection_count=47
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 331448 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x1000000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7fd7289d20a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x79429)[0x7fd726c53429]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7fd726c54a70]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_Znwm+0x1d)[0x7fd72746107d]
/usr/sbin/mysqld(_Z26handle_connections_socketsv+0x40f)[0x5c288f]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0xf22)[0x5ca142]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7fd726bf8ead]
/usr/sbin/mysqld[0x5bc9bd]
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
150416 15:00:14 mysqld_safe Number of processes running now: 0
150416 15:00:14 mysqld_safe WSREP: not restarting wsrep node automatically
150416 15:00:14 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

[Valerii Kravchuk (valerii-kravchuk)]

You had different crashes, but it seems they all were related to the inability of malloc to allocate the memory:

stack_bottom = 7f2b797f3e50 thread_stack 0x1000000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f2c492530a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x78a8e)[0x7f2c474d3a8e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7f2c474d5a70]
/usr/sbin/mysqld(my_malloc+0x32)[0x8c8bc2]
/usr/sbin/mysqld(alloc_root+0x8c)[0x8c4b9c]
/usr/sbin/mysqld[0x82e8dd]
/usr/sbin/mysqld[0x82fa05]
/usr/sbin/mysqld[0x83a0bc]
/usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x60c)[0x83c0bc]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB
_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x21c)[0x71e5fc]
/usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x175)[0x71e8f5]
/usr/sbin/mysqld[0x59a85f]
...

I wonder how much RAM do you have on host box and how many memory is allocated to all VMs running there in total, as I suspect these failures may be related to memory overcommit for your VmWare.

[xamzab (xamzab)]

Hello,

VMware Host has 8GB of RAM, and contains only one VM which is the MySQL's VM. This config is replicated 3 times as there is 3 nodes in that PXC.

Would you want me to reduce allocated RAM to something such as 4GB instead of 6GB actually ?

Regards.

[Valerii Kravchuk (valerii-kravchuk)]

So, you have 3 hardware boxes with 8G of RAM each, and just one VM with 6GB of RAM allocated on each of the boxes? If this is the case, maybe allocate more memory, not less per VM, 7GB or so and check if this helps to have less frequent crashes.

[xamzab (xamzab)]

Updated configurations of VMs to 7GB of RAM. Let's see what happens.

[xamzab (xamzab)]

It keeps crashing. What else to do ?

[Valerii Kravchuk (valerii-kravchuk)]

Try to comment out these suspicious settings in your my.cnf:

# thread_stack = 16M
# join_buffer_size = 8M
# max_binlog_size = 8GM

then restart the node and check if this changes anything.

[xamzab (xamzab)]

Configuration updated. Monitoring the cluster.

[xamzab (xamzab)]

The uptime was a bit longer than usual but the node crashed again.

Here is crashing log:

08:10:03 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=136
max_threads=502
thread_count=92
connection_count=83
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 331448 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x20b39d0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f1fd03c9e50 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f209c3090a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x78a8e)[0x7f209a589a8e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7f209a58ba70]
/usr/sbin/mysqld(my_malloc+0x32)[0x8c8bc2]
/usr/sbin/mysqld(alloc_root+0x8c)[0x8c4b9c]
/usr/sbin/mysqld(_ZN4ItemnwEmP11st_mem_root+0x12)[0x614972]
/usr/sbin/mysqld(_Z10MYSQLparseP3THD+0x119cf)[0x7a776f]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x268)[0x6fd498]
/usr/sbin/mysqld[0x6fd8f2]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x132f)[0x6ff1ef]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x202)[0x6ffbe2]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x2ad)[0x6d11dd]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6d1262]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0xb195d0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f209c300b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f209a5ec95d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f1fbc31ed9e): is an invalid pointer
Connection ID (thread ID): 394517
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
150424 10:10:03 mysqld_safe Number of processes running now: 0
150424 10:10:03 mysqld_safe WSREP: not restarting wsrep node automatically
150424 10:10:03 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

[Valerii Kravchuk (valerii-kravchuk)]

So, again crash for my_malloc with some new stack trace...

Can you, please, check dmest and /var/log/messages for anything referring to mysql around the moment of crash?

[xamzab (xamzab)]

That's something I've already tried but there is/was nothing in dmesg/messages/syslog/daemon.log related to moment of crash.

If there is anything to enable/update in order to get more information from these crashes, then feel free to tell me what to do.

[xamzab (xamzab)]

Got a new crash log after changed some settings such as innodb buffer size.

[CODE]
21:10:03 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=87
max_threads=502
thread_count=77
connection_count=68
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 331448 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fa3cc2d8e20
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fa3d6ef1e50 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7fa4eb3930a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x78580)[0x7fa4e9613580]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7fa4e9615a70]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_Znwm+0x1d)[0x7fa4e9e2207d]
/usr/lib/libgalera_smm.so(_ZNSt8_Rb_treeIlSt4pairIKlPKvESt10_Select1stIS4_ESt4lessIlESaIS4_EE10_M_insert_EPKSt18_Rb_tree_node_baseSD_RKS4_+0x44)[0x7fa4c31b49d4]
/usr/lib/libgalera_smm.so(_ZNSt8_Rb_treeIlSt4pairIKlPKvESt10_Select1stIS4_ESt4lessIlESaIS4_EE17_M_insert_unique_ESt23_Rb_tree_const_iteratorIS4_ERKS4_+0x16a)[0x7fa4c31b4bba]
/usr/lib/libgalera_smm.so(_ZN6gcache6GCache12seqno_assignEPKvll+0x89)[0x7fa4c31b4379]
/usr/lib/libgalera_smm.so(_ZN6galera13ReplicatorSMM4certEPNS_9TrxHandleE+0x12d)[0x7fa4c32cc6bd]
/usr/lib/libgalera_smm.so(_ZN6galera13ReplicatorSMM14cert_and_catchEPNS_9TrxHandleE+0x1e)[0x7fa4c32c21ae]
/usr/lib/libgalera_smm.so(_ZN6galera13ReplicatorSMM10pre_commitEPNS_9TrxHandleEP14wsrep_trx_meta+0x59)[0x7fa4c32c2da9]
/usr/lib/libgalera_smm.so(galera_pre_commit+0x140)[0x7fa4c32d3750]
/usr/sbin/mysqld(_Z22wsrep_run_wsrep_commitP3THDP10handlertonb+0x576)[0x7949b6]
/usr/sbin/mysqld(_Z14ha_prepare_lowP3THDb+0x84)[0x5e0514]
/usr/sbin/mysqld(_Z15ha_commit_transP3THDbb+0x1f8)[0x5dfa28]
/usr/sbin/mysqld(_Z12trans_commitP3THD+0x4a)[0x77b11a]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2d1e)[0x6fa9ce]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x5c8)[0x6fd7f8]
/usr/sbin/mysqld[0x6fd8f2]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x132f)[0x6ff1ef]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x202)[0x6ffbe2]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x2ad)[0x6d11dd]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6d1262]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0xb195d0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7fa4eb38ab50]
/lib/x86_...

[xamzab (xamzab)]

Got another malloc crash log:

mysqld: malloc.c:5002: _int_free: Assertion `nextchunk->bk_nextsize->fd_nextsize == nextchunk' failed.
18:15:18 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=154
max_threads=502
thread_count=91
connection_count=82
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 331448 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f041c2fe520
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f0434341e50 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ccd9e]
/usr/sbin/mysqld(handle_fatal_signal+0x36c)[0x6828dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f04febb20a0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f04fcdec165]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x180)[0x7f04fcdef3e0]
/lib/x86_64-linux-gnu/libc.so.6(+0x75dea)[0x7f04fce2fdea]
/lib/x86_64-linux-gnu/libc.so.6(+0x77753)[0x7f04fce31753]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f04fce3498c]
/usr/sbin/mysqld(_ZN3THDD1Ev+0x2f4)[0x6cb8c4]
/usr/sbin/mysqld(_ZN3THDD0Ev+0x11)[0x6cba91]
/usr/sbin/mysqld(_Z29one_thread_per_connection_endP3THDb+0xc8)[0x5c3ef8]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0xf0)[0x6d1020]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6d1262]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0xb195d0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f04feba9b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f04fce9595d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 269842
Status: KILL_CONNECTION

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
150429 20:15:19 mysqld_safe Number of processes running now: 0
150429 20:15:19 mysqld_safe WSREP: not restarting wsrep node automatically
150429 20:15:19 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

[Krunal Bauskar (krunal-bauskar)]

All you crash are related to memory allocation problem.

Including the comment#27 crash M_insert_EPKSt18_Rb_tree_node_base that fails to allocate memory during insert of entry to stl map.

Generic crashes seems to be related to MySQL inability to handle malloc failure but that is not a common case.

I would suggest you re-check all your setting and see where things are being overallocated.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-527