Percona XtraDB crash on daily basis
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Percona Server moved to https://jira.percona.com/projects/PS | ||||||
5.5 |
New
|
Undecided
|
Unassigned | |||
5.6 |
New
|
Undecided
|
Unassigned | |||
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC |
Incomplete
|
Undecided
|
Unassigned |
Bug Description
Hello,
I made a Percona XtraDB Cluster with 3 same-hardware nodes and having one node crashing each day...
Here is a few lines of log:
====
2015-03-25 09:22:38 4602 [Warning] Client failed to provide its character set. 'latin1' will be used as client character set.
2015-03-25 09:22:39 4602 [Warning] Client failed to provide its character set. 'latin1' will be used as client character set.
08:22:39 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=10002
thread_count=57
connection_count=55
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7ffbb4177c80
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ffbeeff9e50 thread_stack 0x1000000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 442848
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
150325 09:22:40 mysqld_safe Number of processes running now: 0
150325 09:22:40 mysqld_safe WSREP: not restarting wsrep node automatically
150325 09:22:40 mysqld_safe mysqld from pid file /var/run/
===
Hopes to find the issue to get a secure DB Cluster soon.
Thanks in advance.
Valerii Kravchuk (valerii-kravchuk) wrote : | #1 |
Changed in percona-server: | |
status: | New → Incomplete |
xamzab (xamzab) wrote : | #2 |
- MySQL config file Edit (4.0 KiB, text/plain)
Here is some information:
dpkg -l|grep percona
ii percona-xtrabackup 2.2.9-5067-1.wheezy amd64 Open source backup tool for InnoDB and XtraDB
ii percona-
ii percona-
/my.cnf)
ii percona-
/my.cnf)
ii percona-
ii percona-
ii percona-
uname -a
Linux sql1 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u1 x86_64 GNU/Linux
xamzab (xamzab) wrote : | #3 |
Changed in percona-server: | |
status: | Incomplete → New |
Valerii Kravchuk (valerii-kravchuk) wrote : | #4 |
I see at least two potential problems (for a server in VM with <6G of memory) in my.cnf:
max_connections = 10000
query_cache_size = 1G
Can you, please, check if with max_connections of, say, 500, and query_cache_size of 32M you still have crashes?
It would be useful to the output of SHOW GLOBAL STATUS after having this node running for some time. For now even pt-summary is provided while mysqld process is NOT running, so we can not even see how "close" to out of memory condition it may be under real load.
Changed in percona-server: | |
status: | New → Incomplete |
xamzab (xamzab) wrote : | #5 |
Applied your recommendations and waiting for some hours to run again pt-summary.
xamzab (xamzab) wrote : | #6 |
xamzab (xamzab) wrote : | #7 |
Hi,
One node has crashed a few minutes ago. It looks like the VM has just stopped. Restarted it to see if this happens again. Will let you know about what I found.
xamzab (xamzab) wrote : | #8 |
The Percona node has crashed again.
Here is what we got:
04:27:45 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=502
thread_count=71
connection_count=69
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f36ec29bde0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f36d9fece50 thread_stack 0x1000000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (38df1e0): [SQL QUERY REMOVED]
Connection ID (thread ID): 219313
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
150329 06:27:46 mysqld_safe Number of processes running now: 0
150329 06:27:46 mysqld_safe WSREP: not restarting wsrep node automatically
150329 06:27:46 mysqld_safe mysqld from pid file /var/run/
xamzab (xamzab) wrote : | #9 |
Here is another crash log from another crashed node.
13:01:14 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=502
thread_count=72
connection_count=70
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fc280387050
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fc276ff5e50 thread_stack 0x1000000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (48e6b20): [SQL QUERY REMOVED]
Connection ID (thread ID): 1372718
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
150329 15:01:15 mysqld_safe Number of processes running now: 0
150329 15:01:15 mysqld_safe WSREP: not restarting wsrep node automatically
150329 15:01:15 mysqld_safe mysqld from pid file /var/run/
Changed in percona-server: | |
status: | Incomplete → New |
xamzab (xamzab) wrote : | #10 |
Another crash log ...
16:01:05 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=502
thread_count=55
connection_count=53
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f42dc2d0470
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f4316fe9e50 thread_stack 0x1000000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 1018029
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
150331 18:01:06 mysqld_safe Number of processes running now: 0
150331 18:01:06 mysqld_safe WSREP: not restarting wsrep node automatically
150331 18:01:06 mysqld_safe mysqld from pid file /var/run/
xamzab (xamzab) wrote : | #11 |
Here is another crash log from "Bootstraped" PXC node.
02:20:08 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=502
thread_count=70
connection_count=68
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f51d43188d0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f5197ff6e50 thread_stack 0x1000000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/lib/x86_
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 1524272
Status: KILL_CONNECTION
You may download the Percona XtraDB Cluster operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
150404 04:20:09 mysqld_safe Number of processes running now: 0
150404 04:20:09 mysqld_safe WSREP: not restarting wsrep node automatically
150404 04:20:09 mysqld_safe mysqld from pid file /var/run/
xamzab (xamzab) wrote : | #12 |
Just after restarted the Primary PXC node, another node crashed a few minutes after.
Here is crash log
15:30:35 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=502
thread_count=59
connection_count=57
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f82081da690
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f8228e8fe50 thread_stack 0x1000000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f81e03104d0): is an invalid pointer
Connection ID (thread ID): 2459594
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
150404 17:30:35 mysqld_safe Number of processes running now: 0
150404 17:30:35 mysqld_safe WSREP: not restarting wsrep node automatically
150404 17:30:35...
xamzab (xamzab) wrote : | #13 |
Here is another crash log:
02:41:01 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=502
thread_count=64
connection_count=62
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7ff25035cfd0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ff280ffbe50 thread_stack 0x1000000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (60fdbb0): SELECT username, pptp_password, trial_mode FROM users WHERE pptp_status = 1 AND inscrit = 1
Connection ID (thread ID): 653286
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
150409 04:41:01 mysqld_safe Number of processes running now: 0
150409 04:41:01 mysqld_safe WSREP: not restarting wsrep node automatically
150409 04:41:01 mysqld_safe mysqld from pid file /var/run/
xamzab (xamzab) wrote : | #14 |
Here is another different crash log:
08:10:02 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=502
thread_count=141
connection_
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f90d811f650
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f9129ff0e50 thread_stack 0x1000000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (67d5780): SELECT u.email, u.title, u.url, u.location, u.signature, u.email_setting, u.use_pm, u.num_posts, u.registered, u.admin_note, p.id...
xamzab (xamzab) wrote : | #15 |
Reinstalled all nodes from scratch and here is what I've got:
11:27:58 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=502
thread_count=61
connection_count=59
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f2b742af5b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f2b797f3e50 thread_stack 0x1000000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
_S7_yP13select_
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (3ac8dc0): SELECT COUNT(*) AS var1 FROM my_table WHERE my_field = 'my_value'
Connection ID (thread ID): 16968
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
150412 13:27:58 mysqld_safe Number of processes running now: 0
150412 13:27:58 mysqld_safe WSREP: not restarting wsrep node automatically
150412 13:27:58 mysq...
xamzab (xamzab) wrote : | #16 |
Another crash log:
13:00:14 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=502
thread_count=49
connection_count=47
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x1000000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/usr/lib/
/usr/sbin/
/usr/sbin/
/lib/x86_
/usr/sbin/
You may download the Percona XtraDB Cluster operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
150416 15:00:14 mysqld_safe Number of processes running now: 0
150416 15:00:14 mysqld_safe WSREP: not restarting wsrep node automatically
150416 15:00:14 mysqld_safe mysqld from pid file /var/run/
Valerii Kravchuk (valerii-kravchuk) wrote : | #17 |
You had different crashes, but it seems they all were related to the inability of malloc to allocate the memory:
stack_bottom = 7f2b797f3e50 thread_stack 0x1000000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
_S7_yP13select_
/usr/sbin/
/usr/sbin/
...
I wonder how much RAM do you have on host box and how many memory is allocated to all VMs running there in total, as I suspect these failures may be related to memory overcommit for your VmWare.
xamzab (xamzab) wrote : | #18 |
Hello,
VMware Host has 8GB of RAM, and contains only one VM which is the MySQL's VM. This config is replicated 3 times as there is 3 nodes in that PXC.
Would you want me to reduce allocated RAM to something such as 4GB instead of 6GB actually ?
Regards.
Valerii Kravchuk (valerii-kravchuk) wrote : | #19 |
So, you have 3 hardware boxes with 8G of RAM each, and just one VM with 6GB of RAM allocated on each of the boxes? If this is the case, maybe allocate more memory, not less per VM, 7GB or so and check if this helps to have less frequent crashes.
xamzab (xamzab) wrote : | #20 |
Updated configurations of VMs to 7GB of RAM. Let's see what happens.
xamzab (xamzab) wrote : | #21 |
It keeps crashing. What else to do ?
Valerii Kravchuk (valerii-kravchuk) wrote : | #22 |
Try to comment out these suspicious settings in your my.cnf:
# thread_stack = 16M
# join_buffer_size = 8M
# max_binlog_size = 8GM
then restart the node and check if this changes anything.
xamzab (xamzab) wrote : | #23 |
Configuration updated. Monitoring the cluster.
xamzab (xamzab) wrote : | #24 |
The uptime was a bit longer than usual but the node crashed again.
Here is crashing log:
08:10:03 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=502
thread_count=92
connection_count=83
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x20b39d0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f1fd03c9e50 thread_stack 0x40000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f1fbc31ed9e): is an invalid pointer
Connection ID (thread ID): 394517
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
150424 10:10:03 mysqld_safe Number of processes running now: 0
150424 10:10:03 mysqld_safe WSREP: not restarting wsrep node automatically
150424 10:10:03 mysqld_safe mysqld from pid file /var/run/
Valerii Kravchuk (valerii-kravchuk) wrote : | #25 |
So, again crash for my_malloc with some new stack trace...
Can you, please, check dmest and /var/log/messages for anything referring to mysql around the moment of crash?
xamzab (xamzab) wrote : | #26 |
That's something I've already tried but there is/was nothing in dmesg/messages/
If there is anything to enable/update in order to get more information from these crashes, then feel free to tell me what to do.
xamzab (xamzab) wrote : | #27 |
Got a new crash log after changed some settings such as innodb buffer size.
[CODE]
21:10:03 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=502
thread_count=77
connection_count=68
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fa3cc2d8e20
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fa3d6ef1e50 thread_stack 0x40000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_...
xamzab (xamzab) wrote : | #28 |
Got another malloc crash log:
mysqld: malloc.c:5002: _int_free: Assertion `nextchunk-
18:15:18 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=502
thread_count=91
connection_count=82
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f041c2fe520
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f0434341e50 thread_stack 0x40000
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
/lib/x86_
/lib/x86_
/lib/x86_
/lib/x86_
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/x86_
/lib/x86_
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 269842
Status: KILL_CONNECTION
You may download the Percona XtraDB Cluster operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
150429 20:15:19 mysqld_safe Number of processes running now: 0
150429 20:15:19 mysqld_safe WSREP: not restarting wsrep node automatically
150429 20:15:19 mysqld_safe mysqld from pid file /var/run/
affects: | percona-server → percona-xtradb-cluster |
Krunal Bauskar (krunal-bauskar) wrote : | #29 |
All you crash are related to memory allocation problem.
Including the comment#27 crash M_insert_
Generic crashes seems to be related to MySQL inability to handle malloc failure but that is not a common case.
I would suggest you re-check all your setting and see where things are being overallocated.
Changed in percona-xtradb-cluster: | |
status: | New → Incomplete |
Shahriyar Rzayev (rzayev-sehriyar) wrote : | #30 |
Percona now uses JIRA for bug reports so this bug report is migrated to: https:/
What exact version of Percona XtraDB Cluster, 5.x.y-..., do you use?
Please, send my.cnf file content and the output of pt-summary script from the crashing node.