The problem with Node after using innobackupex and "Backup Locks"
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
MySQL patches by Codership |
New
|
Undecided
|
Unassigned | |||
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC | Status tracked in 5.6 | |||||
5.5 |
Invalid
|
Undecided
|
Unassigned | |||
5.6 |
Fix Released
|
High
|
Unassigned |
Bug Description
Good evening, dear colleagues.
After the upgrade of PXC to the latest version, we had some problems with innobackupex and "Backup locks".
Current MySQL server version: "Server version: 5.6.21-70.1-56-log Percona XtraDB Cluster (GPL), Release rel70.1, Revision 938, WSREP version 25.8, wsrep_25.8.r4150"
Before the upgrade: "Server version: 5.6.20-68.0-56-log Percona XtraDB Cluster (GPL), Release rel68.0, Revision 888, WSREP version 25.7, wsrep_25.7.r4126"
OS: Scientific Linux release 6.6 (Carbon)
The number of nodes in the cluster: 3
While using the tool "innobackupex", errors occur during the locks and the node is crashed.
Trying to run innobackupex:
[root@natrium mysql]# innobackupex --parallel=2 /mnt/xtrabackup/
.......
.......
>> log scanned up to (2892514252116)
>> log scanned up to (2892514252116)
[01] ...done
xtrabackup: Creating suspend file '/mnt/xtrabacku
>> log scanned up to (2892514252116)
141210 12:36:34 innobackupex: Continuing after ibbackup has suspended
141210 12:36:34 innobackupex: Executing LOCK TABLES FOR BACKUP...
DBD::mysql::db do failed: MySQL server has gone away at /usr/bin/
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/
innobackupex: Error:
Error executing 'LOCK TABLES FOR BACKUP': DBD::mysql::db do failed: MySQL server has gone away at /usr/bin/
141210 12:36:34 innobackupex: Waiting for ibbackup (pid=24638) to finish
Node immediately crashes. Logs:
10:07:54 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=2050
thread_count=386
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/
/usr/sbin/
/lib64/
/usr/lib64/
/usr/lib64/
/usr/lib64/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib64/
/usr/sbin/
You may download the Percona XtraDB Cluster operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
141210 12:07:55 mysqld_safe Number of processes running now: 0
141210 12:07:55 mysqld_safe WSREP: not restarting wsrep node automatically
141210 12:07:55 mysqld_safe mysqld from pid file /var/lib/
I restarted MySQL (using IST method for recovery) and tried again:
[root@natrium mysql]# innobackupex --parallel=2 /mnt/xtrabackup/
.......
.......
xtrabackup: Creating suspend file '/mnt/xtrabacku
141210 14:34:31 innobackupex: Continuing after ibbackup has suspended
141210 14:34:31 innobackupex: Executing LOCK TABLES FOR BACKUP...
141210 14:34:31 innobackupex: Backup tables lock acquired
141210 14:34:31 innobackupex: Starting to backup non-InnoDB tables and files
innobackupex: in subdirectories of '/var/lib/mysql/'
innobackupex: Backing up files '/var/lib/
>> log scanned up to (2895378455386)
innobackupex: Backing up files '/var/lib/
innobackupex: Backing up file '/var/lib/
innobackupex: Backing up files '/var/lib/
innobackupex: Backing up files '/var/lib/
>> log scanned up to (2895378479147)
>> log scanned up to (2895378479640)
innobackupex: Backing up file '/var/lib/
innobackupex: Backing up file '/var/lib/
innobackupex: Backing up file '/var/lib/
innobackupex: Backing up files '/var/lib/
innobackupex: Backing up file '/var/lib/
innobackupex: Backing up file '/var/lib/
innobackupex: Backing up file '/var/lib/
innobackupex: Backing up file '/var/lib/
innobackupex: Backing up file '/var/lib/
innobackupex: Backing up file '/var/lib/
141210 14:34:34 innobackupex: Finished backing up non-InnoDB tables and files
141210 14:34:34 innobackupex: Executing LOCK BINLOG FOR BACKUP...
DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction at /usr/bin/
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/
innobackupex: Error:
Error executing 'LOCK BINLOG FOR BACKUP': DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction at /usr/bin/
141210 14:34:34 innobackupex: Waiting for ibbackup (pid=10719) to finish
After the second attempt, the error "deadlock" occurred. Server worked for another 15 minutes and crashed again. Logs:
12:55:01 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https:/
key_buffer_
read_buffer_
max_used_
max_threads=2050
thread_count=385
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/
/usr/sbin/
/lib64/
/usr/lib64/
/usr/lib64/
/usr/lib64/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib64/
/usr/sbin/
You may download the Percona XtraDB Cluster operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
141210 14:55:01 mysqld_safe Number of processes running now: 0
141210 14:55:01 mysqld_safe WSREP: not restarting wsrep node automatically
141210 14:55:01 mysqld_safe mysqld from pid file /var/lib/
Prior to the last update such problems did not arise (previously used "FLUSH TABLES WITH READ LOCK").
All packets installed from Percona RPM repository:
[root@natrium /]# rpm -qa | grep Percona
Percona-
Percona-
Percona-
Percona-
Percona-
Percona-
Percona-
Percona-
Percona-
Percona-
All databases summary size: 256G
We also use partitioning.
/etc/my.cnf:
[mysqld_safe]
open-files-limit = 120000
malloc-
[mysqld]
skip-name-resolve
user = mysql
bind-address = 10.10.91.4
port = 3306
max_connections = 2048
datadir = /var/lib/mysql
socket = /var/lib/
tmpdir = /tmp/mysql
symbolic-links = 0
table_open_cache = 8192
table_definitio
table_open_
thread_cache_size = 64
default_
explicit_
ft_min_word_len = 3
large-pages
slow_query_log = On
slow_launch_time = 5
general_log_file = '/var/log/
general_log = Off
query_cache_size = 0
query_cache_type = off
innodb_
innodb_
innodb_
innodb_
innodb_
innodb_
innodb_
innodb_
innodb_
innodb_open_files = 8192
innodb_
innodb_
innodb_
innodb_doublewrite = 1
innodb_flush_method = O_DIRECT
innodb_
innodb_
innodb_io_capacity = 40000
innodb_
innodb_
innodb_
innodb_
innodb_
innodb_support_xa = 0
innodb_
innodb_
innodb_
innodb_
log-bin = /var/lib/
max_binlog_size = 1024M
binlog_format = ROW
binlog_cache_size = 5M
max_binlog_files = 10
expire_logs_days = 0
sync_binlog = 0
relay_log = /var/lib/
slave_load_tmpdir = /tmp/mysql
server-id = 12
skip-slave-start
log_slave_updates = On
log_error = "/var/log/
slow_query_log_file = "/var/log/
wsrep_provider=
wsrep_provider_
wsrep_node_
wsrep_cluster_
wsrep_node_
wsrep_sst_method = xtrabackup-v2
wsrep_sst_auth = sst_xtrabackup:
wsrep_notify_cmd = '/usr/local/
wsrep_replicate
wsrep_forced_
wsrep_log_conflicts = Off
wsrep_auto_
wsrep_retry_
wsrep_slave_threads = 128
wsrep_convert_
wsrep_max_ws_size = 2147483648 # 2G
wsrep_max_ws_rows = 1048576 # 1M
tags: | added: pxc |
tags: | added: crash innobackupex sst xtrabackup |
tags: | added: backup-locks |
@Aleksey,
Thanks for reporting. Would it be possible to provide 'thread apply all bt' 'bt' 'thread apply all bt full' from a coredump?
You can use https:/ /gist.github. com/ronin13/ 8d403c2f88826fc c7aba to generate traces.