mysqld got signal 11 during innobackupex

[Nobuto Murata]

ii percona-galera-3 3.8-3390-0ubuntu6 amd64 Galera replication framework for Percona XtraDB Cluster
ii percona-xtrabackup 2.2.3-2.1build1.1 amd64 Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-server-5.6 5.6.21-25.8-0ubuntu3.2 amd64 Percona XtraDB Cluster database server binaries

percona-cluter charm deployment stopped with an error and mysql daemon did not accept any read/write. So I found mysqld crashed during the cluster deployment.

[/var/log/mysql/error.log]
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/mysql/my.cnf --no-version-check $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2>${DATA}/innobackup.backup.log | socat -u stdio TCP:10.0.0.96:4444; RC=( ${PIPESTATUS[@]} ) (20161202 10:30:51.554)
10:30:56 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=33554432
read_buffer_size=131072
max_used_connections=78
max_threads=20002
thread_count=39
connection_count=37
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 8017301 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x53a8f40
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fc4a5797e58 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x9176db]
/usr/sbin/mysqld(handle_fatal_signal+0x378)[0x663418]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x113e0)[0x7fc65a1de3e0]
[0x7fc3f80000d8]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 192
Status: KILL_CONNECTION

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
WSREP_SST: [ERROR] innobackupex finished with error: 9. Check /var/lib/percona-xtradb-cluster//innobackup.backup.log (20161202 10:30:56.012)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Nobuto Murata (nobuto)]

[/var/lib/percona-xtradb-cluster/innobackup.backup.log]
161202 10:30:54 innobackupex: Continuing after ibbackup has suspended
161202 10:30:54 innobackupex: Executing LOCK TABLES FOR BACKUP...
161202 10:30:54 innobackupex: Backup tables lock acquired

161202 10:30:54 innobackupex: Starting to backup non-InnoDB tables and files
innobackupex: in subdirectories of '/var/lib/percona-xtradb-cluster'
innobackupex: Backing up files '/var/lib/percona-xtradb-cluster/neutron/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (179 files)
>> log scanned up to (9868171)
innobackupex: Backing up files '/var/lib/percona-xtradb-cluster/heat/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (17 files)
innobackupex: Backing up files '/var/lib/percona-xtradb-cluster/glance/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (21 files)
innobackupex: Backing up file '/var/lib/percona-xtradb-cluster/cinder/db.opt'
innobackupex: Backing up files '/var/lib/percona-xtradb-cluster/performance_schema/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (53 files)
innobackupex: Backing up file '/var/lib/percona-xtradb-cluster/keystone/db.opt'
innobackupex: Backing up files '/var/lib/percona-xtradb-cluster/mysql/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (74 files)
>> log scanned up to (9868171)
innobackupex: Backing up file '/var/lib/percona-xtradb-cluster/aodh/db.opt'
161202 10:30:55 innobackupex: Finished backing up non-InnoDB tables and files

161202 10:30:56 innobackupex: Executing LOCK BINLOG FOR BACKUP...
DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction at /usr//bin/innobackupex line 3059.
innobackupex: Error:
Error executing 'LOCK BINLOG FOR BACKUP': DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction at /usr//bin/innobackupex line 3059.
161202 10:30:56 innobackupex: Waiting for ibbackup (pid=56969) to finish

[Nobuto Murata (nobuto)]

With a quick googling LP: #1401133 might be related:
https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1401133
> There are 2 parts to this bug report:
>
> 1. Deadlock is triggered by the LOCK BINLOG FOR BACKUP statement
> executed by XtraBackup
> 2. The XtraBackup connection is closed due to the above error while
> holding a backup (TABLE) lock. Which results in a heap corruption and a
> server crash, possibly some time later.
>
> I believe #2 has been reported and fixed in PS 5.6.24-72.2 with
> https://github.com/percona/percona-server/pull/26
>
> Which means the latest PXC release 5.6.24-25.11 should not crash when
> backup locks are used by XtraBackup if my
> assumption is correct, but XtraBackup may still fail with an error due
> to #1.
>
> #1 (as in, a deadlock with LOCK BINLOG FOR BACKUP) looks like a
> PXC-specific issue, but so far I’ve been unable to reproduce it
> neither with PXC 5.6.22-25.8 nor with PXC 5.6.24-25.11.

[Nobuto Murata (nobuto)]

A quick workaround on charm deployment would be using sst-method other than innobackupex such as sst-method=rsync for example.

[James Page (james-page)]

I don't think this is a charm specific problem, so marking the charm task as invalid - lets track this via the Ubuntu distro task only rather than double accounting.

[James Page (james-page)]

Nobuto

I'm prepping updates for PXC in:

  https://launchpad.net/~james-page/+archive/ubuntu/percona

nice to get your feedback on the proposed packages to see if they resolve this issue for you - the updates include new pxc, percona-xtrabackup and percona-galera packages.