xtrabackup got signal 11

[Andres G. Aragoneses]

(Not sure this is a duplicate of bug https://bugs.launchpad.net/percona-xtrabackup/+bug/1658073 , but I guess not, because that one seems to hint there's an easy and deterministic way to reproduce it if you do certain steps; however, our backups have started receiving this crash even when we haven't changed anything in the way they are done/invoked.)

The log we got is:

170516 10:02:19 >> log scanned up to (15990469686)
170516 10:02:20 >> log scanned up to (15990469991)
170516 10:02:21 >> log scanned up to (15990471865)
170516 10:02:22 [01] ...done
10:02:22 UTC - xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
innobackupex(my_print_stacktrace+0x3b)[0xc7183b]
innobackupex(handle_fatal_signal+0x281)[0xa8da01]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0)[0x7fdc004673d0]
innobackupex[0x725bd5]
innobackupex[0x711e39]
innobackupex[0x719558]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa)[0x7fdc0045d6fa]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fdbfebb1b5d]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup

We're using Ubuntu16.04 LTS server with the following mysql packages:

$ dpkg --list | grep mysql
ii libdbd-mysql-perl 4.033-1ubuntu0.1 amd64 Perl5 database interface to the MySQL database
ii libmysqlclient20:amd64 5.7.17-0ubuntu0.16.04.1 amd64 MySQL database client library
ii mysql-cimprov 1.0.1.5 amd64 Microsoft CIM management interface for MySQL Server
ii mysql-client 5.7.17-0ubuntu0.16.04.1 all MySQL database client (metapackage depending on the latest version)
ii mysql-client-5.7 5.7.17-0ubuntu0.16.04.1 amd64 MySQL database client binaries
ii mysql-client-core-5.7 5.7.17-0ubuntu0.16.04.1 amd64 MySQL database core client binaries
ii mysql-common 5.7.17-0ubuntu0.16.04.1 all MySQL database common files, e.g. /etc/mysql/my.cnf
ii mysql-server 5.7.17-0ubuntu0.16.04.1 all MySQL database server (metapackage depending on the latest version)
ii mysql-server-5.7 5.7.13-0ubuntu0.16.04.2 amd64 MySQL database server binaries and system database setup
ii mysql-server-core-5.7 5.7.13-0ubuntu0.16.04.2 amd64 MySQL database server binaries

$ dpkg --list | grep percon
ii percona-xtrabackup-24 2.4.4-1.xenial amd64 Open source backup tool for InnoDB and XtraDB

[Sergei Glushchenko (sergei.glushchenko)]

Can you please provide the command line you are using to take a backup and your my.cnf?

[Sergei Glushchenko (sergei.glushchenko)]

Crash in the bug you mentioned happens during prepare, yours seems to during the backup, so they are unlikely the same.

[Andres G. Aragoneses (knocte)]

$ cat /etc/mysql/my.cnf
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/

$ cat /etc/mysql/conf.d/*.cnf
[mysql]
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
# http://dev.mysql.com/doc/refman/5.7/en/replication-setup-slaves.html#replication-howto-slaveinit
[mysqld]
server-id=2
slave-sql-verify-checksum=1
log-bin=gtc-slave-bin
binlog-format=MIXED
relay-log=gtc-relay-bin

[mysqld]
read-only=1

$ cat /etc/mysql/mysql.conf.d/*.*
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer_size = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover-options = BACKUP
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#log_slow_queries = /va...

[Andres G. Aragoneses (knocte)]

> Can you please provide the command line you are using to take a backup

innobackupex --user={user} --password={password} --compress --stream=xbstream --incremental {dir_where_to_leave_backup} --incremental-lsn={lsn} > {dest_file}

Sergei, should I try upgrading to a new version first?

[Sergei Glushchenko (sergei.glushchenko)]

Sure, it makes sense to upgrade. I don't see anything unusual and the stack trace unfortunately doesn't give much information. Does xtrabackup crash like this all the time or just occasionally?

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1443