Assertion failure in xtrabackup.cc: lsn_chosen == 1

[Ceri WIlliams]

Repeated, but intermittent assertion failures have been seen on a box running Debian 8 with PXB 2.3.3-1.jessie. This was also noticed in 2.2.3-2.1:

/usr/bin/innobackupex version 2.3.2 - line 2494
/usr/bin/innobackupex version 2.3.3 - line 2503

The stracktrace shows as:
---------------------------------------

2016-04-08 04:21:45 7fb1bbff9740 InnoDB: Assertion failure in thread 140401340028736 in file xtrabackup.cc line 2503
InnoDB: Failing assertion: lsn_chosen == 1
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
02:21:45 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
/usr/bin/innobackupex(my_print_stacktrace+0x2e) [0x88857e]
/usr/bin/innobackupex(handle_fatal_signal+0x261) [0x7d99c1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0) [0x7fb1bbbde8d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7fb1ba2eb067]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7fb1ba2ec448]
/usr/bin/innobackupex() [0x58976d]
/usr/bin/innobackupex(xtrabackup_backup_func()+0x127f) [0x59125f]
/usr/bin/innobackupex(main+0xa4b) [0x577fcb]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fb1ba2d7b45]
/usr/bin/innobackupex() [0x588afa]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup
---------------------------------------

The error has been observed in both full and incremental backups, both of which were streaming via SSH:

ssh -t -o PasswordAuthentication=no -o StrictHostKeyChecking=no server.a "
sudo /usr/bin/innobackupex xtrabackup_tmpdir \
  --slave-info --kill-long-queries-timeout=10 \
  --kill-long-query-type=select --incremental-lsn=xxxxx \
  --incremental --compress --stream=xbstream \
| ssh -o PasswordAuthentication=no -o StrictHostKeyChecking=no server.b 'xbstream -x -C xtrabackup_tmpdir'"

[Shahriyar Rzayev (rzayev-sehriyar)]

# Full my.cnf of source server

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = -10

[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking

bind-address = 0.0.0.0
max_connect_errors = 5000

server-id = 3266239
auto-increment-increment = 1
auto-increment-offset = 1

## TEMP!!!
#binlog-format=MIXED

# MyISAM #
key-buffer-size = 64M
myisam-recover = FORCE,BACKUP

# SAFETY #
max-allowed-packet = 16M
max-connect-errors = 1000000
sysdate-is-now = 1
innodb = FORCE
innodb-strict-mode = 0

# BINARY LOGGING #
log-bin = /var/lib/mysql/mysql-bin
#log-slave-updates = ON
expire-logs-days = 3
max_binlog_size = 750M
sync-binlog = 0 # http://dev.mysql.com/doc/refman/5.0/en/binary-log.html

# CACHES AND LIMITS #
tmp-table-size = 256M
max-heap-table-size = 256M
query-cache-type = 0
query-cache-size = 0
max-connections = 10000
thread-cache-size = 200
open-files-limit = 65535
table-definition-cache = 12000
table-open-cache = 12000

#
# * per-Tread Buffer
#
read_buffer_size = 256K
read_rnd_buffer_size = 256K
join_buffer_size = 512K
sort_buffer_size = 512K
thread_stack = 256K

# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 4G
innodb-file-per-table = 1
innodb_concurrency_tickets = 1250
innodb_buffer_pool_instances = 8
innodb_write_io_threads = 6
innodb_lock_wait_timeout = 10
innodb_io_capacity = 1000 # http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_io_capacity
innodb_flush_log_at_trx_commit = 0 # http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit
innodb_old_blocks_time = 1000

innodb-buffer-pool-size = 80G
innodb_file_format_max = Barracuda
innodb_file_format = Barracuda

# PERCONA SERVER SPECIFIC #
innodb_adaptive_flushing_method = keep_average
innodb_adaptive_hash_index_partitions = 16
innodb_buffer_pool_restore_at_startup = 1000
innodb_flush_neighbor_pages = cont

# LOGGING #
log-error = /var/lib/mysql/mysql-error.log
log-queries-not-using-indexes = 0
long_query_time = 2
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log

[mysqldump]
quick
quote-names
max_allowed_packet = 32M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer = 6G

#
# * IMPORTANT: Additional settings that c...

[Shahriyar Rzayev (rzayev-sehriyar)]

Target server is -> 5.5.45-37.4-log Percona Server (GPL), Release 37.4, Revision 042e02b

[Sergei Glushchenko (sergei.glushchenko)]

https://github.com/percona/percona-xtrabackup/pull/234
https://github.com/percona/percona-xtrabackup/pull/235

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-473