Xtrabackup: "InnoDB: File (unknown): 'read' returned OS error 71. Cannot continue operation"

[rockandska]

Hello,

I encounter a problem when i try to backup my server and never seen this before with my previous backups.

xtrabackup version:
innobackupex --version
innobackupex version 2.3.4 Linux (x86_64) (revision id: e80c779)

Mysql version:
aptitude show mariadb-server
Package: mariadb-server
State: installed
Version: 5.5.39+maria-1~wheezy

Command launch:
innobackupex --stream=xbstream --parallel=10 . 2> log_innobackupex | ssh xxxx@xxxxxxx "xbstream -x -C /home/xxxxx/BACKUP"

Log:
cat log_innobackupex
160513 18:34:14 innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

160513 18:34:14 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;port=3306;mysql_socket=/var/run/mysqld/mysqld.sock' as 'root' (using password: YES).
160513 18:34:14 version_check Connected to MySQL server
160513 18:34:14 version_check Executing a version check against the server...
160513 18:34:14 version_check Done.
160513 18:34:14 Connecting to MySQL server host: localhost, user: root, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
Using server version 5.5.39-MariaDB-1~wheezy
innobackupex version 2.3.4 based on MySQL server 5.6.24 Linux (x86_64) (revision id: e80c779)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /home/mysql/db
xtrabackup: open files limit requested 65000, set to 800000
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = /home/mysql/redo
xtrabackup: innodb_log_files_in_group = 8
xtrabackup: innodb_log_file_size = 1073741824
xtrabackup: using O_DIRECT
160513 18:34:14 >> log scanned up to (121091816410225)
xtrabackup: Generating a list of tablespaces
InnoDB: Retry attempts for reading partial data failed.
InnoDB: Tried to read 16384 bytes at offset 0. Was only able to read 0.
2016-05-13 18:34:15 7f270e484720 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: File (unknown): 'read' returned OS error 71. Cannot continue operation
2016-05-13 18:34:15 7f270e484720 InnoDB: Assertion failure in thread 139805720069920 in file os0file.cc line 658
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
16:34:15 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
innobackupex(my_print_stacktrace+0x2b) [0x8da73b]
innobackupex(handle_fatal_signal+0x252) [0x881ec2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0) [0x7f270e0660a0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f270c682165]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7f270c6853e0]
innobackupex() [0x6e1fb5]
innobackupex(os_file_read_func(int, void*, unsigned long, unsigned long)+0x80) [0x6e3cb0]
innobackupex(fil_read_first_page(int, unsigned long, unsigned long*, unsigned long*, unsigned long*, unsigned long*)+0x6a) [0x741bda]
innobackupex() [0x74893a]
innobackupex(fil_load_single_table_tablespaces(unsigned long (*)(char const*, char const*))+0xa27) [0x7498d7]
innobackupex() [0x5d9bf1]
innobackupex(xtrabackup_backup_func()+0xa52) [0x5dded2]
innobackupex(main+0x21a5) [0x5c6f95]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f270c66eead]
innobackupex() [0x5d8051]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup

Nothing suspicious in dmesg /kern.log about disk

Mysql dir_size ~= 9.7To
Disk space available ~= 1.3To

Best regards,

[Shahriyar Rzayev (rzayev-sehriyar)]

Could not reproduce using:

root@debian:~# mysql -u root -pBaku12345# -e "select @@version"
+-------------------------+
| @@version |
+-------------------------+
| 5.5.49-MariaDB-1~wheezy |
+-------------------------+

root@debian:~# xtrabackup --version
xtrabackup version 2.3.4 based on MySQL server 5.6.24 Linux (x86_64) (revision id: e80c779)

With default my.cnf.

Could you please share your configuration?

So could you please share my.cnf file?

[rockandska (y-panier)]

Sorry I was on holidays

here is my config:

egrep -v "^#" /etc/mysql/my.cnf

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /home/mysql/db
tmpdir = /home/mysql/tmp
log_error = /home/mysql/log/mysqld
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
skip-external-locking
max_connections = 200
connect_timeout = 5
wait_timeout = 600
max_allowed_packet = 16M
thread_cache_size = 128
sort_buffer_size = 4M
bulk_insert_buffer_size = 16M
tmp_table_size = 32M
max_heap_table_size = 32M
myisam_recover = BACKUP
key_buffer_size = 2G
open-files-limit = 65000
table_open_cache = 8000
myisam_sort_buffer_size = 2G
concurrent_insert = 2
read_buffer_size = 2M
read_rnd_buffer_size = 1M
query_cache_limit = 128K
query_cache_size = 64M
log_warnings = 2
slow_query_log_file = /home/mysql/log/mariadb-slow.log
long_query_time = 10
log_slow_verbosity = query_plan

default_storage_engine = InnoDB
innodb_log_file_size = 1G
innodb_log_files_in_group = 8
innodb_buffer_pool_size = 32G
innodb_log_buffer_size = 16M
innodb_file_per_table = 1
innodb_open_files = 400
innodb_io_capacity = 400
innodb_flush_method = O_DIRECT

innodb_thread_concurrency = 32
innodb_log_group_home_dir = /home/mysql/redo
innodb_file_format = Barracuda

innodb_dict_size_limit = 1G

[mysqldump]
quick
quote-names
max_allowed_packet = 16M

[mysql]

[isamchk]
key_buffer = 16M

!includedir /etc/mysql/conf.d/

[Shahriyar Rzayev (rzayev-sehriyar)]

Still unable to reproduce with provided my.cnf.

[Launchpad Janitor (janitor)]

[Expired for Percona XtraBackup because there has been no activity for 60 days.]

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1386