innobackupex say "completed OK!" but backup is failed due out of free space

[Andrew Sitnikov]

[root@oscar tmp]# /usr/bin/innobackupex --user=root --password=******* /home/backup/

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy.
All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex: Using mysql Ver 14.14 Distrib 5.1.50, for unknown-linux-gnu (x86_64) using readline 5.1
innobackupex: Using mysql server version Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

innobackupex: Created backup directory /home/backup/2010-11-07_13-11-01
101107 13:11:01 innobackupex: Starting mysql with options: --password=******* --user=root --unbuffered --
101107 13:11:01 innobackupex: Connected to database with mysql child process (pid=27714)
101107 13:11:05 innobackupex: Connection to database server closed

101107 13:11:05 innobackupex: Starting ibbackup with command: xtrabackup --backup --suspend-at-end --target-dir=/home/backup/2010-11-07_13-11-01
innobackupex: Waiting for ibbackup (pid=27720) to suspend
innobackupex: Suspend file '/home/backup/2010-11-07_13-11-01/xtrabackup_suspended'

xtrabackup Ver 1.2 Rev 132 for 5.1.45 unknown-linux-gnu (x86_64)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:100M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 1073741824
>> log scanned up to (259264105271)
Copying ./ibdata1
     to /home/backup/2010-11-07_13-11-01/ibdata1
xtrabackup: Page 64 seems double write buffer. passing the check.
...skip...
xtrabackup: Page 191 seems double write buffer. passing the check.
>> log scanned up to (259264115850)
        ...done
Copying ./saturn/radreply_backup.ibd
     to /home/backup/2010-11-07_13-11-01/saturn/radreply_backup.ibd
        ...done

... skip ...

101107 13:41:15 InnoDB: Error: Write to file /home/backup/2010-11-07_13-11-01/saturn/dhcplog.ibd failed at offset 6 4208984064.
InnoDB: 524288 bytes should have been written, only 65536 were written.
InnoDB: Operating system error number 2.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: Error number 2 means 'No such file or directory'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html
xtrabackup: Error: xtrabackup_copy_datafile() failed.
xtrabackup: Warining: failed to copy, but continuing.

101107 13:41:16 innobackupex: Continuing after ibbackup has suspended
101107 13:41:16 innobackupex: Starting mysql with options: --password=******* --user=root --unbuffered --
101107 13:41:16 innobackupex: Connected to database with mysql child process (pid=30018)
101107 13:41:20 innobackupex: Starting to lock all tables...
>> log scanned up to (259280275767)
>> log scanned up to (259280323407)
101107 13:41:30 innobackupex: All tables locked and flushed to disk

101107 13:41:30 innobackupex: Starting to backup .frm, .MRG, .MYD, .MYI,
innobackupex: .TRG, .TRN, .ARM, .ARZ and .opt files in
innobackupex: subdirectories of '/var/lib/mysql'
innobackupex: Backing up files '/var/lib/mysql/mysql/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,opt,par}' (65 files)
>> log scanned up to (259280323712)
innobackupex: Backing up files '/var/lib/mysql/saturn/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,opt,par}' (15 files)
101107 13:41:31 innobackupex: Finished backing up .frm, .MRG, .MYD, .MYI, .TRG, .TRN, .ARM, .ARZ and .opt files

innobackupex: Resuming ibbackup

xtrabackup: The latest check point (for incremental): '259267465559'
>> log scanned up to (259280324183)
xtrabackup: Stopping log copying thread.....
xtrabackup: Transaction log of lsn (259261055368) to (259280324183) was copied.
101107 13:41:34 innobackupex: All tables unlocked
101107 13:41:34 innobackupex: Connection to database server closed

innobackupex: Backup created in directory '/home/backup/2010-11-07_13-11-01'
innobackupex: MySQL binlog position: filename '', position
101107 13:41:34 innobackupex: completed OK!

[Alexey Kopytov (akopytov)]

I think I fixed this bug while working on support for compressed InnoDB tables: http://bazaar.launchpad.net/~percona-dev/percona-xtrabackup/trunk/revision/206.2.1

Which means the fix will be available in XtraBackup 1.6.

[Valentine Gostev (longbow)]

I have verified that fix exists in 1.6 and innobackupex throws valid error message and error code:

~# df -h | grep sda1
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 1.2G 1.2G 29M 98% /
[root@gval-cent5-64-n401 ~]# innobackupex /root/backup/
[root@gval-cent5-64-n401 ~]# du -sh /var/lib/mysql
120M /var/lib/mysql
[root@gval-cent5-64-n401 ~]# innobackupex /root/backup/

<snip>

[01] Copying /var/lib/mysql/ibdata1
     to /root/backup/2011-04-29_17-21-10/ibdata1
110429 17:21:14 InnoDB: Error: Write to file /root/backup/2011-04-29_17-21-10/ibdata1 failed at offset 0 29360128.
InnoDB: 1048576 bytes should have been written, only 770048 were written.
InnoDB: Operating system error number 0.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: Error number 0 means 'Success'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.
innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 336.
~# echo $?
2

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-289