xtrabackup_56 fails to create suspend file 'InnoDB: Operating system error number 2 in a file operation.'

Reported by George Ormond Lorch III on 2013-08-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraBackup
Status tracked in 2.2
2.0
High
Alexey Kopytov
2.1
High
Alexey Kopytov
2.2
High
Alexey Kopytov

Bug Description

The soon to be 2.1.4 release with the change from https://code.launchpad.net/~gl-az/percona-xtrabackup/2.1-ST31441-extra-suspend-logging/+merge/177904 fails 100% of the time with MS/PS 5.6:

xtrabackup: Creating suspend file '/data/dev/specify-io-size/test-0/backup-0.0/xtrabackup_suspended' with pid '26091'
InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Error: failed to create file '/data/dev/specify-io-size/test-0/backup-0.0/xtrabackup_suspended' with 71
innobackupex: Error: ibbackup child process has died at /data/bin/specify-io-size/2.0-xtradb56-base/innobackupex line 387.

This is due to a combination of two things, one, the change mentioned above changed the suspend file creation to use xb_file_create_no_error_handling, which eventually makes its way down into the InnoDB call os_file_create_simple_no_error_handling_func; and two, the new 5.6 srv_read_only_mode.

Values at time of call from xb_create_suspend_file:
create_mode=OS_FILE_CREATE
access_type=OS_FILE_READ_WRITE
srv_read_only_mode=1

The code from PS 5.6 os_file_create_simple_no_error_handling_func at os0file.c:1395:
        if (create_mode == OS_FILE_OPEN) {

                if (access_type == OS_FILE_READ_ONLY) {

                        create_flag = O_RDONLY;

                } else if (srv_read_only_mode) {

                        create_flag = O_RDONLY;

                } else {

                        ut_a(access_type == OS_FILE_READ_WRITE
                             || access_type == OS_FILE_READ_ALLOW_DELETE);

                        create_flag = O_RDWR;
                }

        } else if (srv_read_only_mode) {

                create_flag = O_RDONLY;

        } else if (create_mode == OS_FILE_CREATE) {

                create_flag = O_RDWR | O_CREAT | O_EXCL;

        } else {
                ib_logf(IB_LOG_LEVEL_ERROR,
                        "Unknown file create mode (%lu) for file '%s'",
                        create_mode, name);

                return((os_file_t) -1);
        }

So, as you can see, the new srv_read_only_mode logic will kick in first and change the create_flag to O_RDONLY instead of the desired O_RDWR, thus causing the 'open' to fail.

Changed in percona-xtrabackup:
importance: Undecided → Critical
Alexey Kopytov (akopytov) wrote :

2.1.4 is not affected, the extra suspend logging patch was not merged to release-2.1.4.

summary: - 2.1.4 with 5.6 fails to create suspend file 'InnoDB: Operating system
+ xtrabackup_56 fails to create suspend file 'InnoDB: Operating system
error number 2 in a file operation.'
Changed in percona-xtrabackup:
importance: Critical → High
Alexey Kopytov (akopytov) wrote :

Let's use mysys instead of InnoDB I/O routines, i.e. my_create() (with the O_EXCL) / my_write() / my_close(). See an example in ds_local.c

Alexey Kopytov (akopytov) wrote :

Re-assigning to myself, as I need this fixed to test another bugfix.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers