xtrabackup_56 fails to create suspend file 'InnoDB: Operating system error number 2 in a file operation.'

[George Ormond Lorch III]

The soon to be 2.1.4 release with the change from https://code.launchpad.net/~gl-az/percona-xtrabackup/2.1-ST31441-extra-suspend-logging/+merge/177904 fails 100% of the time with MS/PS 5.6:

xtrabackup: Creating suspend file '/data/dev/specify-io-size/test-0/backup-0.0/xtrabackup_suspended' with pid '26091'
InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Error: failed to create file '/data/dev/specify-io-size/test-0/backup-0.0/xtrabackup_suspended' with 71
innobackupex: Error: ibbackup child process has died at /data/bin/specify-io-size/2.0-xtradb56-base/innobackupex line 387.

This is due to a combination of two things, one, the change mentioned above changed the suspend file creation to use xb_file_create_no_error_handling, which eventually makes its way down into the InnoDB call os_file_create_simple_no_error_handling_func; and two, the new 5.6 srv_read_only_mode.

Values at time of call from xb_create_suspend_file:
create_mode=OS_FILE_CREATE
access_type=OS_FILE_READ_WRITE
srv_read_only_mode=1

The code from PS 5.6 os_file_create_simple_no_error_handling_func at os0file.c:1395:
        if (create_mode == OS_FILE_OPEN) {

                if (access_type == OS_FILE_READ_ONLY) {

                        create_flag = O_RDONLY;

                } else if (srv_read_only_mode) {

                        create_flag = O_RDONLY;

                } else {

                        ut_a(access_type == OS_FILE_READ_WRITE
                             || access_type == OS_FILE_READ_ALLOW_DELETE);

                        create_flag = O_RDWR;
                }

        } else if (srv_read_only_mode) {

                create_flag = O_RDONLY;

        } else if (create_mode == OS_FILE_CREATE) {

                create_flag = O_RDWR | O_CREAT | O_EXCL;

        } else {
                ib_logf(IB_LOG_LEVEL_ERROR,
                        "Unknown file create mode (%lu) for file '%s'",
                        create_mode, name);

                return((os_file_t) -1);
        }

So, as you can see, the new srv_read_only_mode logic will kick in first and change the create_flag to O_RDONLY instead of the desired O_RDWR, thus causing the 'open' to fail.

[Alexey Kopytov (akopytov)]

2.1.4 is not affected, the extra suspend logging patch was not merged to release-2.1.4.

[Alexey Kopytov (akopytov)]

Let's use mysys instead of InnoDB I/O routines, i.e. my_create() (with the O_EXCL) / my_write() / my_close(). See an example in ds_local.c

[Alexey Kopytov (akopytov)]

Re-assigning to myself, as I need this fixed to test another bugfix.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-383