SST fails with innodb_data_home_dir/innodb_log_home_dir

Bug #1098566 reported by Raghavendra D Prabhu on 2013-01-11
This bug affects 13 people
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
Status tracked in 5.6
Alex Yurchenko
Alex Yurchenko
Percona XtraDB Cluster
Status tracked in 5.6
Raghavendra D Prabhu
Raghavendra D Prabhu
percona-xtrabackup (Ubuntu)

Bug Description

This bug is similar to lp:1019288 but it shows a different error message.

130111 19:11:05 InnoDB: Completed initialization of buffer pool
InnoDB: The first specified data file /mnt/test/ibdata1 did not exist:
InnoDB: a new database to be created!
130111 19:11:05 InnoDB: Setting file /mnt/test/ibdata1 size to 100 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Progress in MB: 100
130111 19:11:05 InnoDB: Error: all log files must be created at the same time.
130111 19:11:05 InnoDB: All log files must be created also in database creation.
130111 19:11:05 InnoDB: If you want bigger or smaller log files, shut down the
130111 19:11:05 InnoDB: database and make sure there were no errors in shutdown.
130111 19:11:05 InnoDB: Then delete the existing log files. Edit the .cnf file
130111 19:11:05 InnoDB: and start the database again.
130111 19:11:05 [ERROR] Plugin 'InnoDB' init function returned error.
130111 19:11:05 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
130111 19:11:05 [ERROR] Unknown/unsupported storage engine: InnoDB
130111 19:11:05 [ERROR] Aborting


This happens because SST will try to copy the ibdata1 file
to the datadir and InnoDB of the
receiver will also create a ibdata1 in innodb_data_home_dir, and
this clashes.

Workaround: Not to have a separate innodb_data_home_dir

Fix: To make sure SST is aware of innodb_data_home_dir.

To reproduce, have something like:

innodb_data_home_dir = /mnt/test
innodb_data_file_path = ibdata1:100M:autoextend

in my.cnf along with others.

Where it fails:

 for (i = 0; i < srv_n_log_files; i++) {
  err = open_or_create_log_file(create_new_db, &log_file_created,
           log_opened, 0, i);
  if (err != DB_SUCCESS) {

   return((int) err);

  if (log_file_created) {
   log_created = TRUE;
  } else {
   log_opened = TRUE;
  if ((log_opened && create_new_db)
      || (log_opened && log_created)) {
    " InnoDB: Error: all log files must be"
    " created at the same time.\n");
    " InnoDB: All log files must be"
    " created also in database creation.\n");
    " InnoDB: If you want bigger or smaller"
    " log files, shut down the\n");
    " InnoDB: database and make sure there"
    " were no errors in shutdown.\n");
    " InnoDB: Then delete the existing log files."
    " Edit the .cnf file\n");
    " InnoDB: and start the database again.\n");


in srv0start.c

Changed in percona-xtradb-cluster:
status: New → Confirmed

innobackupex --copy-back/--move-back checks innodb_data_home_dir,
innodb_data_file_path and innodb_log_group_home_dir correctly.
So, it is better to use it.

wsrep_sst_xtrabackup can be modified to use a temporary directory and then use innobackupex --move-back to handle correctly.

However, in many instances, the partition is mounted directly on
/var/lib/mysql , so a temporary directory used for SST may be
bad. OTOH, innobackupex --move-back doesn't work if tar is
extracted under /var/lib/mysql/.tmp since it complains of
non-empty data directory.

Changed in percona-xtradb-cluster:
milestone: none → 5.5.30-24.8

Filed lp:1164945 for xtrabackup for issue mentioned in #2.

Changed in percona-xtradb-cluster:
milestone: 5.5.30-24.8 → 5.5.31-25

Waiting for XB 2.0.8 to be released to use this.

Changed in percona-xtradb-cluster:
assignee: nobody → Raghavendra D Prabhu (raghavendra-prabhu)
status: Confirmed → Triaged

This is not fixed in PXB 2.0.8, will be fixed in PXB 2.1.4 or later.

no longer affects: codership-mysql
Changed in percona-xtradb-cluster:
milestone: 5.5.32-24 → 5.5.32
Frederic Descamps (lefred) wrote :

This is also an issue when only referring innodb_log_group_home_dir (which can be good to move those logs to SSD)

summary: - SST fails with innodb_data_home_dir
+ SST fails with innodb_data_home_dir/innodb_log_home_dir

In this case, following will be done:

a) Have a separate setting - sst-log-data-home-dir which will be ON by
default for this to work. (Optional: Detect the options from my.cnf and
set them accordingly). Setting is required because if older PXB is used
then SST shouldn't break. (so it can be turned off).

b) Will go only into xtrabackup-v2 only.

c) This will depend on --move-back --force functionality of PXB
(which will be added in 2.1.6).

d) Minimum required version of PXB will also be bumped to 2.1.6
and above.

e) Here is how it goes down:

    i) On joiner, SST will be streamed into a subdirectory of
    datadir - $datadir/.sst

    ii) Backup will be prepared there.

    iii) innobackupex --move-back --force ../ will be used to
    copy back.

    iv) Based on success, .sst directory will be removed.

f) The $datadir/.sst is required but not any other directory
because data directories usually are mounted on separate mount
points and, otherwise ENOSPC issue can arise.

g) This should also handle innodb-data-home-dir,
innodb-log-home-dir, innodb-log-arch-dir and other exotic
directory variables.

This has been fixed, and will depend on for the fix (it is going to be included in PXB 2.1.6)

Since PXB is going to be released after 25.1, moving it to 25.2 for now.

Consequent to the fix, this requires following in addition:

a) Cleanup of innodb-data-home-dir and innodb-log-home-dir

b) Use of the option - --force-non-empty-directories

Also, move the log files correctly.

For documentation,
This requires sst-special-dirs to be set under [sst] in cnf file to either 0 or 1. 0 being the default. Also, innodb-data-home-dir and/or innodb-log-group-home-dir need to be defined in my.cnf under [mysqld]. Percona Xtrabackup 2.1.6 or higher is required.

tags: added: doc-req
Kolbe (kolbe) wrote :

Why is this resolution so complicated? Why not put this new functionality into the regular "xtrabackup" SST script instead of into a new one? Why require this sst-special-dirs variable? I don't understand why it would not be desirable for the default behavior of the normal xtrabackup SST script to support innodb_log_group_home_dir, innodb_data_home_dir, et cetera.

Serge Kolodeznyh (kolodez) wrote :

I'm agree with Kolbe %)

sst-special-dirs = ! ( $innodb_log_group_home_dir eq $innodb_data_home_dir)

@Kolbe, @Serge,

> Why not put this new functionality into the regular "xtrabackup" SST script instead of into a new one?

The 'regular' script is just there for backward compatibility and, so,
all additions go to v2 only. The v2 script works differently and is not
compatible with old one. This is pretty clear in our
documentation and I don't know what is the ambiguity here on that.

> Why require this sst-special-dirs variable

This is because this relied on a PXB feature added in 2.1.6 which
was also newly released. I didn't want to bump minimum dependency version of PXB onto PXC since 2.1.6 was newly released, and it may have had bugs, so didn't want to restrict in case downgrade of PXB is required.

But, now that, it looks good, I will make it default:

tags: added: sst xtrabackup
Alex Yurchenko (ayurchen) wrote :

This must be fixed with the latest xtrabackup ssp code ported from Percona in:
Marking as fixed optimistically.

hi, I am having the same problem, my MySQL ver. is
Ver 5.5.31 for Linux on x86_64 (Percona XtraDB Cluster (GPL), wsrep_23.7.5.r3880)
xtrabackup ver. is
xtrabackup version 2.0.3 for Percona Server 5.1.59 unknown-linux-gnu (x86_64) (revision id: 470)
I downloaded the srep_sst_xtrabackup-v2
But the problem is still same. i.e., during SST, in prepare stage, it is not copying the files back to its dir. paths as configured in .cnf file
we have
innodb_log_group_home_dir is different from datadir

Please advice which ver. of xtrabackup has the fix.

Thanks for your help in advance

Mikhail Baykov (rusxakep) wrote :

percona-xtrabackup_2.3.2-1 affected

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in percona-xtrabackup (Ubuntu):
status: New → Confirmed
Przemek (pmalkowski) wrote :

Bumping up as it is again a problem in latest PXC 5.6.26. Looks like SST script is broken and ibdata1 and ib_logfile* are copied from .sst temp dir to datadir instead of innodb_data_home_dir and innodb_log_group_home_dir accordingly.
Test logs in attachment.

tags: added: i62766
Ramesh Sivaraman (rameshvs02) wrote :

Server is crashing even with wsrep_sst_method=rsync with different error

dir info


Error info

2015-12-04 03:36:08 12101 [Note] InnoDB: Setting log file /dev/shm/inno_data/ib_logfile101 size to 48 MB
2015-12-04 03:36:08 12101 [ERROR] InnoDB: Cannot create /dev/shm/inno_data/ib_logfile1
2015-12-04 03:36:08 12101 [ERROR] Plugin 'InnoDB' init function returned error.
2015-12-04 03:36:08 12101 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2015-12-04 03:36:08 12101 [ERROR] Unknown/unsupported storage engine: InnoDB
2015-12-04 03:36:08 12101 [ERROR] Aborting

-rw-rw---- 1 mysql mysql 12582912 Dec 4 02:55 ibdata1
-rw-rw---- 1 mysql mysql 50331648 Dec 4 02:55 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 Dec 4 02:55 ib_logfile1
-rw-rw---- 1 mysql mysql 50331648 Dec 4 02:55 ib_logfile101

If we run mysql_install_db on joiner node, startup script starts mysqld with success message but in background mysqld process terminating with wrong SST state

[root@c-p130-u1400-240 mysql]# ./support-files/mysql.server start
 ERROR! MySQL (Percona XtraDB Cluster) is not running, but lock file (/var/lock/subsys/mysql) exists
Starting MySQL (Percona XtraDB Cluster)..State transfer in progress, setting sleep higher
[root@c-p130-u1400-240 mysql]#

2015-12-04 03:41:30 12681 [Note] WSREP: SST received: 00000000-0000-0000-0000-000000000000:5
2015-12-04 03:41:30 12681 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.6.26-74.0-25.12-log' socket: '/usr/local/mysql/data/mysql.sock' port: 3306 Percona XtraDB Cluster binary (GPL) 5.6.26-25.12, Revision 624ef81, wsrep_25.12
2015-12-04 03:41:30 12681 [ERROR] WSREP: Application received wrong state:
        Received: 00000000-0000-0000-0000-000000000000
        Required: f3e24345-9a61-11e5-8823-eabc7692ebb8
2015-12-04 03:41:30 12681 [ERROR] WSREP: Application state transfer failed. This is unrecoverable condition, restart required.


5.6.26 (latest one from trunk)
2.2.12 (Xtrabackup)

What I observed
* on node#2 with datadir and innodb_data_home_dir all files except ibdata1 is created in datadir and ibdata1 is created in innodb_data_home_dir. Server starts perfectly fine. No issue on that front.



5.6.26 (latest one from trunk)
2.3.2 (Xtrabackup)

What I observed

* on node#2 with datadir and innodb_data_home_dir all files are created in datadir and so when server try to consult innodb_data_home_dir for the said ibdata1 file it doesn't find it and so it trigger creation of new ibdata1 and then eventually fails.
(Creation of new ibdata1 files set the mode of create_new_db which then also try to create iblogfiles* but iblogfiles* are created in datadir where they already exist and so finally server then aborts with error).

As confirmed this is a bug in xtrabackup. Check here: and fix for it is committed here It should be part of next xtrabackup release.

Original bug was fixed. This bug was meant to track it so will close this bug on PXC front.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers