Percona XtraDB Cluster - HA scalable solution for MySQL

SST fails with innodb_data_home_dir/innodb_log_home_dir

Reported by Raghavendra D Prabhu on 2013-01-11
48
This bug affects 11 people
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
Status tracked in 5.6
5.5
Medium
Alex Yurchenko
5.6
Medium
Alex Yurchenko
Percona XtraDB Cluster
Status tracked in Trunk
5.6
Undecided
Raghavendra D Prabhu
Trunk
Undecided
Raghavendra D Prabhu

Bug Description

This bug is similar to lp:1019288 but it shows a different error message.

================
130111 19:11:05 InnoDB: Completed initialization of buffer pool
InnoDB: The first specified data file /mnt/test/ibdata1 did not exist:
InnoDB: a new database to be created!
130111 19:11:05 InnoDB: Setting file /mnt/test/ibdata1 size to 100 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Progress in MB: 100
130111 19:11:05 InnoDB: Error: all log files must be created at the same time.
130111 19:11:05 InnoDB: All log files must be created also in database creation.
130111 19:11:05 InnoDB: If you want bigger or smaller log files, shut down the
130111 19:11:05 InnoDB: database and make sure there were no errors in shutdown.
130111 19:11:05 InnoDB: Then delete the existing log files. Edit the .cnf file
130111 19:11:05 InnoDB: and start the database again.
130111 19:11:05 [ERROR] Plugin 'InnoDB' init function returned error.
130111 19:11:05 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
130111 19:11:05 [ERROR] Unknown/unsupported storage engine: InnoDB
130111 19:11:05 [ERROR] Aborting

==========================

This happens because SST will try to copy the ibdata1 file
to the datadir and InnoDB of the
receiver will also create a ibdata1 in innodb_data_home_dir, and
this clashes.

Workaround: Not to have a separate innodb_data_home_dir

Fix: To make sure SST is aware of innodb_data_home_dir.

To reproduce, have something like:

datadir=/var/lib/mysql
innodb_data_home_dir = /mnt/test
innodb_data_file_path = ibdata1:100M:autoextend

in my.cnf along with others.

Where it fails:

 for (i = 0; i < srv_n_log_files; i++) {
  err = open_or_create_log_file(create_new_db, &log_file_created,
           log_opened, 0, i);
  if (err != DB_SUCCESS) {

   return((int) err);
  }

  if (log_file_created) {
   log_created = TRUE;
  } else {
   log_opened = TRUE;
  }
  if ((log_opened && create_new_db)
      || (log_opened && log_created)) {
   ut_print_timestamp(stderr);
   fprintf(stderr,
    " InnoDB: Error: all log files must be"
    " created at the same time.\n");
   ut_print_timestamp(stderr);
   fprintf(stderr,
    " InnoDB: All log files must be"
    " created also in database creation.\n");
   ut_print_timestamp(stderr);
   fprintf(stderr,
    " InnoDB: If you want bigger or smaller"
    " log files, shut down the\n");
   ut_print_timestamp(stderr);
   fprintf(stderr,
    " InnoDB: database and make sure there"
    " were no errors in shutdown.\n");
   ut_print_timestamp(stderr);
   fprintf(stderr,
    " InnoDB: Then delete the existing log files."
    " Edit the .cnf file\n");
   ut_print_timestamp(stderr);
   fprintf(stderr,
    " InnoDB: and start the database again.\n");

   return(DB_ERROR);
  }
 }

in srv0start.c

Changed in percona-xtradb-cluster:
status: New → Confirmed

innobackupex --copy-back/--move-back checks innodb_data_home_dir,
innodb_data_file_path and innodb_log_group_home_dir correctly.
So, it is better to use it.

wsrep_sst_xtrabackup can be modified to use a temporary directory and then use innobackupex --move-back to handle correctly.

However, in many instances, the partition is mounted directly on
/var/lib/mysql , so a temporary directory used for SST may be
bad. OTOH, innobackupex --move-back doesn't work if tar is
extracted under /var/lib/mysql/.tmp since it complains of
non-empty data directory.

Changed in percona-xtradb-cluster:
milestone: none → 5.5.30-24.8

Filed lp:1164945 for xtrabackup for issue mentioned in #2.

Changed in percona-xtradb-cluster:
milestone: 5.5.30-24.8 → 5.5.31-25

Waiting for XB 2.0.8 to be released to use this.

Changed in percona-xtradb-cluster:
assignee: nobody → Raghavendra D Prabhu (raghavendra-prabhu)
status: Confirmed → Triaged

This is not fixed in PXB 2.0.8, will be fixed in PXB 2.1.4 or later.

no longer affects: codership-mysql
Changed in percona-xtradb-cluster:
milestone: 5.5.32-24 → 5.5.32
Frederic Descamps (lefred) wrote :

This is also an issue when only referring innodb_log_group_home_dir (which can be good to move those logs to SSD)

summary: - SST fails with innodb_data_home_dir
+ SST fails with innodb_data_home_dir/innodb_log_home_dir

In this case, following will be done:

a) Have a separate setting - sst-log-data-home-dir which will be ON by
default for this to work. (Optional: Detect the options from my.cnf and
set them accordingly). Setting is required because if older PXB is used
then SST shouldn't break. (so it can be turned off).

b) Will go only into xtrabackup-v2 only.

c) This will depend on --move-back --force functionality of PXB
(which will be added in 2.1.6).

d) Minimum required version of PXB will also be bumped to 2.1.6
and above.

e) Here is how it goes down:

    i) On joiner, SST will be streamed into a subdirectory of
    datadir - $datadir/.sst

    ii) Backup will be prepared there.

    iii) innobackupex --move-back --force ../ will be used to
    copy back.

    iv) Based on success, .sst directory will be removed.

f) The $datadir/.sst is required but not any other directory
because data directories usually are mounted on separate mount
points and, otherwise ENOSPC issue can arise.

g) This should also handle innodb-data-home-dir,
innodb-log-home-dir, innodb-log-arch-dir and other exotic
directory variables.

This has been fixed, and will depend on https://bugs.launchpad.net/percona-xtrabackup/+bug/1164945 for the fix (it is going to be included in PXB 2.1.6)

Since PXB is going to be released after 25.1, moving it to 25.2 for now.

Consequent to the fix https://bugs.launchpad.net/percona-xtrabackup/+bug/1164945, this requires following in addition:

a) Cleanup of innodb-data-home-dir and innodb-log-home-dir

b) Use of the option - --force-non-empty-directories

Also, move the log files correctly.

For documentation,
""
This requires sst-special-dirs to be set under [sst] in cnf file to either 0 or 1. 0 being the default. Also, innodb-data-home-dir and/or innodb-log-group-home-dir need to be defined in my.cnf under [mysqld]. Percona Xtrabackup 2.1.6 or higher is required.
""

tags: added: doc-req
Kolbe (kolbe) wrote :

Why is this resolution so complicated? Why not put this new functionality into the regular "xtrabackup" SST script instead of into a new one? Why require this sst-special-dirs variable? I don't understand why it would not be desirable for the default behavior of the normal xtrabackup SST script to support innodb_log_group_home_dir, innodb_data_home_dir, et cetera.

Serge Kolodeznyh (kolodez) wrote :

I'm agree with Kolbe %)

sst-special-dirs = ! ( $innodb_log_group_home_dir eq $innodb_data_home_dir)

@Kolbe, @Serge,

> Why not put this new functionality into the regular "xtrabackup" SST script instead of into a new one?

The 'regular' script is just there for backward compatibility and, so,
all additions go to v2 only. The v2 script works differently and is not
compatible with old one. This is pretty clear in our
documentation and I don't know what is the ambiguity here on that.

> Why require this sst-special-dirs variable

This is because this relied on a PXB feature added in 2.1.6 which
was also newly released. I didn't want to bump minimum dependency version of PXB onto PXC since 2.1.6 was newly released, and it may have had bugs, so didn't want to restrict in case downgrade of PXB is required.

But, now that, it looks good, I will make it default: https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1276904

tags: added: sst xtrabackup
Alex Yurchenko (ayurchen) wrote :

This must be fixed with the latest xtrabackup ssp code ported from Percona in:
http://bazaar.launchpad.net/~codership/codership-mysql/wsrep-5.5/revision/3962
http://bazaar.launchpad.net/~codership/codership-mysql/5.6/revision/4058
Marking as fixed optimistically.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers