running a backup without "datadir" in my.cnf copies data from / instead of mysql data directory

[Miguel Angel Nieto]

If you run xtrabackup as a normal user, not root, and there is no datadir on the my.cnf, it starts to copy server's storage content instead of mysql.

root@hbase:~# sudo su - mysql
mysql@hbase:/$ strace xtrabackup --backup --target-dir=/tmp
[...]
open("/var/lib/mysql/ib_logfile0", O_RDONLY) = 3
open("/tmp/xtrabackup_logfile", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 4
[...]
stat("./undo001", 0x7fffbe521cd0) = -1 ENOENT (No such file or directory)
open("./undo001", O_RDONLY) = -1 ENOENT (No such file or directory)
open(".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 6
getdents(6, /* 26 entries */, 32768) = 680
stat("./proc", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("./proc", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7
getdents(7, /* 119 entries */, 32768) = 3184
stat("./proc/asound", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
stat("./proc/dri", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
stat("./proc/sysrq-trigger", {st_mode=S_IFREG|0200, st_size=0, ...}) = 0
stat("./proc/partitions", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
stat("./proc/diskstats", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
stat("./proc/crypto", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
stat("./proc/key-users", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
stat("./proc/keys", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
stat("./proc/kpageflags", {st_mode=S_IFREG|0400, st_size=0, ...}) = 0
stat("./proc/kpagecount", {st_mode=S_IFREG|0400, st_size=0, ...}) = 0
stat("./proc/kmsg", {st_mode=S_IFREG|0400, st_size=0, ...}) = 0
stat("./proc/kcore", {st_mode=S_IFREG|0400, st_size=140737486262272, ...}) = 0
stat("./proc/softirqs", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
stat("./proc/version", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
stat("./proc/uptime", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
stat("./proc/stat", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[...]
tat("./bin/zmore", {st_mode=S_IFREG|0755, st_size=2802, ...}) = 0
stat("./bin/mt-gnu", {st_mode=S_IFREG|0755, st_size=68816, ...}) = 0
stat("./bin/findmnt", {st_mode=S_IFREG|0755, st_size=40176, ...}) = 0
stat("./bin/echo", {st_mode=S_IFREG|0755, st_size=27008, ...}) = 0
stat("./bin/bzcmp", {st_mode=S_IFREG|0755, st_size=2140, ...}) = 0
stat("./bin/domainname", {st_mode=S_IFREG|0755, st_size=18760, ...}) = 0
stat("./bin/hostname", {st_mode=S_IFREG|0755, st_size=18760, ...}) = 0
stat("./bin/mt", {st_mode=S_IFREG|0755, st_size=68816, ...}) = 0
stat("./bin/tempfile", {st_mode=S_IFREG|0755, st_size=10376, ...}) = 0
stat("./bin/bzdiff", {st_mode=S_IFREG|0755, st_size=2140, ...}) = 0
stat("./bin/dumpkeys", {st_mode=S_IFREG|0755, st_size=78048, ...}) = 0
stat("./bin/uncompress", {st_mode=S_IFREG|0755, st_size=2251, ...}) = 0
stat("./bin/netcat", {st_mode=S_IFREG|0755, st_size=27160, ...}) = 0
stat("./bin/bzmore", {st_mode=S_IFREG|0755, st_size=1297, ...}) = 0
[...]
write(2, "2014-10-02 04:05:39 7fec22f64720", 322014-10-02 04:05:39 7fec22f64720) = 32
write(2, " InnoDB: Operating system error"..., 64 InnoDB: Operating system error number 13 in a file operation.
) = 64
write(2, "InnoDB: The error means mysqld d"..., 89InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.

This also makes PXC to fail, because SST is run under mysql user:

xtrabackup version 2.2.4 based on MySQL server 5.6.17 Linux (x86_64) (revision id: )
xtrabackup: uses posix_fadvise().
xtrabackup: cd to
xtrabackup: open files limit requested 65535, set to 65535
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = /var/lib/mysql
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = /var/lib/mysql
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
xtrabackup: using O_DIRECT
>> log scanned up to (689721549)
2014-10-02 04:02:02 7f60b6b33720 InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
innobackupex: Error: The xtrabackup child process has died at /usr//bin/innobackupex line 2679.

ii percona-xtrabackup 2.2.4-5004-1.wheezy amd64 Open source backup tool for InnoDB and XtraDB

[Miguel Angel Nieto (miguelangelnieto)]

This is the my.cnf I used when trying to reproduce another bug:

[client]
  loose-default-character-set = utf8mb4
  port = 3306

[isamchk]
  key_buffer = 16M

[mysql]
  default-character-set = utf8mb4
  max_allowed_packet = 64M

[mysqld]
  binlog_cache_size = 32M
  binlog_format = ROW
  character-set-client-handshake = FALSE
  character-set-server = utf8mb4
  collation-server = utf8mb4_unicode_ci
  default-storage-engine = INNODB
  expire_logs_days = 10
  init-connect = 'SET collation_connection = utf8mb4_unicode_ci; SET NAMES utf8mb4'
  innodb_autoinc_lock_mode = 2
  innodb_file_format = Barracuda
  innodb_file_per_table = true
  innodb_flush_log_at_trx_commit = 0
  innodb_flush_method = O_DIRECT
  innodb_large_prefix = true
  innodb_lock_wait_timeout = 30
  innodb_locks_unsafe_for_binlog = 1
  innodb_log_buffer_size = 8M
  innodb_log_file_size = 50331648

  key_buffer = 16M
  log_bin = mysql-bin.log
  log_error = error.log
  log_slave_updates = true
  log_slow_verbosity = microtime,query_plan,innodb
  long_query_time = 1
  max_allowed_packet = 64M
  max_binlog_size = 100M
  max_connections = 800
  max_heap_table_size = 67108864
  myisam-recover = BACKUP
  open_files_limit = 65535
  port = 3306
  query_cache_size = 0
  query_cache_type = 0
  server-id = 2127359171
  skip-external-locking = true
  slave-net-timeout = 10
  sql-mode = NO_ENGINE_SUBSTITUTION
  table_definition_cache = 1024
  table_open_cache = 2048
  thread_cache_size = 80
  thread_stack = 192K
  thread_statistics = OFF
  tmp_table_size = 67108864
  user = mysql
  userstat = ON
  wsrep_OSU_method = TOI
  wsrep_causal_reads = ON
  wsrep_cluster_address = gcomm://192.168.1.106,192.168.1.107,192.168.1.108
  wsrep_cluster_name = mycluster
  wsrep_node_address = 192.168.1.106
  wsrep_node_name = xtradb1
  wsrep_provider = /usr/lib/libgalera_smm.so
  wsrep_slave_threads = 64
  wsrep_sst_auth = sst:sst
  wsrep_sst_method = xtrabackup-v2
  innodb_data_home_dir = /var/lib/mysql
  innodb_log_group_home_dir = /var/lib/mysql
[xtrabackup]
  default-character-set = latin1

[Alexey Kopytov (akopytov)]

This is a regression from the original xtrabackup port to 5.6. If there's no datadir in my.cnf, the xtrabackup _binary_ should fail with the error message saying "Please set parameter 'datadir'". But due to different mysql_data_home handling in 5.6 and 5.5-, it proceeds with an empty datadir (which still leads to an error, but it's hard to understand what's wrong).

PXC does not call the xtrabackup binary directly, it uses innobackupex instead. After https://blueprints.launchpad.net/percona-xtrabackup/+spec/read-server-options-with-show-variables is implemented, innobackupex will be able to read datadir from the server connection and pass it to the xtrabackup binary if necessary (i.e. if there's no datadir definition in my.cnf).

[Miguel Angel Nieto (miguelangelnieto)]

PXC uses innobackupex, but the SST fails with the same error:

2014-10-02 04:02:02 7f60b6b33720 InnoDB: Operating system error number 13 in a file operation.

If found this bug when debugging why SST wasn't working. I had to run the SST process manually with strace to finally find the problem. So most probably, that implementation that reads the datadir from the server connection doesn't work in all cases.

[Alexey Kopytov (akopytov)]

The difference with innobackupex is that it has a server connection, so it is possible to 'autodetect' the default datadir, which I'm working on right now. The xtrabackup binary does not connect to the server, so it should fail with an error if there is no datadir in my.cnf. I was just clarifying that a fix for this bug should target the latter rather than the former.

[Miguel Angel Nieto (miguelangelnieto)]

Something is wrong in former too, this is the log from SST:

141002 13:42:08 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/etc/mysql/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp/tmp.MUwYvJ7cQk --tmpdir=/tmp/tmp.MUwYvJ7cQk --extra-lsndir='/tmp/tmp.MUwYvJ7cQk' --stream=xbstream
innobackupex: Waiting for ibbackup (pid=19683) to suspend
innobackupex: Suspend file '/tmp/tmp.MUwYvJ7cQk/xtrabackup_suspended_2'

xtrabackup version 2.2.4 based on MySQL server 5.6.17 Linux (x86_64) (revision id: )
xtrabackup: uses posix_fadvise().
xtrabackup: cd to
xtrabackup: open files limit requested 65535, set to 65535
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = /var/lib/mysql
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = /var/lib/mysql
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
xtrabackup: using O_DIRECT
>> log scanned up to (47464191)
2014-10-02 13:42:09 7f800c053720 InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
innobackupex: Error: The xtrabackup child process has died at /usr//bin/innobackupex line 2679.

innobackupex is not passing the default datadir to xtrabackup, so xtrabackup fails without a message while tries to copy contents from server storage (what you are checking now).

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-925