xtrabackup crashes with docker

Bug #1526467 reported by Rémy Phelipot
28
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Status tracked in 2.4
2.3
Fix Released
Medium
Sergei Glushchenko
2.4
Fix Released
Medium
Sergei Glushchenko

Bug Description

MariaDB is running in a docker container and a volume is mounted on the host for the database files. I'm trying to backup the database with xtrabackup but I've got this error:

 xtrabackup --backup --target-dir=test --host=10.10.10.96 --user=root
151215 17:24:39 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;host=10.10.10.96;port=3306;mysql_socket=/var/run/mysqld/mysqld.sock' as 'root' (using password: YES).
151215 17:24:39 version_check Connected to MySQL server
151215 17:24:39 version_check Executing a version check against the server...
151215 17:24:39 version_check Done.
151215 17:24:39 Connecting to MySQL server host: 10.10.10.96, user: root, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
Using server version 10.0.21-MariaDB-1~jessie
2015-12-15 17:24:39 7fec27b46780 InnoDB: Assertion failure in thread 140652255143808 in file backup_copy.cc line 648
InnoDB: Failing assertion: realpath(first, real_first) != NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
17:24:39 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x2e) [0x94eebe]
xtrabackup(handle_fatal_signal+0x262) [0x90fcf2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fec27722340]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7fec25e86cc9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7fec25e8a0d8]
xtrabackup() [0x5a9f03]
xtrabackup(get_mysql_vars(st_mysql*)+0x42d) [0x5a639d]
xtrabackup(xb_init()+0x12b) [0x59b67b]
xtrabackup(main+0x641) [0x57e101]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fec25e71ec5]
xtrabackup() [0x5922c6]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup

I've tried to execute this command inside the container and it work successfully, but it seems impossible to save the database from the host or another container. I've also tried unsuccessfully to set the --datadir option to the host directory .

Can someone reproduce this issue? Do you have a solution?

Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :

Looks like either directory specified with --datadir or the one obtained from "show variables like 'datadir'" doesn't exist.

Changed in percona-xtrabackup:
status: New → Incomplete
Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :

Please also check if bug 1508448 is relevant. It is fixed in upcoming 2.3.3

Revision history for this message
Rémy Phelipot (remy-phelipot-deactivatedaccount) wrote :

MariaDb data directory on the host and in the container are not located in the same path. Inside the container, it's the default directory that is used (/var/lib/mysql/...) but it's a completely different directory on the host.
I'm using the host directory path in the command line for the --data-dir parameter, is it correct?

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Could not reproduce with following steps:

I have run MariaDB docker image as:

sh@percona-XPS-15:~$ sudo docker run --name mariadb-docker -v /home/dokcerdatadir:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=12345 -d mariadb:10.0.23
1dd163611045fe04033940377ca26268dc4d209e3b807e069e563d5e160bf53a

Mounted /var/lib/mysql of MariaDB in /home/dokcerdatadir of host.

Could connect as:

sh@percona-XPS-15:~$ mysql --user=root --password=12345 --host=172.17.0.2
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 12
Server version: 5.5.5-10.0.23-MariaDB-1~jessie mariadb.org binary distribution

mysql> select @@version;
+--------------------------+
| @@version |
+--------------------------+
| 10.0.23-MariaDB-1~jessie |
+--------------------------+
1 row in set (0.00 sec)

mysql> create database docker_test;
Query OK, 1 row affected (0.00 sec)

Synced as:
root@percona-XPS-15:~# ls /home/dokcerdatadir/
aria_log.00000001 aria_log_control docker_test ibdata1 ib_logfile0 ib_logfile1 multi-master.info mysql performance_schema

If you try to backup as:

sh@percona-XPS-15:~$ sudo xtrabackup --backup --datadir=/home/dokcerdatadir --target-dir=/home/xtrabackupdir --host=172.17.0.2 --user=root --password=12345

151224 14:36:22 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;host=172.17.0.2;port=3306;mysql_socket=/var/run/mysqld/mysqld.sock' as 'root' (using password: YES).
151224 14:36:22 version_check Connected to MySQL server
151224 14:36:22 version_check Executing a version check against the server...
151224 14:36:22 version_check Done.
151224 14:36:22 Connecting to MySQL server host: 172.17.0.2, user: root, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
Using server version 10.0.23-MariaDB-1~jessie
Error: option 'datadir' has different values:
  '/home/dokcerdatadir' in defaults file
  '/var/lib/mysql/' in SHOW VARIABLES

sh@percona-XPS-15:~$ xtrabackup --version
xtrabackup version 2.3.3 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 525ca7d)

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for Percona XtraBackup because there has been no activity for 60 days.]

Changed in percona-xtrabackup:
status: Incomplete → Expired
Revision history for this message
gdw2 (gdwarner) wrote :

I followed the same steps as ~rzayev-sehriyar (used the same docker command, same version of db, etc), but using the most recent version of xtrabackup (2.4.2). This is what I see:

# xtrabackup --backup --datadir=/home/dokcerdatadir --target-dir=/home/xtrabackupdir --host=172.17.0.4 --user=root --password=12345
160419 21:17:54 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;host=172.17.0.4;mysql_socket=/var/lib/mysql/mysql.sock' as 'root' (using password: YES).
160419 21:17:54 version_check Connected to MySQL server
160419 21:17:54 version_check Executing a version check against the server...
160419 21:17:54 version_check Done.
160419 21:17:54 Connecting to MySQL server host: 172.17.0.4, user: root, password: set, port: 0, socket: /var/lib/mysql/mysql.sock
Using server version 10.0.23-MariaDB-1~jessie
2016-04-19 21:17:54 0x7fea7849c740 InnoDB: Assertion failure in thread 140645017175872 in file backup_copy.cc line 650
InnoDB: Failing assertion: realpath(second, real_second) != NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
21:17:54 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x2c)[0xc7705c]
xtrabackup(handle_fatal_signal+0x262)[0xaefdf2]
/lib64/libpthread.so.0(+0xf100)[0x7fea7807c100]
/lib64/libc.so.6(gsignal+0x37)[0x7fea763d45f7]
/lib64/libc.so.6(abort+0x148)[0x7fea763d5ce8]
xtrabackup[0x6ee25d]
xtrabackup[0x73162c]
xtrabackup(_Z14get_mysql_varsP8st_mysql+0x723)[0x72e1a3]
xtrabackup(_Z7xb_initv+0x12b)[0x71e05b]
xtrabackup(main+0x76b)[0x6fde2b]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fea763c0b15]
xtrabackup[0x715ea5]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup
[root@vwsphx1prerbo-dbv04 ~]# xtrabackup -v
xtrabackup version 2.4.2 based on MySQL server 5.7.11 Linux (x86_64) (revision id: 8e86a84)

Changed in percona-xtrabackup:
status: Expired → New
status: New → Confirmed
Revision history for this message
gdw2 (gdwarner) wrote :

My apologies for messing with the issue status. If a new issue should be created instead, please let me know.

Revision history for this message
Ovais Tariq (ovais-tariq) wrote :

XtraBackup expects the datadir specified on the command-line to be the one that is reported by MySQL. This is not always true, for example when MySQL is running inside a container and XtraBackup is running on the host machine. Can this check be relaxed ?

Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :

Lets datadir from my.cnf override the one from SHOW VARIABLES. xtrabackup will print warning that they don't match, but continue.

Revision history for this message
Ovais Tariq (ovais-tariq) wrote :

Is the fix being targeted for any release soon?

Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :

See also bug 1503454

Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-741

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.