MySQL galera SST may fail innobackupex because of a race: /var/lib/mysql/ib_logfile0 failed: Destination file exists
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Committed
|
High
|
Bogdan Dobrelya | ||
Mitaka |
Fix Committed
|
High
|
Bogdan Dobrelya |
Bug Description
The bug is similar to the https:/
When an SST finished streaming data to a joiner, the "innobackupex-
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:
2016-04-27 14:21:21 22037 [Note] WSREP: 0.0 (n1): State transfer to 3.0 (n3) complete.
2016-04-27 14:21:21 22037 [Note] WSREP: Member 0.0 (n1) synced with group.
2016-04-27 14:21:21 22037 [Note] WSREP: 1.0 (n5): State transfer to 2.0 (n4) complete.
2016-04-27 14:21:21 22037 [Note] WSREP: Member 1.0 (n5) synced with group.
WSREP_SST: [INFO] Preparing the backup at /var/lib/
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}
WSREP_SST: [INFO] Moving the backup to /var/lib/mysql/ (20160427 14:21:33.597)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-
WSREP_SST: [ERROR] Cleanup after exit with status:1 (20160427 14:21:33.602)
2016-04-27 14:21:33 22037 [ERROR] WSREP: Process completed with error: wsrep_sst_
2016-04-27 14:21:33 22037 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2016-04-27 14:21:33 22037 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2016-04-27 14:21:33 22037 [ERROR] Aborting
And here is the innobackup log with straces:
execve(
* 55 vars */]) = 0
(... snip ...)
write(2, "160428 07:43:06 innobackupex: St"..., 236160428 07:43:06 innobackupex: Starting the move-back operation
IMPORTANT: Please check that the move-back run completes successfully.
At the end of a successful move-back run innobackupex
prints "completed OK!".
) = 236
write(2, "innobackupex version 2.3.4 based"..., 94innobackupex version 2.3.4 based on MySQL server 5.6.24 Linux (x86_64) (revision id: e80c779)
) = 94
stat("/
openat(AT_FDCWD, "/var/lib/mysql", O_RDONLY|
close(3) = 0
stat("/
openat(AT_FDCWD, "/var/lib/mysql", O_RDONLY|
close(3) = 0
chdir("
brk(0x25a5000) = 0x25a5000
mkdir("
stat("ib_logfile0", {st_mode=
stat("/
openat(AT_FDCWD, "/var/lib/mysql/", O_RDONLY|
close(3) = 0
stat("/
write(2, "Error: Move file ib_logfile0 to "..., 91Error: Move file ib_logfile0 to /var/lib/
) = 91
exit_group(1) = ?
+++ exited with 1 +++
____
Note, to get strace debugs, I applied the patch to the
--- a/wsrep_
+++ b/wsrep_
@@ -578,8 +578,8 @@ fi
INNOEXTRA=""
INNOAPPLY=
-INNOMOVE=
-INNOBACKUP=
+INNOMOVE="strace -f ${INNOBACKUPEX_BIN} --defaults-
+INNOBACKUP="strace -f ${INNOBACKUPEX_BIN} --defaults-
if [ "$WSREP_
then
Packages used:
percona-
Changed in fuel: | |
importance: | Undecided → High |
no longer affects: | fuel/newton |
Upstream bug https:/ /bugs.launchpad .net/percona- server/ +bug/1576080