xtrabackup-v2 SST is broken on MariaDB 10.1/Galera w/ binary logs enabled
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Percona XtraBackup moved to https://jira.percona.com/projects/PXB | Status tracked in 2.4 | |||||
2.3 |
Invalid
|
High
|
Unassigned | |||
2.4 |
Invalid
|
High
|
Unassigned |
Bug Description
With SST innobackupex is run with --galera-info and XtraBackup will flush the binary logs and copy the latest log under a FTWRL lock. This fails under MariaDB 10.1 (at least) as there seems to be a race condition where FLUSH BINARY LOGS starts a new log, but there is a period of time before a "binlog checkpoint" is written. On first start after prepare on the joiner node, MariaDB fails with an error along the lines of:
[Note] Recovering after a crash using /var/log/
[ERROR] Binlog file '/var/log/
[ERROR] Crash recovery failed. Either correct the problem (if it's, for example, out of memory error and restart, or delete (or rename) binary log and start mysqld with --tc-heuristic-
[ERROR] Can't init tc log
[ERROR] Aborting
There are further details in https:/
This problem appears to be resolved by having xtrabackup perform an extra "FLUSH LOCAL ENGINE LOGS" prior to copying the binary log.
Currently xtrabackup 2.3 (and 2.4) does not enable FLUSH ENGINE LOGS on MariaDB due to simple check that fails on MariaDB:
if (strcmp(
}
And further, even when enabled, FLUSH ENGINE LOGS is currently only run after copying the binlog for Galera, so this wouldn't help MariaDB.
A simple patch again the percona-xtrabackup 2.3 branches seems to fix this problem reliably in my environment, but I think this could be probably improved to avoid the double flush engine logs under mariadb.
$ git rev-parse HEAD
6d3a30519702ba9
$ git diff
diff --git a/storage/
index 14d93ab..45350d4 100644
--- a/storage/
+++ b/storage/
@@ -1205,6 +1205,9 @@ write_current_
}
+ /* MariaDB requires FLUSH ENGINE LOGS or we may get an incomplete binlog */
+ xb_mysql_
+
https:/ /github. com/percona/ percona- xtrabackup/ pull/196