An Xtrabackup SST may fail because of a ./ibdata1 race condition

Bug #1575051 reported by Bogdan Dobrelya
This bug report is a duplicate of:  Bug #1566228: XB crashes with insufficient param.. Edit Remove
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
New
Undecided
Unassigned

Bug Description

Repro steps:
- Run a galera cluster of 5 nodes with a random-halves network partitions on/off (via an iptables reject rule), unless the issue's reproduced an a node or nodes. For each partition, kill -9 the nodes in a minority halve, then start them back once the partition ended.

An example logs snippet:
WSREP_SST: [INFO] Proceeding with SST (20160426 08:41:30.576)
WSREP_SST: [INFO] Cleaning the existing datadir (20160426 08:41:30.578)
removed '/var/lib/mysql/ib_logfile0'
removed '/var/lib/mysql/ib_logfile1'
removed '/var/lib/mysql/mysql/slave_worker_info.ibd'
removed '/var/lib/mysql/mysql/slave_master_info.ibd'
removed directory: '/var/lib/mysql/mysql'
removed '/var/lib/mysql/ibdata1'
removed '/var/lib/mysql/backup-my.cnf'
removed '/var/lib/mysql/auto.cnf'
WSREP_SST: [INFO] Cleaning the binlog directory /var/log/mysql as well (20160426 08:41:30.609)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160426 08:41:30.613)
xbstream: Can't create/write to file './ibdata1' (Errcode: 17 - File exists)

Similar topic
https://groups.google.com/forum/#!topic/codership-team/UEuWLDtCqh8

Packages used: percona-xtradb-cluster-server-5.6 5.6.21-25.8-0ubuntu2,
percona-xtrabackup 2.2.3-2.1build1

To ensure the issue will always happen, add to the config file:
[sst]
cpat = '.*galera\.cache$\|.*ibdata.*$\|.*sst_in_progress$\|.*gvwstate\.dat$\|.*grastate\.dat$\|.*\.err$\|.*\.log$\|.*RPM_UPGRADE_MARKER$\|.*RPM_UPGRADE_HISTORY$'

this will make the ibdata file to be preserved and the File exists error to pop up.

description: updated
description: updated
description: updated
no longer affects: percona-server
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.