[regression] MySQL may fail to SST because of a ./ib* files race condition
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Committed
|
High
|
Bogdan Dobrelya | ||
6.1.x |
Won't Fix
|
High
|
MOS Maintenance | ||
7.0.x |
Won't Fix
|
High
|
MOS Maintenance | ||
8.0.x |
Won't Fix
|
High
|
MOS Maintenance | ||
Mitaka |
Fix Released
|
High
|
Volodymyr Shypyguzov |
Bug Description
When a Percona MySQL 3.6 node is starting by OCF RA, it may fail to finish an xtrabackup-v2 SST because there is a race condition with accessing and removing the ./ib* files. A logs snippet example:
2016-04-26 07:21:57 5036 [Note] WSREP: Requesting state transfer: success, donor: 3
WSREP_SST: [INFO] Proceeding with SST (20160426 07:21:58.143)
WSREP_SST: [INFO] Cleaning the existing datadir (20160426 07:21:58.144)
removed '/var/lib/
removed '/var/lib/
removed '/var/lib/
removed '/var/lib/
removed '/var/lib/
WSREP_SST: [INFO] Cleaning the binlog directory /var/log/mysql as well (20160426 07:21:58.178)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:
2016-04-26 07:22:00 5036 [Note] WSREP: (8efee881, 'tcp://
^Gxbstream: Can't create/write to file './ibdata1' (Errcode: 17 - File exists)
Packages used: percona-
percona-xtrabackup 2.2.3-2.1build1 / 2.3.4-1.wily
How to reproduce:
* deploy an env
* unmanage the mysql clone resource
* pick a node and kill -9 myslqd
* Run mysqld binary for the node manually:
/usr/sbin/mysqld --verbose --basedir=/usr --datadir=
* issue kill -STOP `pgrep xbstream` a couple of times
* touch /var/lib/
* kill -CONT `pgrep xbstream`
This happens as the SST-time /var/lib/mysql/.sst dir is not managed by the xtrabackup-v2 as appropriate, if there is no innodb-
Another race condition is reported in the duplicated bug https:/
Changed in fuel: | |
importance: | Undecided → High |
milestone: | none → 10.0 |
assignee: | nobody → Fuel Library Team (fuel-library) |
tags: | added: area-library galera |
description: | updated |
description: | updated |
Changed in fuel: | |
status: | New → Confirmed |
Changed in fuel: | |
assignee: | Fuel Library Team (fuel-library) → Bogdan Dobrelya (bogdando) |
status: | Confirmed → In Progress |
description: | updated |
description: | updated |
summary: |
- MySQL may fail to SST because of a ./ibdata1 race condition + MySQL may fail to SST because of a ./ib* files race condition |
description: | updated |
no longer affects: | fuel/newton |
tags: | added: tech-debt |
tags: | added: on-verification |
Similar issue https:/ /groups. google. com/forum/ #!topic/ codership- team/UEuWLDtCqh 8