cluster SST fail - /usr/bin/du: Argument list too long

Bug #1294431 reported by pawouk on 2014-03-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
Status tracked in 5.6
5.5
Medium
Alex Yurchenko
5.6
Medium
Alex Yurchenko
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Status tracked in 5.6
5.5
Fix Released
Undecided
Unassigned
5.6
Fix Released
Undecided
Unassigned

Bug Description

Hi Guys,

we have quite a lot of tables and databases (1000DB, each with 60tables).

When I try to add a new node to the cluster I receive an error:

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio TCP:10.100.2.3:4444; RC=( ${PIPESTATUS[@]} ) (20140319 02:22:23.431)

/usr//bin/wsrep_sst_xtrabackup-v2: line 234: /usr//bin/du: Argument list too long

WSREP_SST: [INFO] Sleeping before data transfer for SST (20140319 02:22:25.456)
WSREP_SST: [INFO] Streaming the backup to joiner at 10.100.2.3 4444 (20140319 02:22:35.463)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/mysql/my.cnf $INNOEXTRA --galera-info --stream=$sfmt ${TMPDIR} 2>${DATA}/innobackup.backup.log | pv -f -i 10 -N donor -s 2>0 | socat -u stdio TCP:10.100.2.3:4444; RC=( ${PIPESTATUS[@]} ) (20140319 02:22:35.468)
WSREP_SST: [ERROR] innobackupex finished with error: 1. Check /var/lib/mysql/data//innobackup.backup.log (20140319 02:22:35.697)
WSREP_SST: [ERROR] Cleanup after exit with status:22 (20140319 02:22:35.702)
140319 2:22:35 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.100.2.3:4444/xtrabackup_sst' --auth 'xtrabackuper:h@wagaV8$uyE' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --gtid '7002b138-9d04-11e2-0800-0dbb30030abe:1402150009'
140319 2:22:35 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.100.2.3:4444/xtrabackup_sst' --auth 'xtrabackuper:h@wagaV8$uyE' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --gtid '7002b138-9d04-11e2-0800-0dbb30030abe:1402150009': 22 (Invalid argument)
140319 2:22:35 [Warning] WSREP: 0 (sql1002): State transfer to 1 (sql1003) failed: -1 (Operation not permitted)
140319 2:22:35 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 1402150009)

The problematic code line is:

payload=$(du --block-size=1 -c **/*.ibd **/*.MYI **/*.MYI ibdata1 | awk 'END { print $1 }')

When I try to run it manualy at mysql data directory I receive the same error.

root@sql1002:/var/lib/mysql/data# du --block-size=1 -c **/*.ibd **/*.MYI **/*.MYI ibdata1
-bash: /usr/bin/du: Argument list too long
root@sql1002:/var/lib/mysql/data#

I believe it can be fixed by replacing the line with this command:

payload=$(find . -name "*.ibd" -name "*.MYI" -name "ibdata1" -type f -print0 | xargs -0 du --block-size=1 -c | awk 'END { print $1 }')

tags: added: sst xtrabackup

Yes, it can hit this bug when the number of files matched by " **/*.ibd **/*.MYI **/*.MYI ibdata1 " is really high.

Also, this bug is hit only when 'progress' sst option is used.

Using find should help I guess.

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1653

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers