cluster SST fail - /usr/bin/du: Argument list too long

Bug #1294431 reported by pawouk on 2014-03-19
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
Status tracked in 5.6
Alex Yurchenko
Alex Yurchenko
Percona XtraDB Cluster moved to
Status tracked in 5.6
Fix Released
Fix Released

Bug Description

Hi Guys,

we have quite a lot of tables and databases (1000DB, each with 60tables).

When I try to add a new node to the cluster I receive an error:

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio TCP:; RC=( ${PIPESTATUS[@]} ) (20140319 02:22:23.431)

/usr//bin/wsrep_sst_xtrabackup-v2: line 234: /usr//bin/du: Argument list too long

WSREP_SST: [INFO] Sleeping before data transfer for SST (20140319 02:22:25.456)
WSREP_SST: [INFO] Streaming the backup to joiner at 4444 (20140319 02:22:35.463)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/mysql/my.cnf $INNOEXTRA --galera-info --stream=$sfmt ${TMPDIR} 2>${DATA}/innobackup.backup.log | pv -f -i 10 -N donor -s 2>0 | socat -u stdio TCP:; RC=( ${PIPESTATUS[@]} ) (20140319 02:22:35.468)
WSREP_SST: [ERROR] innobackupex finished with error: 1. Check /var/lib/mysql/data//innobackup.backup.log (20140319 02:22:35.697)
WSREP_SST: [ERROR] Cleanup after exit with status:22 (20140319 02:22:35.702)
140319 2:22:35 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '' --auth 'xtrabackuper:h@wagaV8$uyE' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --gtid '7002b138-9d04-11e2-0800-0dbb30030abe:1402150009'
140319 2:22:35 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '' --auth 'xtrabackuper:h@wagaV8$uyE' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --gtid '7002b138-9d04-11e2-0800-0dbb30030abe:1402150009': 22 (Invalid argument)
140319 2:22:35 [Warning] WSREP: 0 (sql1002): State transfer to 1 (sql1003) failed: -1 (Operation not permitted)
140319 2:22:35 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 1402150009)

The problematic code line is:

payload=$(du --block-size=1 -c **/*.ibd **/*.MYI **/*.MYI ibdata1 | awk 'END { print $1 }')

When I try to run it manualy at mysql data directory I receive the same error.

root@sql1002:/var/lib/mysql/data# du --block-size=1 -c **/*.ibd **/*.MYI **/*.MYI ibdata1
-bash: /usr/bin/du: Argument list too long

I believe it can be fixed by replacing the line with this command:

payload=$(find . -name "*.ibd" -name "*.MYI" -name "ibdata1" -type f -print0 | xargs -0 du --block-size=1 -c | awk 'END { print $1 }')

tags: added: sst xtrabackup

Yes, it can hit this bug when the number of files matched by " **/*.ibd **/*.MYI **/*.MYI ibdata1 " is really high.

Also, this bug is hit only when 'progress' sst option is used.

Using find should help I guess.

Percona now uses JIRA for bug reports so this bug report is migrated to:

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers