xbcloud tool freezing after about 90% of backup

[Alex Alexander]

Running XtraBackup using xbcloud to backup database to swift. XtraBackup tool, freezing after about 80 90 % of backup in progress

160801 22:51:35 >> log scanned up to (14051196438888)
160801 22:51:36 >> log scanned up to (14051196587113)
160801 22:51:37 >> log scanned up to (14051196756661)
160801 22:51:38 >> log scanned up to (14051196853103)
160801 22:51:39 >> log scanned up to (14051196966643)
160801 22:51:40 >> log scanned up to (14051197174286)
160801 22:51:41 >> log scanned up to (14051197295650)

....and freezing at this point with no error

and xbclud and xtranackup are both running

[Sergei Glushchenko (sergei.glushchenko)]

Hello Alex,

Can you please run xbcloud with --verbose option and capture its stderr into a file?

[Alex Alexander (alexarder)]

Sergei,

Will re run the backup shortly and post the error logs

[Alex Alexander (alexarder)]

Here we go:

< Content-Length: 131
< Content-Type: text/html; charset=UTF-8
< Www-Authenticate: Swift realm="unknown"
< X-Trans-Id: tx765a52f43ff84fc6a20ed-0057a23697
< Date: Wed, 03 Aug 2016 18:23:19 GMT
< Connection: keep-alive
<
* Operation too slow. Less than 1024 bytes/sec transfered the last 5 seconds
* Expire cleared
* Closing connection #0
* Done waiting for 100-continue
* Operation too slow. Less than 1024 bytes/sec transfered the last 5 seconds
* Closing connection #1
error: chunk 64 'protocom/cdrs#P#p2010#SP#p2010sp0.ibd' 04adc3e2cddd95b9089d62bbdb964430 is not uploaded, but socket closed (10485831 bytes of 10485831 left to upload)
warning: retrying to upload chunk 64 of 'protocom/cdrs#P#p2010#SP#p2010sp0.ibd'
uploading chunk cadb-2.bri/full_backup/protocom/cdrs#P#p2010#SP#p2010sp0.ibd.00000000000000000064 (md5: 04adc3e2cddd95b9089d62bbdb964430, size: 10485831)
error: chunk 65 'protocom/cdrs#P#p2010#SP#p2010sp0.ibd' e4cb3c237ebf15c8b1bbd93c04d671d3 is not uploaded, but socket closed (10485831 bytes of 10485831 left to upload)
warning: retrying to upload chunk 65 of 'protocom/cdrs#P#p2010#SP#p2010sp0.ibd'
uploading chunk cadb-2.bri/full_backup/protocom/cdrs#P#p2010#SP#p2010sp0.ibd.00000000000000000065 (md5: e4cb3c237ebf15c8b1bbd93c04d671d3, size: 10485831)

[Alex Alexander (alexarder)]

The above are the only errors that I was able to locate in the entire log file.

Our Environment:

CentOS 6.5
XtraBackup percona-xtrabackup-24-2.4.4-1.el6.x86_64.rpm
Database: 10.0.3-MariaDB-log MariaDB Server

[Alex Alexander (alexarder)]

PS. the backup process did write over 50 GB of data to Swift in 10mb chuncks, here is our backup command:

xtrabackup --backup --stream=xbstream --extra-lsndir=/data/xtrbkup --target-dir=/data/xbackup/ | xbcloud put --verbose --swift-container="cadb-2.bri" --swift-user="database" --swift-auth-url="http://swift-proxy1-bri.telmate.cc/" --swift-storage-url="http://xxxxxv1/AUTH_database" --swift-key="XXXXXXXX" --parallel=2 full_backup

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1396