The easiest way to test this, is to use Swift-All-In-One vagrant. Follow instructions:
https://github.com/swiftstack/vagrant-swift-all-in-one
Edit Vagrantfile and give IP -> 192.168.3.171
The backup command:
xtrabackup --defaults-file=/etc/mysql/my.cnf --backup --user=root --password=Baku12345# --stream=xbstream --extra-lsndir=/tmp | xbcloud put --storage=Swift --swift-container=test --swift-user=test:tester --swift-password=tester --swift-auth-url=http://192.168.3.171:8080/auth/ --swift-key=testing --swift-auth-version=1.0 --parallel=1000 full_backup --verbose
Ended with:
* Closing connection 22
error: chunk 1 'performance_schema/session_connect_attrs.frm' 016c0095b076d032d03ef6868bf485fe is not uploaded, but socket closed (58 bytes of 58 left to upload)
error: retry count limit reached
again at a later time.</p></html><html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html><html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html><html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html><html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>
From dmesg:
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.927953] XFS (loop0): Internal error xfs_attr3_leaf_write_verify at line 216 of file /build/linux-FPNhks/linux-3.13.0/fs/
xfs/xfs_attr_leaf.c. Caller 0xffffffffa0249700
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928730] CPU: 0 PID: 21510 Comm: xfsaild/loop0 Tainted: G OX 3.13.0-79-generic #123-Ubuntu
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928731] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928732] 0000000000000001 ffff88003d1b9bd0 ffffffff817257b8 ffff88000002f800
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928735] ffff88003d1b9be8 ffffffffa024d70b ffffffffa0249700 ffff88003d1b9c20
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928736] ffffffffa024d765 000000d800201300 ffff88003c388600 ffff880027a9a570
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928738] Call Trace:
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928744] [<ffffffff817257b8>] dump_stack+0x45/0x56
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928758] [<ffffffffa024d70b>] xfs_error_report+0x3b/0x40 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928765] [<ffffffffa0249700>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928772] [<ffffffffa024d765>] xfs_corruption_error+0x55/0x80 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928782] [<ffffffffa026abe0>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928788] [<ffffffffa0249700>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928795] [<ffffffffa024b3e5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928801] [<ffffffffa0249700>] _xfs_buf_ioapply+0x70/0x3a0 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928804] [<ffffffff8109aa30>] ? wake_up_state+0x20/0x20
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928810] [<ffffffffa024b3e5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928817] [<ffffffffa024b346>] xfs_buf_iorequest+0x46/0x90 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928823] [<ffffffffa024b3e5>] xfs_bdstrat_cb+0x55/0xb0 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928830] [<ffffffffa024b57b>] __xfs_buf_delwri_submit+0x13b/0x210 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928837] [<ffffffffa024c010>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928849] [<ffffffffa02aab70>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928856] [<ffffffffa024c010>] xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928866] [<ffffffffa02aada7>] xfsaild+0x237/0x5c0 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928876] [<ffffffffa02aab70>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928879] [<ffffffff8108b8a2>] kthread+0xd2/0xf0
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928880] [<ffffffff8108b7d0>] ? kthread_create_on_node+0x1c0/0x1c0
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928883] [<ffffffff81736328>] ret_from_fork+0x58/0x90
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928884] [<ffffffff8108b7d0>] ? kthread_create_on_node+0x1c0/0x1c0
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.928885] XFS (loop0): Corruption detected. Unmount and run xfs_repair
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 kernel: [ 1281.929229] XFS (loop0): xfs_do_force_shutdown(0x8) called from line 1320 of file /build/linux-FPNhks/linux-3.13.0/fs/xfs/xfs_buf.c. Return address = 0xffffffffa024972c
From swift side:
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 container-6041: STDERR: 127.0.0.1 - - [10/Mar/2016 12:34:08] "PUT /sdb4/450/AUTH_test/test/full_backup/performance_schema/accounts.frm.00000000000000000001 HTTP/1.1" 500 962 0.090000 (txn: tx6ace47793be1470fabe54-0056e169bf)
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 container-6011: STDERR: (21892) accepted ('127.0.0.1', 56327)
Mar 10 12:34:08 vagrant-ubuntu-trusty-64 object-6040: ERROR Container update failed (saving for async update later): 404 response from 127.0.0.1:6011/sdb1 (txn: txdea5aea8710249478829a-0056e169bf)
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 container-6041: STDERR: 127.0.0.1 - - [10/Mar/2016 12:34:09] "PUT /sdb4/450/AUTH_test/test/full_backup/performance_schema/events_waits_summary_global_by_event_name.frm.00000000000000000001 HTTP/1.1" 404 193 0.004141 (txn: tx613b5b06683248578c781-0056e169bf)
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6010: STDERR: os.path.join(device_path, get_tmp_dir(policy)))
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6010: STDERR: File "/vagrant/swift/swift/common/utils.py", line 3264, in run_in_thread
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6020: STDERR: os.path.join(device_path, get_tmp_dir(policy)))
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6020: STDERR: File "/vagrant/swift/swift/common/utils.py", line 3264, in run_in_thread
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6020: STDERR: result = func(*args, **kwargs)
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6010: STDERR: result = func(*args, **kwargs)
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6030: STDERR: Traceback (most recent call last):
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6010: STDERR: File "/vagrant/swift/swift/common/utils.py", line 2236, in write_pickle
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6020: STDERR: File "/vagrant/swift/swift/common/utils.py", line 2236, in write_pickle
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6010: STDERR: fd, tmppath = mkstemp(dir=tmp, suffix='.tmp')
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 proxy-server: STDERR: 192.168.3.1 - - [10/Mar/2016 12:34:09] "PUT /v1/AUTH_test/test/full_backup/performance_schema/table_lock_waits_summary_by_table.frm.00000000000000000001 HTTP/1.1" 201 278 2.211205 (txn: tx3c4b1173478a4024b6426-0056e169bf)
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 115, in wait
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6010: STDERR: File "/usr/lib/python2.7/tempfile.py", line 308, in mkstemp
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6010: STDERR: return _mkstemp_inner(dir, prefix, suffix, flags)
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6030: STDERR: listener.cb(fileno)
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6030: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6010: STDERR: File "/usr/lib/python2.7/tempfile.py", line 239, in _mkstemp_inner
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6030: STDERR: result = function(*args, **kwargs)
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6010: STDERR: fd = _os.open(file, flags, 0600)
Mar 10 12:34:09 vagrant-ubuntu-trusty-64 object-6030: STDERR: File "/vagrant/swift/swift/obj/server.py", line 256, in async_update
OS info:
vagrant@saio:/var/log$ uname -a
Linux saio 3.13.0-79-generic #123-Ubuntu SMP Fri Feb 19 14:27:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
If it is possible to update DOC and add a caution section. Maybe it is caused by -> paralel >= 1000
Percona now uses JIRA for bug reports so this bug report is migrated to: https:/