Upload --change uploads everything

[Guillaume Dondero]

The --change flag for swift client is not used, if I had files in my directory and try to continue upload, it uploads everything.

Here is how I checked :

root@test-server:~/test_upload_swift# dd if=/dev/urandom of=test_file_01 bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.53983 s, 11.0 MB/s
root@test-server:~/test_upload_swift# dd if=/dev/urandom of=test_file_02 bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.57748 s, 10.9 MB/s
root@test-server:~/test_upload_swift# dd if=/dev/urandom of=test_file_03 bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.55197 s, 11.0 MB/s
root@test-server:~/test_upload_swift# dd if=/dev/urandom of=test_file_04 bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.55002 s, 11.0 MB/s
root@test-server:~/test_upload_swift# cd ..
root@test-server:~# time swift -v -V 2 -U userAccount:userAccount -K userPassword -A http://keystone.local:5000/v2.0 upload -c test-container test_upload_swift
test_upload_swift/test_file_02
test_upload_swift/test_file_01
test_upload_swift/test_file_03
test_upload_swift/test_file_04

real 0m6.557s
user 0m0.748s
sys 0m0.496s
root@test-server:~# cd test_upload_swift/
root@test-server:~/test_upload_swift# dd if=/dev/urandom of=test_file_05 bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.56193 s, 11.0 MB/s
root@test-server:~/test_upload_swift# dd if=/dev/urandom of=test_file_06 bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.52996 s, 11.0 MB/s
root@test-server:~/test_upload_swift# dd if=/dev/urandom of=test_file_07 bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.56446 s, 11.0 MB/s
root@test-server:~/test_upload_swift# dd if=/dev/urandom of=test_file_08 bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.53834 s, 11.0 MB/s
root@test-server:~/test_upload_swift# cd
root@test-server:~# time swift -v -V 2 -U userAccount:userAccount -K userPassword -A http://keystone.local:5000/v2.0 upload -c test-container test_upload_swift
test_upload_swift/test_file_03
test_upload_swift/test_file_08
test_upload_swift/test_file_02
test_upload_swift/test_file_01
test_upload_swift/test_file_07
test_upload_swift/test_file_06
test_upload_swift/test_file_05
test_upload_swift/test_file_04

real 0m14.067s
user 0m1.156s
sys 0m0.948s

[Guillaume Dondero (gdondero)]

Forgot to say the version of packages we use :
python-swift 1.4.8-0ubuntu2
swift 1.4.8-0ubuntu2
swift-account 1.4.8-0ubuntu2
swift-container 1.4.8-0ubuntu2
swift-object 1.4.8-0ubuntu2
swift-proxy 1.4.8-0ubuntu2

[Guillaume Dondero (gdondero)]

I have created a patch to fix it, I'm beginner with python, so maybe it could be better.

[Clint Byrum (clint-fewbar)]

Marking Triaged in Ubuntu as this is an upstream issue and it is attached to python-swiftclient already.

[Chmouel Boudjnah (chmouel)]

@Guillaume it would be nice if you can submit up for review to python-swiftclient sees the process how to do that here:

https://wiki.openstack.org/wiki/How_To_Contribute

[Naveen T J (naveen-tj)]

Please try to list the objects of container along with time stamp. May be it can give clear picture if it is uploading all files or just listing the objects which are there in directory.

A I ran a same test and found only latest changed files are getting uploaded.

Also please check I have used python-client of different version. My suggestion in this post to cross check one more time with time stamp with your packages.

I have listed the packages list too which are used by me to do this test.

# time swift -v -A http://x.x.x.76:35357/v3 upload -c container2 test-upload-swift/
test-upload-swift/test_file_02
test-upload-swift/test_file_01
test-upload-swift/object1

real 0m48.316s
user 0m0.652s
sys 0m0.120s

# swift list -l container2
        1198 2016-02-02 05:37:22 Desktop/BLOB_to_FILE.txt
           0 2016-02-02 05:34:02 Desktop/Untitled Folder
         344 2016-02-05 04:31:17 keystone_rcadmin
          39 2016-02-10 07:06:10 object1
          27 2016-02-10 07:06:11 object2
          74 2016-02-10 07:08:25 object3
          39 2016-02-10 15:54:02 test-upload-swift/object1
   104857600 2016-02-10 15:54:02 test-upload-swift/test_file_01
   104857600 2016-02-10 15:54:02 test-upload-swift/test_file_02
   209716921

# touch test-upload-swift/object1

# time swift -v -A http://x.x.x.76:35357/v3 upload -c container2 test-upload-swift/
test-upload-swift/test_file_01
test-upload-swift/test_file_02
test-upload-swift/object1

real 0m1.265s
user 0m0.597s
sys 0m0.101s

# swift list -l container2
        1198 2016-02-02 05:37:22 Desktop/BLOB_to_FILE.txt
           0 2016-02-02 05:34:02 Desktop/Untitled Folder
         344 2016-02-05 04:31:17 keystone_rcadmin
          39 2016-02-10 07:06:10 object1
          27 2016-02-10 07:06:11 object2
          74 2016-02-10 07:08:25 object3
          39 2016-02-10 15:55:49 test-upload-swift/object1
   104857600 2016-02-10 15:54:02 test-upload-swift/test_file_01
   104857600 2016-02-10 15:54:02 test-upload-swift/test_file_02
   209716921

# ls -ltr test-upload-swift/*
-rw-r--r-- 1 root root 104857600 Feb 10 10:53 test-upload-swift/test_file_02
-rw-r--r-- 1 root root 104857600 Feb 10 10:53 test-upload-swift/test_file_01
-rw-r--r-- 1 root root 39 Feb 10 10:55 test-upload-swift/object1

# rpm -qa | grep swift
openstack-swift-plugin-swift3-1.7-4.el7.noarch
openstack-swift-container-2.5.0-1.el7.noarch
openstack-swift-proxy-2.5.0-1.el7.noarch
python-swiftclient-2.6.0-1.el7.noarch
openstack-swift-2.5.0-1.el7.noarch