object save is very slow

[Steve Martinelli]

See the mailing list thread http://lists.openstack.org/pipermail/openstack/2017-January/018227.html

in short, `openstack object save` is significantly slower than `swift download`.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/417490

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Martinelli (<email address hidden>) on branch: master
Review: https://review.openstack.org/417490

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/417892
Committed: https://git.openstack.org/cgit/openstack/python-openstackclient/commit/?id=1cdc1319d6cbfc4087551e5bf0a9875c016eca1c
Submitter: Jenkins
Branch: master

commit 1cdc1319d6cbfc4087551e5bf0a9875c016eca1c
Author: Jordan Pittier <email address hidden>
Date: Mon Jan 9 15:40:59 2017 +0100

    Make 'object save' fast again

    'openstack object save' is critically slow to download big objects. While
    we 'stream' (chunked download) the data, the default chunks_size
    is 1 byte [1], which is terribly inefficient.

    [1] : http://docs.python-requests.org/en/master/api/#requests.Response.iter_content

    Closes-Bug: 1654645

    Change-Id: I2223e5897346acd2f2c1fae638d1193cff833c19

[Christian Schwede (cschwede)]

Unfortunately this patch doesn't fix the issue (though it should help after the issue itself got fixed).

The slow part is happening a few lines before: https://github.com/openstack/python-openstackclient/blob/master/openstackclient/api/object_store_v1.py#L372-L377

I just run a download of a 10MB object using the merged patch from openstackclient, and it took 204 seconds (compared to around 2 seconds with the swift CLI).

I continue investigating this.

[Steve Martinelli (stevemar)]

Christian, jordanp mentioned that https://review.openstack.org/#/c/416249/6 was also needed, so he installed the master version of keystoneauth as well.

[Jordan Pittier (jordan-pittier)]

I confirm. You need https://review.openstack.org/#/c/416249/6 to see the real benefit of increasing the chunk size.

[Christian Schwede (cschwede)]

Ah, nice - I just stumbled upon the same this minute. Thanks!

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-openstackclient 3.7.0 release.

[Amit Bhardwaj (bhardwajamit23)]

How can I get this in Newton release?
Latest release of python-openstackclient on my machine is 3.2.0 and I am also facing the same issue. Downloads are very slow.

[Steve Martinelli (stevemar)]

install python-openstackclient in a virtualenv so you can avoid dependency hell and mixing server releases with client releases:

  $ virtualenv osc
  $ source osc/bin/activate
  (osc) $ pip install python-openstackclient (should install latest)

[Amit Bhardwaj (bhardwajamit23)]

Thanks Steve.
It seems the problem is somewhere else. Even after using latest openstackclient, download is very slow.