object save is very slow

Bug #1654645 reported by Steve Martinelli on 2017-01-06
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
python-openstackclient
Fix Released
High
Jordan Pittier

Bug Description

See the mailing list thread http://lists.openstack.org/pipermail/openstack/2017-January/018227.html

in short, `openstack object save` is significantly slower than `swift download`.

Fix proposed to branch: master
Review: https://review.openstack.org/417490

Changed in python-openstackclient:
status: Triaged → In Progress

Change abandoned by Steve Martinelli (<email address hidden>) on branch: master
Review: https://review.openstack.org/417490

Changed in python-openstackclient:
assignee: Steve Martinelli (stevemar) → Jordan Pittier (jordan-pittier)

Reviewed: https://review.openstack.org/417892
Committed: https://git.openstack.org/cgit/openstack/python-openstackclient/commit/?id=1cdc1319d6cbfc4087551e5bf0a9875c016eca1c
Submitter: Jenkins
Branch: master

commit 1cdc1319d6cbfc4087551e5bf0a9875c016eca1c
Author: Jordan Pittier <email address hidden>
Date: Mon Jan 9 15:40:59 2017 +0100

    Make 'object save' fast again

    'openstack object save' is critically slow to download big objects. While
    we 'stream' (chunked download) the data, the default chunks_size
    is 1 byte [1], which is terribly inefficient.

    [1] : http://docs.python-requests.org/en/master/api/#requests.Response.iter_content

    Closes-Bug: 1654645

    Change-Id: I2223e5897346acd2f2c1fae638d1193cff833c19

Changed in python-openstackclient:
status: In Progress → Fix Released
Christian Schwede (cschwede) wrote :

Unfortunately this patch doesn't fix the issue (though it should help after the issue itself got fixed).

The slow part is happening a few lines before: https://github.com/openstack/python-openstackclient/blob/master/openstackclient/api/object_store_v1.py#L372-L377

I just run a download of a 10MB object using the merged patch from openstackclient, and it took 204 seconds (compared to around 2 seconds with the swift CLI).

I continue investigating this.

Steve Martinelli (stevemar) wrote :

Christian, jordanp mentioned that https://review.openstack.org/#/c/416249/6 was also needed, so he installed the master version of keystoneauth as well.

Jordan Pittier (jordan-pittier) wrote :

I confirm. You need https://review.openstack.org/#/c/416249/6 to see the real benefit of increasing the chunk size.

Christian Schwede (cschwede) wrote :

Ah, nice - I just stumbled upon the same this minute. Thanks!

This issue was fixed in the openstack/python-openstackclient 3.7.0 release.

Amit Bhardwaj (bhardwajamit23) wrote :

How can I get this in Newton release?
Latest release of python-openstackclient on my machine is 3.2.0 and I am also facing the same issue. Downloads are very slow.

Steve Martinelli (stevemar) wrote :

install python-openstackclient in a virtualenv so you can avoid dependency hell and mixing server releases with client releases:

  $ virtualenv osc
  $ source osc/bin/activate
  (osc) $ pip install python-openstackclient (should install latest)

Amit Bhardwaj (bhardwajamit23) wrote :

Thanks Steve.
It seems the problem is somewhere else. Even after using latest openstackclient, download is very slow.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers