Add functionality to python-cinderclient to list > osapi_max_limit (by default 1000 volumes)

Bug #1342192 reported by Gerry Fahy
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Wishlist
Vincent Hou
python-cinderclient
Fix Released
Wishlist
Vincent Hou

Bug Description

My understanding is that functionality was added to the Cinder API to allow a user to list > 1000 volumes (by setting a marker and allowing the user to request a further page of volumes starting at that marker).

I do not yet see that functionality in python-cinderclient 1.0.9 - is it being added to the next version of python-cinderclient ?

Revision history for this message
Vincent Hou (houshengbo) wrote :

Hi Gerry.
This functionality has been added into python-cinderclient via the patch https://review.openstack.org/#/c/98010/.
Parameters like marker, limit, sort_key and sort_dir have been implemented for API v2.0.

tags: added: pagination
Revision history for this message
Duncan Thomas (duncan-thomas) wrote :

I'm not sure the fix referenced is really a fix.... exposing the REST API is rather different to making the client friendly to use. Currently, if you have 500 volumes and type 'cinder list' then they all show up. If you have 1500 then only 2/3 of them show up and there is no indication that you should go use some weird syntax to see the rest. We should just fix 'cinder list' to show them all (same with snapshots, backups and any other list operation)

Changed in cinder:
status: New → Confirmed
importance: Undecided → Wishlist
Mike Perez (thingee)
tags: added: low-hanging-fruit
removed: pagination
Changed in python-cinderclient:
status: New → Confirmed
importance: Undecided → Wishlist
Vincent Hou (houshengbo)
Changed in cinder:
assignee: nobody → Vincent Hou (houshengbo)
Changed in python-cinderclient:
assignee: nobody → Vincent Hou (houshengbo)
Revision history for this message
Vincent Hou (houshengbo) wrote :

So far, if we request to list 500 out of 1500 volumes via the parameter 'limit', for example, we will get a reference link 'next' returned in the response to contain the link to request the next page. The size of the next page remains the same to the previous one, till we reach the last page. I guess what we need to do is to extend the support to other list operation for snapshot, backup, etc.

The pagination spec can be found here: http://docs.openstack.org/api/openstack-compute/2/content/Paginated_Collections-d1e664.html

The method _get_collection_links in cinder/api/common.py is used to provide the 'next' link, and 'volumes = self._view_builder.detail_list(req, limited_list)' and 'volumes = self._view_builder.summary_list(req, limited_list)' in the method _get_volumes in cinder/api/v2/volumes.py, will make sure the 'next' link is returned if available.

Please check:
https://github.com/openstack/cinder/blob/master/cinder/api/common.py#L251
https://github.com/openstack/cinder/blob/master/cinder/api/v2/volumes.py#L247
https://github.com/openstack/cinder/blob/master/cinder/api/v2/volumes.py#L249

Duncan, Gerry, Mike... do you think it makes sense?

Revision history for this message
Duncan Thomas (duncan-thomas) wrote :

Hi Vincent

This bug is specifically that python-cinderclient does not make use of the next links to return more data - you can use weird command line options to get at more, but the client should just list all of the volumes if asked to, driving the API appropriately.

Vincent Hou (houshengbo)
summary: - Add functionality to python-cinderclient to list > 1000 volumes
+ Add functionality to python-cinderclient to list > osapi_max_limit (by
+ default 1000 volumes)
Revision history for this message
Vincent Hou (houshengbo) wrote :

If we have more volumes than the limit set by osapi_max_limit, cinder client should query multiple times to retrieve all the volumes instead of requesting once.

Revision history for this message
Vincent Hou (houshengbo) wrote :

It seems that there is no need to change the cinder side, only the cinder client needs to add automatic queries to get all the available volume list, if one response is unable to contain all.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-cinderclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/134133

Changed in python-cinderclient:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-cinderclient (master)

Reviewed: https://review.openstack.org/134133
Committed: https://git.openstack.org/cgit/openstack/python-cinderclient/commit/?id=592099475c5c02d281abe036b0a4fa1d6a0a6473
Submitter: Jenkins
Branch: master

commit 592099475c5c02d281abe036b0a4fa1d6a0a6473
Author: Vincent Hou <email address hidden>
Date: Thu Nov 13 00:19:28 2014 -0800

    List all the request items when the list is over osapi_max_limit

    Convert the function _list into a loop function, which can retrieve
    the items from the next link till the limit or the end of items has
    been reached. This works for v2 only.

    So far, only volume list in v2 support limit. The limit parameter
    work for volume list in v2 only, but other list can extend it in
    future work.

    Change-Id: I011f0ed1a4ab639f67db6cae580d978c0b44c1bb
    closes-bug: #1342192

Changed in python-cinderclient:
status: In Progress → Fix Committed
Revision history for this message
Vincent Hou (houshengbo) wrote :

Since the client side fixes this issue, the server side does not need to change any more.

Changed in cinder:
status: Confirmed → Fix Committed
Thierry Carrez (ttx)
Changed in cinder:
milestone: none → kilo-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: kilo-3 → 2015.1.0
Changed in python-cinderclient:
milestone: none → 1.3.1
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.