Handle paginated responses potentially being disabled

[Brian Curtin]

Resources which respond with paginated bodies can have the pagination ability disabled. Currently, Neutron on devstack seems to do this, and Britt Houser reported this being the case in his internal environment.

According to at least the Neutron docs, pagination being enabled or disabled is not discoverable. If you make a request to /networks and pass parameters like limit or marker, they'll be accepted if pagination is enabled, or silently discarded if pagination is disabled. Silently discarding is where the problem lies.

In the disabled case, it causes our list generator to enter an infinite loop. We get back a body, make another call with the marker param set to the last ID in the list, get back the same body, set that same marker again, and on and on and on.

A possibility to solve this would be adding a User Preference to set whether or not pagination is supported, but this means we'd need to choose a default of our own. It also puts an additional configuration burden on the user to know this, and has potentially strong implications if set improperly.

* Disabled by default means we don't take advantage of what seem to be most services enabling pagination by default, thus causing larger responses and slower response time.
* Enabled by default means what we currently have, where some things may cause infinite loops or otherwise be broken.

One possible way of being smart about this would be to shift when we start yielding responses. Currently we make a request, get a response, yield from that response, then make another request. We can determine if pagination isn't supported by holding off on yielding until we've made the next request, e.g., req1, resp1, req2, if resp1 != resp2 then yield resp1, req3, if resp2 != resp3 then yield resp2, etc.

The User Preference part will work, but I'm going to see if the self-discovery part is feasible. It might be weird, it might work.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Brian Curtin (<email address hidden>) on branch: master
Review: https://review.openstack.org/147686
Reason: This is too heavy-handed to be a general purpose solution. The way forward is to use page for one page responses, and list for paginated responses. We'll make them look the same in the higher level so in the end the problem is solved.

[Everett Toews (everett-toews)]

You might want to have a look at the Zaqar API for listing messages.

https://wiki.openstack.org/wiki/Zaqar/specs/api/v1.1#List_Messages

You can literally get an endless stream of messages. e.g. A producer(s) is putting messages on the queue faster than a consumer(s).

I mention it because such a condition *might* impact your design here. Could be worthwhile to consider it when going through your design phase.

[Brian Curtin (brian.curtin)]

That's something we'll have to handle but it'll need to be an override. Making list do two things - paginated and non-paginted - is already hard enough. We're on the third or maybe fourth iteration of it and we went from magic discovery to being sort of smart to putting it on the caller (really, the dev who is implementing) to know what the service is supporting. Each iteration got less fancy and worked better.

The change in https://review.openstack.org/#/c/156664/ is probably the best we can get for a base class. True if it's paginated, False if it isn't. It's simple and it works, and it can be hidden from the user (most cases, where pagination is explicitly not supported) or have the flag passed in from above (when pagination is supported but can be disabled).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/156664
Committed: https://git.openstack.org/cgit/stackforge/python-openstacksdk/commit/?id=d76624895c27121d514c2f465f893e9a18dda4a3
Submitter: Jenkins
Branch: master

commit d76624895c27121d514c2f465f893e9a18dda4a3
Author: Brian Curtin <email address hidden>
Date: Tue Feb 17 10:57:10 2015 -0600

    Support listing non-paginated response bodies

    We previously tried to discover whether or not a resource supported
    pagination by making extra calls, and then another attempt was made to
    separate the idea of paginated and unpaginated by making separate calls
    and modifying Resource.page. We can instead pass in a pagination
    parameter to list, defaulting to True, which remains the one call to
    make in order to be yielded Resource objects.

    Change-Id: Ie97ef5b43fcabef145ab2758082ae9f2f400a0ab
    Related-Bug: 1410980