GET /v1/resources/generic has no facility to limit output

[Chris Dent]

I've been running a gnocchi+ceilometer test env with 5 nova instances and a polling interval of 5 seconds. It's been up for 2-3 hours.

Requesting GET /v1/resources/generic?history=true results in a 23MB JSON file, the generation of which caused my cpus fans to kick in.

Some of this problem will be resolved by the fixes currently being worked on in the gnocchi dispatcher for ceilometer.

However in a situation where there are legitimately many resources or many resource history records, it would be useful to be able to limit things explicitly and perhaps limit things tacitly (as ceilometer now does with the automatic result set limiting).

The "standard" appears to be some kind of pagination with 'limit' and 'marker' parameters.

[Chris Dent (cdent)]

The other impact of no limit is that if you do something like:

/v1/resources/instance?history=true

when there are lots of instances and lots of history you can very easily create a very large httpd process. I currently have two which are over 5GB. Python won't give this back to the OS so it would be useful to document that mod_wsgi installation should set some parameters so that it restarts itself: https://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIDaemonProcess

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/209438

[Mehdi Abaakouk (sileht)]

Nova have an osapi_max_limit = 1000 to avoid such issue. I plan to do the same for gnocchi

[Chris Dent (cdent)]

Gordon implemented something similar to the nova thing across the extent of the ceilo api and made the limit quite low. The idea being if that people want a lot they should have to ask for it explicitly.

https://review.openstack.org/#/q/status:merged+project:openstack/ceilometer+branch:master+topic:bp/mandatory-limit,n,z

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/209438
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=f2f31d79e83cb0cf302a2a0216562d1d9768a1e9
Submitter: Jenkins
Branch: master

commit f2f31d79e83cb0cf302a2a0216562d1d9768a1e9
Author: Mehdi Abaakouk <email address hidden>
Date: Wed Aug 5 09:18:17 2015 +0200

    Implements list_resources limit/ordering

    This change implements limit and ordering
    for the listing of resource.

    It also adds a [api]/max_limit to limit the number of
    elements returned, to avoid too many memory consumption
    on the server side.

    Partial bug: #1479830

    Change-Id: I9f4a7557809b04934adcef268f45b0fbb4ec0826

[Mehdi Abaakouk (sileht)]

Note: this is only partially fixed, because we have currently no ways to returns the 'links' informations for pagination.

[fengchaoyang (fengchaoyang)]

Hello All:
   I believe that in order to make better use of our paging function, we need to increase this gnocchi api that GET /v1/metric?limit=100&start=100 HTTP/1.1, and it will return {"next": "/v1/metric?limit=100&start=200"},

   I think compared to /v1/metric?limit=100&marker=metricID HTTP/1.1，It will be better with the new api

   I do not know what I think right, I hope you can give me advice, thank you

[Julien Danjou (jdanjou)]

This is right feng, the problem is that it would break the current API if we change anything.

We could return the new format if a client passes a micro version header. We never implemented that so far, but I guess that'd be a solution.