Default ceilometer list requests fails with Gateway timeout on longliving environment

[Ilya Tyaptin]

For getting ceilometer data everybody firstly makes ceilometer list request without any filter.
These requests return all stored data by default. This behavior causes issues with Ceilometer api: timeouts, api-freezing for MOS 5.x, 6.x and impacts user experience of Ceilometer using.

So, we need to use better way for presenting data from Ceilometer db.

I suggest to backport patches which adds a configurable mandatory limit for list requests from Liberty developing cycle.
By default limit has value 1000.
It does not freeze api and affect queried requests minimally.

[Nadya Privalova (nprivalova)]

6.0 https://review.fuel-infra.org/#/q/status:open+%28project:openstack/ceilometer+OR+openstack/python-ceilometerclient%29+branch:openstack-ci/fuel-6.0-updates/2014.2+topic:mandatory-limit

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Igor Degtiarov <email address hidden>
Review: https://review.fuel-infra.org/13490

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Igor Degtiarov <email address hidden>
Review: https://review.fuel-infra.org/13491

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Igor Degtiarov <email address hidden>
Review: https://review.fuel-infra.org/13516

[Oleksandr Bryndzii (obryndzii)]

High for 6.1 due to customer-found

[Vitaly Sedelnik (vsedelnik)]

Please don't change bug importance due to the fact it was found at customer site - just adding customer-found tag is enough. We would like to have Importance field to reflect actual impact of the issue.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11890
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: 661b30acb0a6c21d5feb7f4c537560043b266989
Author: Ilya Tyaptin <email address hidden>
Date: Thu Oct 29 16:36:30 2015

Add support of mandatory limit

Now ceilometer-api has mandatory limit option
for mostly list requests.
Supporting of this option is added in this CR

Change-Id: I5ef91a7b432613a86dd3cf930dddd05f8db5eddd
Closes-Bug: #1500426

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13489
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 31372a2d702a7f60193f89fcacfb9af5ffc2edc3
Author: Igor Degtiarov <email address hidden>
Date: Thu Nov 26 13:55:40 2015

set default limit to meter/sample queries

this patch makes existing limit options in meter and samples apis
mandatory. if not set explicitly, a default limit will be applied.

DocImpact ApiImpact
Partial-Bug: #1500426
Change-Id: I5bfd2a5c1ec9a0658236acaadc168b72625d9baa
Implements: blueprint mandatory-limit
cherry-picked from 1c546b10b88256c96467c541f725fd753397e09d

[Vitaly Gusev (vgusev)]

Verified on Ubuntu 6.0.
Packages:
ceilometer-agent-central,ceilometer-agent-notification,ceilometer-alarm-evaluator,ceilometer-alarm-notifier,ceilometer-api,ceilometer-collector,ceilometer-common,python-ceilometer
Version:
2014.2-fuel6.0~mira25

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13490
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 03d6331b3b0c884a35830afd707d367707d5223e
Author: Igor Degtiarov <email address hidden>
Date: Tue Dec 1 10:23:21 2015

add mandatory limit value to meter list

unrestricted listing of meters can require significant memory.
this patch implements mandatory limit on meter-list

Partial-Bug: #1500426
Change-Id: I991c1bf0a35f5304c856a30a00997a913a690284
Implements: blueprint mandatory-limit
cherry-picked from f3c01e7528a10e9291b6879504aa0540e785fbe2

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13491
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: d3ca2934b3aca91b5b236064afa6c792eff6986f
Author: Igor Degtiarov <email address hidden>
Date: Tue Dec 1 10:23:21 2015

add mandatory limit value to resource list

unrestricted listing of resources can require significant memory.
this patch implements mandatory limit on resource-list

Closes-Bug: #1500426
Change-Id: I5a7d80f13fffecf4269b686765dd351f9bb4fb2c
Implements: blueprint mandatory-limit
cherry-picked from 1852eaf48e965a00f1fb7458cfb2ec0a2125a2ab

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13516
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 131a5ee9832979a54b882db382c5ca8df904db52
Author: Igor Degtiarov <email address hidden>
Date: Fri Oct 30 12:34:05 2015

Add support of mandatory limit

Now ceilometer-api has mandatory limit option for mostly list requests.
Supporting of this option is added in this CR

Change-Id: I0c8f5774eb95a10790971c0d1174dde67d8d6061
Closes-Bug: #1500426

[Vitaly Gusev (vgusev)]

Verified on Ubuntu 6.1
Packages:
ceilometer-agent-central,ceilometer-agent-notification,ceilometer-alarm-evaluator,ceilometer-alarm-notifier,ceilometer-api,ceilometer-collector,ceilometer-common,python-ceilometer
Version:
2014.2.2-1~u14.04+mos13
and package python-ceilometerclient with version 1.0.12-1~u14.04+mos9

[Vitaly Gusev (vgusev)]

Verified on ISO:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "466"
  build_id: "466"
  fuel-nailgun_sha: "f81311bbd6fee2665e3f96dcac55f72889b2f38c"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "6823f1d4005a634b8436109ab741a2194e2d32e0"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "fe03d887361eb80232e9914eae5b8d54304df781"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "727f7076f04cb0caccc9f305b149a2b5b5c2af3a"

[Alexey Stupnikov (astupnikov)]

Uploaded first patch to review (there will be 2 or 3 additional patches to close this bug):
https://review.fuel-infra.org/#/c/20582/2

[Alexey Stupnikov (astupnikov)]

There is a problem with py27 tests in 7.0 ceilometer's branch. I have to fix those tests first, opened bug #1581507

[Alexey Stupnikov (astupnikov)]

Estimated time for MOS7 patch-set is Thu or Fri this week.

[Alexey Stupnikov (astupnikov)]

Patchset to backport: https://review.openstack.org/#/q/topic:bp/mandatory-limit

MOS 6.0 patchset: https://review.fuel-infra.org/#/q/%28project:openstack/ceilometer+OR+openstack/python-ceilometerclient%29+branch:openstack-ci/fuel-6.0-updates/2014.2+topic:mandatory-limit

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: gordon chung <email address hidden>
Review: https://review.fuel-infra.org/21216

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: gordon chung <email address hidden>
Review: https://review.fuel-infra.org/21225

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: gordon chung <email address hidden>
Review: https://review.fuel-infra.org/21226

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: gordon chung <email address hidden>
Review: https://review.fuel-infra.org/21227

[Alexey Stupnikov (astupnikov)]

Patch works for me:
# ceilometer meter-list | wc -l
104

# ceilometer meter-list -l 10 | wc -l
14

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20665
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 5210fa45f7ebbb15273bf45340eae7480f31ed78
Author: gordon chung <email address hidden>
Date: Thu May 26 17:11:09 2016

Set default limit to meter/sample queries

This patch makes existing limit options in meter and samples apis
mandatory. if not set explicitly, a default limit will be applied.

Change-Id: Ie354e8e64a6f8677a0be7e75c34e8901cdffbc6a
Partial-Bug: #1500426

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/21216
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: db17eddd5155900498d939e2bf5074ab76b112e4
Author: gordon chung <email address hidden>
Date: Fri May 27 05:53:03 2016

add mandatory limit value to meter list

unrestricted listing of meters can require significant memory.
this patch implements mandatory limit on meter-list

Change-Id: I8b38136c2493e75fbc7d9aef75aa055463a26319
Partial-Bug: #1500426

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11040
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: ce9e32ecc573230705e5bd67209ed6b8a56dd0ae
Author: gordon chung <email address hidden>
Date: Fri May 27 07:49:35 2016

add mandatory limit value to resource list

unrestricted listing of resources can require significant memory.
this patch implements mandatory limit on resource-list

Change-Id: I5820ce759b20757febc1b92c100f584234b36b68
Partial-Bug: #1500426

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/21225
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: a981254815b6647de2b3ebed1ab388103312c7b1
Author: gordon chung <email address hidden>
Date: Fri May 27 09:40:20 2016

Add mandatory limit value to event list

Unrestricted listing of events can require significant memory.
this patch implements mandatory limit on event-list

SQL backend requires a little rewrite due to fact that we grab data
in two steps: first we grab all events matching query, then we grab
appropriate traits (we don't use orm relationships because it's slow).
we can not just append order_by and limit to first query because we
use query in second part as well and sqlite will complain about
order_by/limit coming before join. this patch changes it to use a
subquery. based on testing, this does not appear to degrade
performance.

Change-Id: Ib8009d6e4ab6d066ba5fca43b824c24c8cb32d4e
Partial-Bug: #1500426

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/21227
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 1cf4c8469a46626611d0b4167e5344cd3d9bc1c5
Author: gordon chung <email address hidden>
Date: Wed May 25 12:41:16 2016

add limit support

limit is now a mandatory condition when querying. this patch
allows users to control this value.

Change-Id: Ied5f80dc8cfe8b5f85e60fb8ca1d4a8f26663456
Closes-Bug: #1500426

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/21226
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 3b2390ed48b204b8b10859461cbe4c02b03e2fda
Author: gordon chung <email address hidden>
Date: Fri May 27 11:22:25 2016

add mandatory limit value to complex query list

unrestricted listing of data can require significant memory.
this patch implements mandatory limit on complex queries

Change-Id: I1fd21bbec63711a8887ba9363f791b6c67d75837
Partial-Bug: #1500426

[Alexander Gromov (agromov)]

Verified on MOS 7.0 mu4

Without updates:
http://paste.openstack.org/show/506990/

With updates:
http://paste.openstack.org/show/506991/

[ZHI BING WANG (zwang)]

I have the same issue on MOS 7 MU3, 200+ nodes. The ceilometer API always freezes.
Is there a workaround like to define the limit in some configuration file?
Thanks!