ceilometer python client returns error response for LIST requests if mongodb has too much records

[Vadim Rovachev]

when ceilometer python client tries to return more than 500k records from mongodb, response from Ceilometer API will have error status because of timeout.

[Vadim Rovachev (vrovachev)]

This problem reproduced on OSTF tests. Bug for OSTF: https://bugs.launchpad.net/fuel/+bug/1405744

[Ivan Berezovskiy (iberezovskiy)]

If we have a lot of samples, resources written in MongoDB, requests for all of them can take a lot of time to process. And this time could be more that default timeout in ceilometerclient and in horizon (when we use tab Resource Usage). So, the best way to get meters and resources is to use timestamps, queries and limits.
So, this issue isn't bug, and anyone who want to use ceilometer, should know about this workaround.

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2014.1.3, package release == fuel5.1.2.mira8.git.a44f344.bca223d

Changeset: https://review.fuel-infra.org/4866
project: openstack/ceilometer
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Improve native MongoDB mapreduce job for statitics
status: patchset-created

Files placed on repository:
openstack-ceilometer-alarm-2014.1.3-fuel5.1.2.mira8.git.a44f344.bca223d.noarch.rpm
openstack-ceilometer-api-2014.1.3-fuel5.1.2.mira8.git.a44f344.bca223d.noarch.rpm
openstack-ceilometer-central-2014.1.3-fuel5.1.2.mira8.git.a44f344.bca223d.noarch.rpm
openstack-ceilometer-collector-2014.1.3-fuel5.1.2.mira8.git.a44f344.bca223d.noarch.rpm
openstack-ceilometer-common-2014.1.3-fuel5.1.2.mira8.git.a44f344.bca223d.noarch.rpm
openstack-ceilometer-compute-2014.1.3-fuel5.1.2.mira8.git.a44f344.bca223d.noarch.rpm
openstack-ceilometer-notification-2014.1.3-fuel5.1.2.mira8.git.a44f344.bca223d.noarch.rpm
python-ceilometer-2014.1.3-fuel5.1.2.mira8.git.a44f344.bca223d.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable-4866/centos

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2014.1.3, package release == fuel5.1.2.mira8.git.2713e58.bca223d

Changeset: https://review.fuel-infra.org/4866
project: openstack/ceilometer
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Improve native MongoDB mapreduce job for statitics
status: patchset-created

Files placed on repository:
openstack-ceilometer-alarm-2014.1.3-fuel5.1.2.mira8.git.2713e58.bca223d.noarch.rpm
openstack-ceilometer-api-2014.1.3-fuel5.1.2.mira8.git.2713e58.bca223d.noarch.rpm
openstack-ceilometer-central-2014.1.3-fuel5.1.2.mira8.git.2713e58.bca223d.noarch.rpm
openstack-ceilometer-collector-2014.1.3-fuel5.1.2.mira8.git.2713e58.bca223d.noarch.rpm
openstack-ceilometer-common-2014.1.3-fuel5.1.2.mira8.git.2713e58.bca223d.noarch.rpm
openstack-ceilometer-compute-2014.1.3-fuel5.1.2.mira8.git.2713e58.bca223d.noarch.rpm
openstack-ceilometer-notification-2014.1.3-fuel5.1.2.mira8.git.2713e58.bca223d.noarch.rpm
python-ceilometer-2014.1.3-fuel5.1.2.mira8.git.2713e58.bca223d.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable-4866/centos

[OSCI Robot (oscirobot)]

DEB package ceilometer has been built for project openstack/ceilometer
Package version == 2014.1.3, package release == fuel5.1.2~mira8+git.2713e58.bca223d

Changeset: https://review.fuel-infra.org/4866
project: openstack/ceilometer
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Improve native MongoDB mapreduce job for statitics
status: patchset-created

Files placed on repository:
ceilometer-agent-central_2014.1.3-fuel5.1.2~mira8+git.2713e58.bca223d_all.deb
ceilometer-agent-compute_2014.1.3-fuel5.1.2~mira8+git.2713e58.bca223d_all.deb
ceilometer-agent-notification_2014.1.3-fuel5.1.2~mira8+git.2713e58.bca223d_all.deb
ceilometer-alarm-evaluator_2014.1.3-fuel5.1.2~mira8+git.2713e58.bca223d_all.deb
ceilometer-alarm-notifier_2014.1.3-fuel5.1.2~mira8+git.2713e58.bca223d_all.deb
ceilometer-api_2014.1.3-fuel5.1.2~mira8+git.2713e58.bca223d_all.deb
ceilometer-collector_2014.1.3-fuel5.1.2~mira8+git.2713e58.bca223d_all.deb
ceilometer-common_2014.1.3-fuel5.1.2~mira8+git.2713e58.bca223d_all.deb
python-ceilometer_2014.1.3-fuel5.1.2~mira8+git.2713e58.bca223d_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable-4866/ubuntu

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2014.1.3, package release == fuel5.1.2.mira8.git.37c93fa.bca223d

Changeset: https://review.fuel-infra.org/4866
project: openstack/ceilometer
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Improve native MongoDB mapreduce job for statitics
status: patchset-created

Files placed on repository:
openstack-ceilometer-alarm-2014.1.3-fuel5.1.2.mira8.git.37c93fa.bca223d.noarch.rpm
openstack-ceilometer-api-2014.1.3-fuel5.1.2.mira8.git.37c93fa.bca223d.noarch.rpm
openstack-ceilometer-central-2014.1.3-fuel5.1.2.mira8.git.37c93fa.bca223d.noarch.rpm
openstack-ceilometer-collector-2014.1.3-fuel5.1.2.mira8.git.37c93fa.bca223d.noarch.rpm
openstack-ceilometer-common-2014.1.3-fuel5.1.2.mira8.git.37c93fa.bca223d.noarch.rpm
openstack-ceilometer-compute-2014.1.3-fuel5.1.2.mira8.git.37c93fa.bca223d.noarch.rpm
openstack-ceilometer-notification-2014.1.3-fuel5.1.2.mira8.git.37c93fa.bca223d.noarch.rpm
python-ceilometer-2014.1.3-fuel5.1.2.mira8.git.37c93fa.bca223d.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable-4866/centos

[OSCI Robot (oscirobot)]

DEB package ceilometer has been built for project openstack/ceilometer
Package version == 2014.1.3, package release == fuel5.1.2~mira8+git.37c93fa.bca223d

Changeset: https://review.fuel-infra.org/4866
project: openstack/ceilometer
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Improve native MongoDB mapreduce job for statitics
status: patchset-created

Files placed on repository:
ceilometer-agent-central_2014.1.3-fuel5.1.2~mira8+git.37c93fa.bca223d_all.deb
ceilometer-agent-compute_2014.1.3-fuel5.1.2~mira8+git.37c93fa.bca223d_all.deb
ceilometer-agent-notification_2014.1.3-fuel5.1.2~mira8+git.37c93fa.bca223d_all.deb
ceilometer-alarm-evaluator_2014.1.3-fuel5.1.2~mira8+git.37c93fa.bca223d_all.deb
ceilometer-alarm-notifier_2014.1.3-fuel5.1.2~mira8+git.37c93fa.bca223d_all.deb
ceilometer-api_2014.1.3-fuel5.1.2~mira8+git.37c93fa.bca223d_all.deb
ceilometer-collector_2014.1.3-fuel5.1.2~mira8+git.37c93fa.bca223d_all.deb
ceilometer-common_2014.1.3-fuel5.1.2~mira8+git.37c93fa.bca223d_all.deb
python-ceilometer_2014.1.3-fuel5.1.2~mira8+git.37c93fa.bca223d_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable-4866/ubuntu

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2014.1.3, package release == fuel5.1.2.mira8.git.9b3d718.bca223d

Changeset: https://review.fuel-infra.org/4866
project: openstack/ceilometer
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Improve native MongoDB mapreduce job for statitics
status: patchset-created

Files placed on repository:
openstack-ceilometer-alarm-2014.1.3-fuel5.1.2.mira8.git.9b3d718.bca223d.noarch.rpm
openstack-ceilometer-api-2014.1.3-fuel5.1.2.mira8.git.9b3d718.bca223d.noarch.rpm
openstack-ceilometer-central-2014.1.3-fuel5.1.2.mira8.git.9b3d718.bca223d.noarch.rpm
openstack-ceilometer-collector-2014.1.3-fuel5.1.2.mira8.git.9b3d718.bca223d.noarch.rpm
openstack-ceilometer-common-2014.1.3-fuel5.1.2.mira8.git.9b3d718.bca223d.noarch.rpm
openstack-ceilometer-compute-2014.1.3-fuel5.1.2.mira8.git.9b3d718.bca223d.noarch.rpm
openstack-ceilometer-notification-2014.1.3-fuel5.1.2.mira8.git.9b3d718.bca223d.noarch.rpm
python-ceilometer-2014.1.3-fuel5.1.2.mira8.git.9b3d718.bca223d.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable-4866/centos

[OSCI Robot (oscirobot)]

DEB package ceilometer has been built for project openstack/ceilometer
Package version == 2014.1.3, package release == fuel5.1.2~mira8+git.9b3d718.bca223d

Changeset: https://review.fuel-infra.org/4866
project: openstack/ceilometer
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Improve native MongoDB mapreduce job for statitics
status: patchset-created

Files placed on repository:
ceilometer-agent-central_2014.1.3-fuel5.1.2~mira8+git.9b3d718.bca223d_all.deb
ceilometer-agent-compute_2014.1.3-fuel5.1.2~mira8+git.9b3d718.bca223d_all.deb
ceilometer-agent-notification_2014.1.3-fuel5.1.2~mira8+git.9b3d718.bca223d_all.deb
ceilometer-alarm-evaluator_2014.1.3-fuel5.1.2~mira8+git.9b3d718.bca223d_all.deb
ceilometer-alarm-notifier_2014.1.3-fuel5.1.2~mira8+git.9b3d718.bca223d_all.deb
ceilometer-api_2014.1.3-fuel5.1.2~mira8+git.9b3d718.bca223d_all.deb
ceilometer-collector_2014.1.3-fuel5.1.2~mira8+git.9b3d718.bca223d_all.deb
ceilometer-common_2014.1.3-fuel5.1.2~mira8+git.9b3d718.bca223d_all.deb
python-ceilometer_2014.1.3-fuel5.1.2~mira8+git.9b3d718.bca223d_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable-4866/ubuntu

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2014.1.3, package release == fuel5.1.2.mira8.git.849b247.bca223d

Changeset: https://review.fuel-infra.org/4866
project: openstack/ceilometer
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Improve native MongoDB mapreduce job for statitics
status: patchset-created

Files placed on repository:
openstack-ceilometer-alarm-2014.1.3-fuel5.1.2.mira8.git.849b247.bca223d.noarch.rpm
openstack-ceilometer-api-2014.1.3-fuel5.1.2.mira8.git.849b247.bca223d.noarch.rpm
openstack-ceilometer-central-2014.1.3-fuel5.1.2.mira8.git.849b247.bca223d.noarch.rpm
openstack-ceilometer-collector-2014.1.3-fuel5.1.2.mira8.git.849b247.bca223d.noarch.rpm
openstack-ceilometer-common-2014.1.3-fuel5.1.2.mira8.git.849b247.bca223d.noarch.rpm
openstack-ceilometer-compute-2014.1.3-fuel5.1.2.mira8.git.849b247.bca223d.noarch.rpm
openstack-ceilometer-notification-2014.1.3-fuel5.1.2.mira8.git.849b247.bca223d.noarch.rpm
python-ceilometer-2014.1.3-fuel5.1.2.mira8.git.849b247.bca223d.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable-4866/centos

[OSCI Robot (oscirobot)]

DEB package ceilometer has been built for project openstack/ceilometer
Package version == 2014.1.3, package release == fuel5.1.2~mira8+git.849b247.bca223d

Changeset: https://review.fuel-infra.org/4866
project: openstack/ceilometer
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Improve native MongoDB mapreduce job for statitics
status: patchset-created

Files placed on repository:
ceilometer-agent-central_2014.1.3-fuel5.1.2~mira8+git.849b247.bca223d_all.deb
ceilometer-agent-compute_2014.1.3-fuel5.1.2~mira8+git.849b247.bca223d_all.deb
ceilometer-agent-notification_2014.1.3-fuel5.1.2~mira8+git.849b247.bca223d_all.deb
ceilometer-alarm-evaluator_2014.1.3-fuel5.1.2~mira8+git.849b247.bca223d_all.deb
ceilometer-alarm-notifier_2014.1.3-fuel5.1.2~mira8+git.849b247.bca223d_all.deb
ceilometer-api_2014.1.3-fuel5.1.2~mira8+git.849b247.bca223d_all.deb
ceilometer-collector_2014.1.3-fuel5.1.2~mira8+git.849b247.bca223d_all.deb
ceilometer-common_2014.1.3-fuel5.1.2~mira8+git.849b247.bca223d_all.deb
python-ceilometer_2014.1.3-fuel5.1.2~mira8+git.849b247.bca223d_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable-4866/ubuntu

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2014.1.3, package release == fuel5.1.2.mira8

Changeset: https://review.fuel-infra.org/4866
project: openstack/ceilometer
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Improve native MongoDB mapreduce job for statitics
status: change-merged

Files placed on repository:
openstack-ceilometer-alarm-2014.1.3-fuel5.1.2.mira8.noarch.rpm
openstack-ceilometer-api-2014.1.3-fuel5.1.2.mira8.noarch.rpm
openstack-ceilometer-central-2014.1.3-fuel5.1.2.mira8.noarch.rpm
openstack-ceilometer-collector-2014.1.3-fuel5.1.2.mira8.noarch.rpm
openstack-ceilometer-common-2014.1.3-fuel5.1.2.mira8.noarch.rpm
openstack-ceilometer-compute-2014.1.3-fuel5.1.2.mira8.noarch.rpm
openstack-ceilometer-notification-2014.1.3-fuel5.1.2.mira8.noarch.rpm
python-ceilometer-2014.1.3-fuel5.1.2.mira8.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable/centos

[OSCI Robot (oscirobot)]

DEB package ceilometer has been built for project openstack/ceilometer
Package version == 2014.1.3, package release == fuel5.1.2~mira8

Changeset: https://review.fuel-infra.org/4866
project: openstack/ceilometer
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Improve native MongoDB mapreduce job for statitics
status: change-merged

Files placed on repository:
ceilometer-agent-central_2014.1.3-fuel5.1.2~mira8_all.deb
ceilometer-agent-compute_2014.1.3-fuel5.1.2~mira8_all.deb
ceilometer-agent-notification_2014.1.3-fuel5.1.2~mira8_all.deb
ceilometer-alarm-evaluator_2014.1.3-fuel5.1.2~mira8_all.deb
ceilometer-alarm-notifier_2014.1.3-fuel5.1.2~mira8_all.deb
ceilometer-api_2014.1.3-fuel5.1.2~mira8_all.deb
ceilometer-collector_2014.1.3-fuel5.1.2~mira8_all.deb
ceilometer-common_2014.1.3-fuel5.1.2~mira8_all.deb
python-ceilometer_2014.1.3-fuel5.1.2~mira8_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable/ubuntu

[Ilya Tyaptin (ityaptin)]

Current bug was affected by https://review.fuel-infra.org/4866, that little improve map-reduce MongoDB job in statistics request. This job is written on MongoDB's verion of JS and contain required map and reduce functions. These functions process BSON (special JSON-like format) documents. For improving performance of this map-reduce job we have made following changes: we have added the usage of native javascript objects instead of BSON documents in job's functions; we have implemented presorting of documents in "meter" collection by timestamp before starting a map function.
These improvements reduce time of GET "/v2/meter/<type>/statistics" up to 30%. Using of native javascript objects in map-reduce job is configurable via ceilometer config bool parameter "mongodb_jsmode" in "database" group (by default it is True). This option could be used only if statistics periods count is less than 500 000. Otherwise (if periods' count is bigger than 500 thousands) this option should be set to False.

In same time processing of many records could take a long time that exceeds a default api timeout. You may reduce proccesing time via timestamp, resource_id or metadata queries. Also you could configure timeout for request from cli via option "--timeout" on MOS 6.0 and above.

Good luck, Ilya Tyaptin,
Junioir Ceilometer developer.

[OSCI Robot (oscirobot)]

package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2.2, package release == fuel6.1.mira6.git.d8d0e94.c277f65

Changeset: https://review.fuel-infra.org/5022
project: openstack/ceilometer
branch: openstack-ci/fuel-6.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Improve native MongoDB mapreduce job for statitics
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-5022/

[OSCI Robot (oscirobot)]

package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2.2, package release == fuel6.1~mira5+git.d8d0e94.c277f65

Changeset: https://review.fuel-infra.org/5022
project: openstack/ceilometer
branch: openstack-ci/fuel-6.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Improve native MongoDB mapreduce job for statitics
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /trusty-fuel-6.1-stable-5022/

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2014.1.3, package release == fuel5.1.2.mira9

Changeset: https://review.fuel-infra.org/5046
project: openstack/ceilometer
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ivan Berezovskiy
committer: Ivan Berezovskiy
subject: Revert \"Improve native MongoDB mapreduce job for statitics\"
status: change-merged

Files placed on repository:
openstack-ceilometer-alarm-2014.1.3-fuel5.1.2.mira9.noarch.rpm
openstack-ceilometer-api-2014.1.3-fuel5.1.2.mira9.noarch.rpm
openstack-ceilometer-central-2014.1.3-fuel5.1.2.mira9.noarch.rpm
openstack-ceilometer-collector-2014.1.3-fuel5.1.2.mira9.noarch.rpm
openstack-ceilometer-common-2014.1.3-fuel5.1.2.mira9.noarch.rpm
openstack-ceilometer-compute-2014.1.3-fuel5.1.2.mira9.noarch.rpm
openstack-ceilometer-notification-2014.1.3-fuel5.1.2.mira9.noarch.rpm
python-ceilometer-2014.1.3-fuel5.1.2.mira9.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable/centos

[OSCI Robot (oscirobot)]

DEB package ceilometer has been built for project openstack/ceilometer
Package version == 2014.1.3, package release == fuel5.1.2~mira9

Changeset: https://review.fuel-infra.org/5046
project: openstack/ceilometer
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ivan Berezovskiy
committer: Ivan Berezovskiy
subject: Revert \"Improve native MongoDB mapreduce job for statitics\"
status: change-merged

Files placed on repository:
ceilometer-agent-central_2014.1.3-fuel5.1.2~mira9_all.deb
ceilometer-agent-compute_2014.1.3-fuel5.1.2~mira9_all.deb
ceilometer-agent-notification_2014.1.3-fuel5.1.2~mira9_all.deb
ceilometer-alarm-evaluator_2014.1.3-fuel5.1.2~mira9_all.deb
ceilometer-alarm-notifier_2014.1.3-fuel5.1.2~mira9_all.deb
ceilometer-api_2014.1.3-fuel5.1.2~mira9_all.deb
ceilometer-collector_2014.1.3-fuel5.1.2~mira9_all.deb
ceilometer-common_2014.1.3-fuel5.1.2~mira9_all.deb
python-ceilometer_2014.1.3-fuel5.1.2~mira9_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable/ubuntu

[Vadim Rovachev (vrovachev)]

This problem sometime reproduced on deployed long-lived env.
Bug for this: https://bugs.launchpad.net/fuel/+bug/1405744

it's expected behavior, because we can have a huge amount of metering data from long-lived env.