"Too much data for sort() with no index." While Getting Resources

[Eren]

We have 47GB of mongo data. When I run "ceilometer resource-list", I get the error in the subject. I isolated the problem down to storage/impl_mongodb.py:746 _get_floating_resources(). It builds an empty query with sort options "{user_id: -1, project_id: -1, last_sample_timestamp: -1}" and it fails.

I ran the query in mongodb to see if it was generating the same error and confirmed it. After a bit digging, I learned that mongodb needs compound index for that kind of sort. I added the following index via mongocl:

db.resource.createIndex({user_id: -1, project_id: -1, last_sample_timestamp: -1}, {background: true})

After this index, the problem went away. I can run "ceilometer resource-list" and "ceilometer meter-list". However, those commands take ~1minute to complete with the index. The performance may be another issue that needs investigation.

I need to mention that this error is different than https://bugs.launchpad.net/ceilometer/+bug/1193906. The timestamp index is in place but we need compound indexes as I explained above.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/207409

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/207409
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=4b8929206acdd28fecb14652d78f9ca675a4fdd4
Submitter: Jenkins
Branch: master

commit 4b8929206acdd28fecb14652d78f9ca675a4fdd4
Author: Ilya Tyaptin <email address hidden>
Date: Thu Jul 30 14:18:42 2015 +0300

    [MongoDB] Refactor indexes for meter and resources

    Currently Ceilometer uses redundant indexes in MongoDB and in same
    time doesn't cover get_resources sorting by indexes.
    In this CR source field indexing is deleted and resource.project_id,
    resource.user_id indexes are combined to one.
    New compound index covers the get_resource sorting and its parts
    are used in queries by timestamp, project_id, user_id.

    Change-Id: I077317533fd30865eea8a3f32e3c3a7f53fa6343
    Closes: bug 1433924

[James Page (james-page)]

b3 has been uploaded to wily - setting to Fix Released.

[OpenStack Infra (hudson-openstack)]

Change abandoned by gordon chung (<email address hidden>) on branch: master
Review: https://review.openstack.org/196645
Reason: cleanup