"Too much data for sort() with no index." While Getting Resources

Bug #1433924 reported by Eren on 2015-03-19
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fix Released
Ilya Tyaptin
ceilometer (Ubuntu)

Bug Description

We have 47GB of mongo data. When I run "ceilometer resource-list", I get the error in the subject. I isolated the problem down to storage/impl_mongodb.py:746 _get_floating_resources(). It builds an empty query with sort options "{user_id: -1, project_id: -1, last_sample_timestamp: -1}" and it fails.

I ran the query in mongodb to see if it was generating the same error and confirmed it. After a bit digging, I learned that mongodb needs compound index for that kind of sort. I added the following index via mongocl:

db.resource.createIndex({user_id: -1, project_id: -1, last_sample_timestamp: -1}, {background: true})

After this index, the problem went away. I can run "ceilometer resource-list" and "ceilometer meter-list". However, those commands take ~1minute to complete with the index. The performance may be another issue that needs investigation.

I need to mention that this error is different than https://bugs.launchpad.net/ceilometer/+bug/1193906. The timestamp index is in place but we need compound indexes as I explained above.

gordon chung (chungg) on 2015-03-19
Changed in ceilometer:
status: New → Triaged
Changed in ceilometer:
assignee: nobody → Aleksey Ripinen (aripinen)
status: Triaged → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/207409

Changed in ceilometer:
assignee: Aleksey Ripinen (aripinen) → Ilya Tyaptin (ityaptin)

Reviewed: https://review.openstack.org/207409
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=4b8929206acdd28fecb14652d78f9ca675a4fdd4
Submitter: Jenkins
Branch: master

commit 4b8929206acdd28fecb14652d78f9ca675a4fdd4
Author: Ilya Tyaptin <email address hidden>
Date: Thu Jul 30 14:18:42 2015 +0300

    [MongoDB] Refactor indexes for meter and resources

    Currently Ceilometer uses redundant indexes in MongoDB and in same
    time doesn't cover get_resources sorting by indexes.
    In this CR source field indexing is deleted and resource.project_id,
    resource.user_id indexes are combined to one.
    New compound index covers the get_resource sorting and its parts
    are used in queries by timestamp, project_id, user_id.

    Change-Id: I077317533fd30865eea8a3f32e3c3a7f53fa6343
    Closes: bug 1433924

Changed in ceilometer:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2015-09-03
Changed in ceilometer:
milestone: none → liberty-3
status: Fix Committed → Fix Released
James Page (james-page) wrote :

b3 has been uploaded to wily - setting to Fix Released.

Changed in ceilometer (Ubuntu Wily):
importance: Undecided → Medium
status: New → Fix Released
Changed in ceilometer (Ubuntu Vivid):
status: New → Triaged
Changed in ceilometer (Ubuntu Trusty):
status: New → Triaged
importance: Undecided → Medium
Changed in ceilometer (Ubuntu Vivid):
importance: Undecided → Medium
Thierry Carrez (ttx) on 2015-10-15
Changed in ceilometer:
milestone: liberty-3 → 5.0.0

Change abandoned by gordon chung (<email address hidden>) on branch: master
Review: https://review.openstack.org/196645
Reason: cleanup

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers