"Too much data for sort() with no index." While Getting Resources

Bug #1433924 reported by Eren on 2015-03-19
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Undecided
Ilya Tyaptin
ceilometer (Ubuntu)
Medium
Unassigned
Trusty
Medium
Unassigned
Vivid
Medium
Unassigned
Wily
Medium
Unassigned

Bug Description

We have 47GB of mongo data. When I run "ceilometer resource-list", I get the error in the subject. I isolated the problem down to storage/impl_mongodb.py:746 _get_floating_resources(). It builds an empty query with sort options "{user_id: -1, project_id: -1, last_sample_timestamp: -1}" and it fails.

I ran the query in mongodb to see if it was generating the same error and confirmed it. After a bit digging, I learned that mongodb needs compound index for that kind of sort. I added the following index via mongocl:

db.resource.createIndex({user_id: -1, project_id: -1, last_sample_timestamp: -1}, {background: true})

After this index, the problem went away. I can run "ceilometer resource-list" and "ceilometer meter-list". However, those commands take ~1minute to complete with the index. The performance may be another issue that needs investigation.

I need to mention that this error is different than https://bugs.launchpad.net/ceilometer/+bug/1193906. The timestamp index is in place but we need compound indexes as I explained above.

gordon chung (chungg) on 2015-03-19
Changed in ceilometer:
status: New → Triaged
Changed in ceilometer:
assignee: nobody → Aleksey Ripinen (aripinen)
status: Triaged → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/207409

Changed in ceilometer:
assignee: Aleksey Ripinen (aripinen) → Ilya Tyaptin (ityaptin)

Reviewed: https://review.openstack.org/207409
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=4b8929206acdd28fecb14652d78f9ca675a4fdd4
Submitter: Jenkins
Branch: master

commit 4b8929206acdd28fecb14652d78f9ca675a4fdd4
Author: Ilya Tyaptin <email address hidden>
Date: Thu Jul 30 14:18:42 2015 +0300

    [MongoDB] Refactor indexes for meter and resources

    Currently Ceilometer uses redundant indexes in MongoDB and in same
    time doesn't cover get_resources sorting by indexes.
    In this CR source field indexing is deleted and resource.project_id,
    resource.user_id indexes are combined to one.
    New compound index covers the get_resource sorting and its parts
    are used in queries by timestamp, project_id, user_id.

    Change-Id: I077317533fd30865eea8a3f32e3c3a7f53fa6343
    Closes: bug 1433924

Changed in ceilometer:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2015-09-03
Changed in ceilometer:
milestone: none → liberty-3
status: Fix Committed → Fix Released
James Page (james-page) wrote :

b3 has been uploaded to wily - setting to Fix Released.

Changed in ceilometer (Ubuntu Wily):
importance: Undecided → Medium
status: New → Fix Released
Changed in ceilometer (Ubuntu Vivid):
status: New → Triaged
Changed in ceilometer (Ubuntu Trusty):
status: New → Triaged
importance: Undecided → Medium
Changed in ceilometer (Ubuntu Vivid):
importance: Undecided → Medium
Thierry Carrez (ttx) on 2015-10-15
Changed in ceilometer:
milestone: liberty-3 → 5.0.0

Change abandoned by gordon chung (<email address hidden>) on branch: master
Review: https://review.openstack.org/196645
Reason: cleanup

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers