mongodb failure when querying for resource-list

[Stefano Zilli]

In Havana, when using a sharded mongodb deployment resource-list queries fail with the following error:

2013-12-10 08:49:48.745 64835 ERROR wsme.api [-] Server-side error: "command SON([('aggregate', u'meter'), ('pipeline', [{'$match': {}}, {'$sort': {'timestamp': -1, 'project_id': -1, 'user_id': -1}}, {'$group': {'meters_unit': {'$push': '$counter_unit'}, 'source': {'$first': '$source'}, 'project_id': {'$first': '$project_id'}, 'user_id': {'$first': '$user_id'}, 'last_sample_timestamp': {'$max': '$timestamp'}, 'meters_name': {'$push': '$counter_name'}, 'first_sample_timestamp': {'$min': '$timestamp'}, 'meters_type': {'$push': '$counter_type'}, '_id': '$resource_id', 'metadata': {'$first': '$resource_metadata'}}}])]) failed: exception: sharded pipeline failed on shard shard_01: { errmsg: "exception: aggregation result exceeds maximum document size (16MB)", code: 16389, ok: 0.0 }". Detail:
Traceback (most recent call last):

  File "/usr/lib/python2.6/site-packages/wsmeext/pecan.py", line 72, in callfunction
    result = f(self, *args, **kwargs)

  File "/usr/lib/python2.6/site-packages/ceilometer/api/controllers/v2.py", line 965, in get_all
    for r in pecan.request.storage_conn.get_resources(**kwargs)]

  File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 650, in get_resources
    "meters_unit": {"$push": "$counter_unit"},

  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 1061, in aggregate
    _use_master=use_master)

  File "/usr/lib64/python2.6/site-packages/pymongo/database.py", line 393, in command
    msg, allowable_errors)

  File "/usr/lib64/python2.6/site-packages/pymongo/helpers.py", line 147, in _check_command_response
    raise OperationFailure(msg % errmsg, code)

OperationFailure: command SON([('aggregate', u'meter'), ('pipeline', [{'$match': {}}, {'$sort': {'timestamp': -1, 'project_id': -1, 'user_id': -1}}, {'$group': {'meters_unit': {'$push': '$counter_unit'}, 'source': {'$first': '$source'}, 'project_id': {'$first': '$project_id'}, 'user_id': {'$first': '$user_id'}, 'last_sample_timestamp': {'$max': '$timestamp'}, 'meters_name': {'$push': '$counter_name'}, 'first_sample_timestamp': {'$min': '$timestamp'}, 'meters_type': {'$push': '$counter_type'}, '_id': '$resource_id', 'metadata': {'$first': '$resource_metadata'}}}])]) failed: exception: sharded pipeline failed on shard shard_01: { errmsg: "exception: aggregation result exceeds maximum document size (16MB)", code: 16389, ok: 0.0 }

[Eoghan Glynn (eglynn)]

Marking as a dupe of LP 1262571 as that contains more detail and background.

[Eoghan Glynn (eglynn)]

Stefano, thanks for the report, can you mark bug #1262571 as affecting you also?

Thanks,
Eoghan

[Stefano Zilli (szilli)]

Done.

Thanks,
Stefano