Some sample metadata are missing when we request samples via the v2 API

[Mehdi Abaakouk]

Some sample metadata are missing when we request samples via the v2 API

For example when you have some properties set on a glance image and you get all samples with this properties, this properties don't appear in the returned json.

# curl -X GET -H "X-Auth-Token: $t" -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-ceilometerclient' 'http://192.168.3.51:8777/v2/meters/image?q.op=eq&q.value=toto&q.field=metadata.properties.toto&limit=1'

[
    {
        "counter_name": "image",
        "counter_type": "gauge",
        "counter_unit": "image",
        "counter_volume": 1.0,
        "message_id": "d7ad42d8-f2b5-11e2-ac72-00163ef7ca5e",
        "project_id": "a61f3c429ef5401d82d0e6b59fb276a4",
        "resource_id": "aa4a7285-986b-4dff-bd59-bf7f9665b109",
        "resource_metadata": {
            "checksum": "f8a2eeee2dc65b3d9b6e63678955bd83",
            "container_format": "ami",
            "created_at": "2013-07-22T07:40:58",
            "deleted": "False",
            "deleted_at": "None",
            "disk_format": "ami",
            "is_public": "True",
            "min_disk": "0",
            "min_ram": "0",
            "name": "cirros-0.3.1-x86_64-uec",
            "protected": "False",
            "size": "25165824",
            "status": "active",
            "updated_at": "2013-07-22T09:24:12"
        },
        "source": "openstack",
        "timestamp": "2013-07-22T10:02:39",
        "user_id": null
    }
]

I think is related to the type declaration in the Sample v2 controller that declare resource_metadata as a dictionary of text or the value of the dictionary can be a other dictionary.

ie:
in ceilometer/api/controllers/v2.py:
'resource_metadata = {wtypes.text: wtypes.text}'

Cheers

[Feilong Wang (flwang)]

I'm not really sure this is a dup of https://bugs.launchpad.net/ceilometer/+bug/1200577, but I believe they're related.

[Doug Hellmann (doug-hellmann)]

Are you sure we are given the user-defined properties for an image when we get notifications from glance or when we poll? Maybe we're just not storing the values. For instances, we "flatten" the metadata that comes from nova so we can return a series of keypairs, and we could do the same thing for glance.

[Mehdi Abaakouk (sileht)]

image properties are in the database
the query works, it return only sample with metadata.properties.toto=toto but the returned json don't have this metadata

Same query in mongo:

MongoDB shell version: 2.4.5
connecting to: ceilometer
> db.meter.findOne({'counter_name':'image', 'resource_metadata.properties.toto':'toto'})
{
 "_id" : ObjectId("51ed36f8eb0a1975fedb9620"),
 "counter_name" : "image",
 "user_id" : null,
 "message_signature" : "fe9515719b66d0840db5ed5f58834389ee13a7c0c8caf711cc2dbd9aa90aeede",
 "timestamp" : ISODate("2013-07-22T13:43:20Z"),
 "resource_id" : "aa4a7285-986b-4dff-bd59-bf7f9665b109",
 "resource_metadata" : {
  "status" : "active",
  "name" : "cirros-0.3.1-x86_64-uec",
  "deleted" : false,
  "container_format" : "ami",
  "created_at" : "2013-07-22T07:40:58",
  "disk_format" : "ami",
  "updated_at" : "2013-07-22T09:24:12",
  "properties" : {
   "kernel_id" : "38dd825b-3b08-4cb2-9f1c-76b207c74f0e",
   "ramdisk_id" : "705c0c2a-2d3e-4da4-8740-4382bb56f8f6",
   "toto" : "toto"
  },
  "protected" : false,
  "checksum" : "f8a2eeee2dc65b3d9b6e63678955bd83",
  "min_disk" : 0,
  "is_public" : true,
  "deleted_at" : null,
  "min_ram" : 0,
  "size" : 25165824
 },
 "source" : "openstack",
 "counter_unit" : "image",
 "counter_volume" : 1,
 "project_id" : "a61f3c429ef5401d82d0e6b59fb276a4",
 "message_id" : "abfc6cc6-f2d4-11e2-ac72-00163ef7ca5e",
 "counter_type" : "gauge"
}

[Doug Hellmann (doug-hellmann)]

Ah, my mistake. The API code (https://github.com/openstack/ceilometer/blob/master/ceilometer/api/controllers/v2.py#L273) discards any metadata that is not at the top level of the dictionary. We should come up with a scheme for including the nested data.

[Feilong Wang (flwang)]

Doug, after remove "dict" from the line of https://github.com/openstack/ceilometer/blob/master/ceilometer/api/controllers/v2.py#L273, it works.

Is there any specific consideration to filter dict out? Thanks.

[Feilong Wang (flwang)]

BTW, Doug, can you help mark the "Importance" of this bug? Thanks.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/38597

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/38597
Committed: http://github.com/openstack/ceilometer/commit/149e4d64109d3b7a2ac2be19e8fc7c0546f9e542
Submitter: Jenkins
Branch: master

commit 149e4d64109d3b7a2ac2be19e8fc7c0546f9e542
Author: Fei Long Wang <email address hidden>
Date: Thu Jul 25 15:23:45 2013 +0800

    Fix the dict type metadata missing issue

    Based on current implement, the metadata with dict type will be
    missed. But some metadata are using dict type, such as the extra
    properties of Glance images.

    Fix bug 1203699

    Change-Id: Ie13e4e5231c789e0efcadace11347a15741472a1