when polling VM cpu/instance/disk/memory metrics, populate discovered Instance's metadata into Metrics Sample

[BrianLing]

Requirement:
When polling VM resource metrics, we need reply on nova metadata to get more information (E.g. VM hostname, IP, VPC).

Hence:
We need populate nova metadata into metrics Sample. we also need backward compatible for other cases (hardware metrics) which make sure nothing is broken.

Doc Impact:
Add one more config for cared key "reserved_metadata_caredkey" in
ceilometer.conf, so user can define the key name they cared in
instance.metadata.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/134877

[Pradyumna Sampath (pradysam-deactivatedaccount)]

Can you please write up a more detailed bug description along with a method to reproduce this ?

[Zeng Ming (mizeng)]

Thanks to Fan Zhiqiang, I found ceilometer/compute/util.py has a method "add_reserved_user_metadata" can have the func with configurable getting metadata.

However, there seems some bug there that will made return a dict with empty key.
For example:

If src_metadata is like 'metadata': {'project_cos': 'dev', 'fqdn': 'localhost'}
Add I define in ceilometer.conf with "reserved_metadata_namespace=project_cos,fqdn",
then what I got with add_reserved_user_metadata method will be {'': 'localhost'}. Notice the key is empty, and there's only 1 entry rather than 2.

I was confusing with below statement:
md = dict(
            (k[len(prefix):].replace('.', '_'),
             v[:limit] if isinstance(v, six.string_types) else v)

why key have slicing operation with length of prefix? Then how could I define prefix if metadata's key didn't start with one same prefix?

So I suggest to change to below statement to make it more common to use with various naming convention key in metadata.
md = dict(
            (k.replace('.', '_'),
             v[:limit] if isinstance(v, six.string_types) else v)

[ZhiQiang Fan (aji-zqfan)]

namespace will be removed. and duplicate entry will be ignored, so from your settings, you will get {'': 'localhost'}

if you can set the resource's metadata, you can do the following:
set reserved_metadata_namespace=metering.
then if resource has metadata: {'metering.project_cos': 'dev', 'metering.fqdn': 'localhost'}
the final sample's resource_metadata can have user_metadata.project_cos: dev and user_metadata.fqdn: localhost

for more doc, read http://docs.openstack.org/developer/ceilometer/measurements.html#user-defined-sample-metadata-for-nova

[Zeng Ming (mizeng)]

Thanks Zhiqiang. I got your point. However, I still think we may add one more config for cared key, because sometimes resource will has metadata which don't have one particular prefix such as {'project_cos': 'dev', 'fqdn': 'localhost'}. Please help to review my latest patch-set.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/134877
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=568717ae53e676b8b947bf139a0e22e9ddfc094d
Submitter: Jenkins
Branch: master

commit 568717ae53e676b8b947bf139a0e22e9ddfc094d
Author: lzhijun <email address hidden>
Date: Mon Nov 17 15:27:12 2014 +0800

    Expose vm's metadata to metrics

    We have customized ceilometer publisher to send metrics into openTSDB service.
    Hence, we need more user-friendly VM information (e.g. VM hostname, IP
    and project_cos) as metrics tags during populate metrics into openTSDB.

    So we expected samples can attach nova meta data at publisher.PublisherBase's
    publish_samples function. Then it's available for customized publisher
    to access metadata.

    However the current method add_reserved_user_metadata can't support various
    naming convention which didn't have one particular prefix. So we need to
    add one more config for metadata keys "reserved_metadata_keys" in
    ceilometer.conf, and user can define the key name they cared in
    instance.metadata.

    DocImpact

    Change-Id: I588aaf22a0e8593042dad4bbc37be27adc96c2b4
    Closes-bug: #1391778