user metadata for instances are inconsistently handled

[Eoghan Glynn]

Depending on the origin of the instance metadata (i.e. among one of a number different notification event_types, or from polling), the user metadata may be handled in one of at least 3 different ways by ceilometer.

1. Origin: nova notification such as compute.instance.create.{start|end}

In this case, the metadata are presented in the notification payload as the full instance_metadata representation from the nova DB, i.e. a list of dicts, one per item of user metadata, each of form:

  {"instance_uuid" : UUID,
   "deleted" : 0,
   "created_at" : TIMESTAMP,
   "updated_at" : null,
   "value" : METADATA_VALUE,
   "key" : METADATA_KEY,
   "deleted_at" : null,
   "id" : N }

and these dicts are what the collector persists in full under the sample['resource_metadata']['metadata']

2. Origin: nova notification such as compute.instance.create.update (not uniformly though, depends on the actual update).

In this case, the metadata are presented in the notification payload as a simple dict:

  {METADATA_KEY: METADATA_VALUE}

and is persisted as such under the sample['resource_metadata']['metadata']

3. Origin: polling of nova-api by the compute agent.

In this case, the metadata are again retrieved as a simple dict, e.g:

  {metering.METADATA.KEY: METADATA_VALUE}

However in this case a filtering rule is applied to exclude all user metadata other than that with a recognized prefix (default: 'metering.'), the prefix is then stripped and any embedded periods in the key are replaced with underscores ('.' -> '_') to avoid problems with mongodb failing with 'OperationFailure: not okForStorage' when these data are persisted.

The value is then persisted as such under the the sample['resource_metadata']['user_metadata.METADATA_KEY']

So the net effect is that for a single meter (e.g. 'instance' for a particular VM) user metadata can be stored or not stored in at least 3 different ways, which is highly inconsistent.

Alsoo the presence of embedded periods in the user metadata key causes the collector in case #2 to fail with 'OperationFailure: not okForStorage'.

Finally the approach adopted for case #1 is wasteful of space in the ceilometer metering store as it repeats the instance UUID and stores values such as deleted: 0 which are only of interest within the scope of the nova DB.

[Julien Danjou (jdanjou)]

I've marked this as triaged but I wonder if that was not fixed some time ago by you Eoghan?

[gordon chung (chungg)]

i feel like this is covered by: bug 1391778 and bug 1284073