Message signatures don't match in some cases

[Yolanda Robla]

I detected a problem in the message for some metering, such as network.incoming.bytes.
The signature that is generated in ceilometer-agent-compute is different than the one that ceilometer-collector is calculating.
This is causing that the messages are rejected and don't collected by ceilometer

The different i found was in instance_type_metadata. In ceilometer-agent-compute it's generated like (u'1',) and in collector it arrives like [u'1'].

Complete message extraction and generated signatures are here: http://paste.ubuntu.com/1689662/

[Nick Barcet (nijaba)]

Thank you very much for your bug report Yolanda. In order to ensure easier reproduction of your issue, could you please specify which source you use for deploying Ceilometer and possibly the version you are using?

[Doug Hellmann (doug-hellmann)]

Julien commented in IRC that this is likely caused by "the str() of the dict" being encoded. It sounds like our dictionary flattening code, used to produce the value to go in to be signed, isn't handling recursion into data structures other than dictionaries. Early on the metadata was just nested dictionaries with strings as keys and values but apparently we've got a sequence in one of the values now. JSON encodes all sequences as lists, but the sender is actually a tuple in this case so the str() of that value comes out differently.

[Lianhao Lu (lianhao-lu)]

Steps to reproduce this issue:

Using the latest master branch of the tip commit:

commit 22b1e582016caf356a9eb76853442d84b0f70695
Author: Doug Hellmann <email address hidden>
Date: Wed Feb 20 12:14:56 2013 -0500

    Disable notifier tests

    Temporarily disable the nova notifier tests so we can work on
    decoupling the notifier from the rest of ceilometer to fix
    the issue with oslo.config.

    Change-Id: Ib5c33a37a8eed464a36724e9bb8af162af44fc5c
    Signed-off-by: Doug Hellmann <email address hidden>

1. launch ceilometer-agent-compute and ceilometer-collector
2. launch a new nova instance by "nova boot ..."
3. on ceilometer-agent-compute, the meters from NetPollster are published:

2013-02-21 11:33:18.594 21487 INFO ceilometer.compute.pollsters.net [-] NETWORK USAGE: instance-00000002 vnet0: read-bytes=462623 write-bytes=11561
2013-02-21 11:33:18.596 21487 AUDIT ceilometer.pipeline [-] Pipeline meter_pipeline: Transform counter Counter(name='network.incoming.bytes', type='cumulative', unit='B', volume=462623L, user_id=u'70d670d5a87f4f12a7eaad16067b8019', project_id=u'dae04fa4f0c54b28ad8b21875b46c079', resource_id='nova-instance-instance-00000002-fa163ee5c95f', timestamp='2013-02-21T03:33:18Z', resource_metadata={'name': 'vnet0', 'parameters': {'ip': '10.0.0.2', 'dhcpserver': '10.0.0.1'}, 'fref': 'nova-instance-instance-00000002-fa163ee5c95f', 'instance_id': u'0d889630-5dc0-4bd3-9093-aacfd8e7af7c', 'instance_type': (u'1',), 'mac': 'fa:16:3e:e5:c9:5f'}) from 0 transformer
2013-02-21 11:33:18.596 21487 AUDIT ceilometer.pipeline [-] Pipeline meter_pipeline: Publishing counters

4. on ceilometer-collector, the signature invalid error occured:

2013-02-21 11:33:18.686 INFO ceilometer.collector.service [req-b9ae7011-958f-4d34-8f33-a40fa10bdcce admin None] metering data network.incoming.bytes for nova-instance-instance-00000002-fa163ee5c95f @ 2013-02-21T03:33:18Z: 462623
2013-02-21 11:33:18.687 WARNING ceilometer.collector.service [req-b9ae7011-958f-4d34-8f33-a40fa10bdcce admin None] message signature invalid, discarding message: {u'counter_name': u'network.incoming.bytes', u'user_id': u'70d670d5a87f4f12a7eaad16067b8019', u'message_signature': u'f03ce4a86cd5d84cbfacc42bfcf9dc6bad6f8305787c1a3b8d59a2c690bb8d62', u'timestamp': u'2013-02-21T03:33:18Z', u'resource_id': u'nova-instance-instance-00000002-fa163ee5c95f', u'message_id': u'6f0f6d46-7bd7-11e2-bc04-5e7de0408e29', u'source': u'openstack', u'counter_unit': u'B', u'counter_volume': 462623, u'project_id': u'dae04fa4f0c54b28ad8b21875b46c079', u'resource_metadata': {u'name': u'vnet0', u'parameters': {u'ip': u'10.0.0.2', u'dhcpserver': u'10.0.0.1'}, u'fref': u'nova-instance-instance-00000002-fa163ee5c95f', u'instance_id': u'0d889630-5dc0-4bd3-9093-aacfd8e7af7c', u'instance_type': [u'1'], u'mac': u'fa:16:3e:e5:c9:5f'}, u'counter_type': u'cumulative'}

[Lianhao Lu (lianhao-lu)]

Ah, the root cause is the evil "," in the following function NetPollster.make_vnic_counter():

resource_metadata['instance_type'] = \
            instance.flavor['id'] if instance.flavor else None,

The trailing ',' change it from a unicode type to tuple type.

However, the limitation Doug mentioned in comment #2 is still valid.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/22526

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/22526
Committed: http://github.com/openstack/ceilometer/commit/1fc532371d073a565a38d6bce881dc6dbab61b51
Submitter: Jenkins
Branch: master

commit 1fc532371d073a565a38d6bce881dc6dbab61b51
Author: Lianhao Lu <email address hidden>
Date: Thu Feb 21 17:44:02 2013 +0800

    Support list/tuple as meter message value.

    Support generating meter signatures for meter messages with values of
    list/tuple type.

    Also fixed bug #1130601.

    Change-Id: Iee1951196983c1e4f941fc0c6303c3c1bd3dcc0d