samples from notifications for autoscaled instances cannot be persisted in mongo

Bug #1284073 reported by Eoghan Glynn
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
High
Eoghan Glynn
Havana
Fix Released
Undecided
Unassigned

Bug Description

For polled samples related to autoscaled instances, we apply a mapping to the user metadata so as to ensure that there are no embedded periods:

  https://github.com/openstack/ceilometer/blob/master/ceilometer/compute/pollsters/util.py

However, the same user metdata will also be present for notifications that related to the same instance, but the mapping is not applied in that case.

This causes the mongo driver to fail with not okForStorage when the sample derived from the notification is dispatched.

For example, when the sample derived from a compute.instance.delete.end is processed:

  ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received {u'_context_request_id': u'req-1fa1edc3-7cc2-482a-8526-7a308daacbe0', u'args': {u'data': [{u'counter_name': u'disk.root.size', u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id': u'0b900e89-cf10-44a8-a79c-f2f68f55fc07', u'timestamp': u'2014-02-24 11:59:55.295790', u'message_signature': u'49ea1df965fd09b6ce3c84812977e9c21474030006510ef402afd6b2e12c9f90', u'resource_metadata': {u'state_description': u'', u'event_type': u'compute.instance.delete.end', u'availability_zone': None, u'terminated_at': u'2014-02-24T11:59:54.965388', u'ephemeral_gb': 0, u'instance_type_id': 2, u'deleted_at': u'', u'reservation_id': u'r-8pz6nwrl', u'instance_id': u'0b900e89-cf10-44a8-a79c-f2f68f55fc07', u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname': u'tyky-group-a-wste7bd63rkl-group-a-1-xzgy6fkgqvhc', u'state': u'deleted', u'launched_at': u'2014-02-24T11:58:16.000000', u'metadata': {u'metering.server_group': u'Group_A', u'AutoScalingGroupName': u'tyky-Group_A-wste7bd63rkl', u'assign_floating_ip': u'true'}, u'node': u'node-17', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, u'access_ip_v4': None, u'kernel_id': u'', u'host': u'compute.node-17', u'display_name': u'tyky-Group_A-wste7bd63rkl-Group_A-1-xzgy6fkgqvhc', u'image_ref_url': u'http://192.168.100.4:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b', u'root_gb': 1, u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at': u'2014-02-24 11:58:08+00:00', u'memory_mb': 512, u'instance_type': u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1', u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, u'architecture': None, u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack', u'counter_unit': u'GB', u'counter_volume': 1, u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'message_id': u'2d03ed96-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]}, u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False, u'_context_tenant': None, u'_unique_id': u'6bf474db7f9f4bc3a3a4342508c54218', u'_context_is_admin': True, u'version': u'1.0', u'_context_read_only': False, u'_context_user': None, u'method': u'record_metering_data'}
<47>Feb 24 11:59:55 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked context: {'read_only': False, 'show_deleted': False, 'auth_token': '<SANITIZED>', 'is_admin': True, 'user': None, 'request_id': u'req-1fa1edc3-7cc2-482a-8526-7a308daacbe0', 'tenant': None}
<47>Feb 24 11:59:55 node-16 <U+FEFF>ceilometer-ceilometer.collector.dispatcher.database DEBUG: metering data disk.root.size for 0b900e89-cf10-44a8-a79c-f2f68f55fc07 @ 2014-02-24 11:59:55.295790: 1
<47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: Channel open
<43>Feb 24 11:59:55 node-16 ceilometer-ceilometer.collector.dispatcher.database ERROR: Failed to record metering data: not okForStorage
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py", line 65, in record_metering_data
    self.storage_conn.record_metering_data(meter)
  File "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py", line 417, in record_metering_data
    upsert=True,
  File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 487, in update
    check_keys, self.__uuid_subtype), safe)
  File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 969, in _send_message
    rv = self.__check_response_to_last_error(response)
  File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 911, in __check_response_to_last_error
    raise OperationFailure(details["err"], details["code"])
OperationFailure: not okForStorage

Eoghan Glynn (eglynn)
Changed in ceilometer:
assignee: nobody → Eoghan Glynn (eglynn)
importance: Undecided → High
milestone: none → icehouse-3
Eoghan Glynn (eglynn)
Changed in ceilometer:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/77959

Eoghan Glynn (eglynn)
tags: added: havana-backport-potential
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: icehouse-3 → icehouse-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/77959
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=ddeb54bb0d2c4203332a5ecc09a4aedfce353a4c
Submitter: Jenkins
Branch: master

commit ddeb54bb0d2c4203332a5ecc09a4aedfce353a4c
Author: Eoghan Glynn <email address hidden>
Date: Tue Mar 4 18:46:29 2014 +0000

    Ensure user metadata mapped for instance notifications

    Fixes bug 1284073

    For pollster-originated samples related to instances, we apply a
    mapping to the user metadata so as to ensure that there are no
    embedded periods in metadata keys and also to exclude any metadata
    not matching the configured reserved namespace.

    Now this logic is also applied to user metadata for samples derived
    from instance-related notifications in order to avoid failures when
    persisting these data in mongodb.

    Change-Id: I77eab59023fa23c5879b05c948bd4cb1db7fa177

Changed in ceilometer:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/84096

Thierry Carrez (ttx)
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: icehouse-rc1 → 2014.1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (stable/havana)

Reviewed: https://review.openstack.org/84096
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=b5091cab16bcb501c36725f1c4c86634732069d1
Submitter: Jenkins
Branch: stable/havana

commit b5091cab16bcb501c36725f1c4c86634732069d1
Author: Eoghan Glynn <email address hidden>
Date: Tue Mar 4 18:46:29 2014 +0000

    Ensure user metadata mapped for instance notifications

    Fixes bug 1284073

    For pollster-originated samples related to instances, we apply a
    mapping to the user metadata so as to ensure that there are no
    embedded periods in metadata keys and also to exclude any metadata
    not matching the configured reserved namespace.

    Now this logic is also applied to user metadata for samples derived
    from instance-related notifications in order to avoid failures when
    persisting these data in mongodb.

    (cherry picked from commit ddeb54bb0d2c4203332a5ecc09a4aedfce353a4c)
    Change-Id: I77eab59023fa23c5879b05c948bd4cb1db7fa177

tags: added: in-stable-havana
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.