Failure to insert image documents containing tags with . or $

Bug #1246264 reported by Matthew Farrellee
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fix Released
Igor Degtiarov

Bug Description

Savanna [Glance] image tags require a Hadoop version for determing the correct base image to suggest for launching a cluster. These tags if they contain special charaters (most noteably '.') cause issue with the MongoDB that Ceilometer uses. Currently, I have hacked the image tag information to sub all special charaters to '_', and will submit this as a bug if that is the next step for this question.

The details are: once an image is inserted in Glance, Ceilometer starts collecting meters about the image. In the details that it collects about the image are all its properties. Ceilometer stores the meters in MongoDB. If a property name (not value) has a dot and/or dollar symbol in it, as you saw in case of Savanna, the insert of the meter into MongoDB fails.

Below is a document (called meter) that Ceilometer tries to insert into MongoDB for an image. Notice that it is a recursive set of key, value pairs, and in the hierarchy path, there is a key (_savanna_...) that has a dot in it. MongoDB refuses to accept the insert when Ceilometer tries to save the data. It never will.

Actually any meter (not just for Glance images) that have a key name containing a dot or dollar will not get in to MongoDB. But, OS services themselves (like Glance) may accept key names with dots and dollars. That is the crux of the issue. The only way to solve it to not have property names with the forbidden MongoDB characters.

     'counter_name': 'image.size',
     'user_id': None,
     'message_signature': 'ff2881317978023f21b28710d6dbed18fd1ddb83af6d75cd3a601e2ec6719136',
     'timestamp': datetime.datetime(2013,10,29,6,50,15),
     'resource_id': '28cf4ad9-75ad-42e9-b230-cf98bfa64c70',
     'message_id': '5e32cef6-4066-11e3-b649-78e3b50a8129',
     'source': 'openstack',
     'counter_unit': 'B',
     'counter_volume': 729067520,
     'project_id': 'f1e1f2f638e54627aa0c86a3acd81d4a',
     'resource_metadata': {
          'status': 'active',
          'name': 'XXX.image',
          'deleted': False,
          'container_format': 'bare',
          'created_at': '2013-10-28T23: 57: 56',
          'disk_format': 'iso',
          'updated_at': '2013-10-29T00: 13: 14',
          'properties': {
               '_savanna_tag_1.2.1': 'false',
               'metadata_savanna_tag_1.2.1': 'false'
          'protected': False,
          'checksum': 'e235b63c02644e219b7bf3668f479c9e',
          'min_disk': 8,
          'is_public': True,
          'deleted_at': None,
          'min_ram': 512,
          'size': 729067520
     'counter_type': 'gauge'

Revision history for this message
Matthew Farrellee (mattf) wrote :
affects: savanna → ceilometer
Revision history for this message
gordon chung (chungg) wrote :

to play devil's advocate, i would question whether the fix should be done in Ceilometer as it is just a collector/listener. from an audit view, if we store the data different from how we receive it, it breaks the integrity of the record since it's been tampered with (although knowingly).

also, the overhead of going through every key of every record and normalising it makes me believe it there's a better way... (is there a better way in Ceilometer?)

Julien Danjou (jdanjou)
Changed in ceilometer:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Matthew Farrellee (mattf) wrote :

re: devil's advocate, an implementation detail from ceilometer (that it uses mongodb) should not be exposed and impact implementations of the projects ceilometer hopes to monitor

Revision history for this message
Daniele Venzano (venza) wrote :

Any news? This bug is 10 months old.

Why is mongodb the recommended DB for Ceilometer if it causes interoperability issues with another OpenStack project?
I just finished converting my installation from mysql to mongodb looking for better performance and I got them, at the price of losing valuable data. I have to go back to mysql now.

Revision history for this message
Sam Morrison (sorrison) wrote :

This is a big issue for us to, would really love a fix. Ideally the mongo storage driver in ceilometer sanitises the data?

Roman Vyalov (r0mikiam)
Changed in fuel:
assignee: nobody → MOS Ceilometer (mos-ceilometer)
Revision history for this message
Dina Belova (dbelova) wrote :

Let's track this bug for MOS in - as we'll add too many Mirantis specific comments to this one and people will get lots of useless e-mails :)

no longer affects: fuel
Eoghan Glynn (eglynn)
Changed in ceilometer:
milestone: none → kilo-1
assignee: nobody → Igor Degtiarov (idegtiarov)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Submitter: Jenkins
Branch: master

commit f379b96dd2a301d1cfa70562454ec957d8341125
Author: Igor Degtiarov <email address hidden>
Date: Mon Dec 1 15:51:25 2014 +0200

    [MongoDB] Fix bug with 'bad' chars in metadatas keys

    MongoDB has two restrictions on fields names, they cannot contain '.' and
    start with '$'. In resource_metadata field we can get any data, so it could
    be dict with any keys, and if some key contain i.e. dots sample with this
    metadata wouldn't be stored in MongoDB.

    To protect samples from missing this patch propose to check metadata
    for occurrence of dots and '$' in keys if metadata is dict, and to transform
    dict with dots before storing it in MongoDB. Transformation means that
    we make dict more nested and new node appears where there is dot in dict

    It means that if we get metadata as a dict {'a.b.c': v} before storing in
    MongoDB it would be transform in {'a': {'b': {'c': v}}}.

    Closes-Bug: 1246264
    Change-Id: I279ae67cd24d4dadd7d7e58007e7c5c297d91772

Changed in ceilometer:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: kilo-1 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.