Failure to insert image documents containing tags with . or $

Bug #1360240 reported by Dina Belova
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Medium
Igor Degtiarov
Mirantis OpenStack
Won't Fix
High
MOS Maintenance

Bug Description

Savanna [Glance] image tags require a Hadoop version for determing the correct base image to suggest for launching a cluster. These tags if they contain special charaters (most noteably '.') cause issue with the MongoDB that Ceilometer uses. Currently, I have hacked the image tag information to sub all special charaters to '_', and will submit this as a bug if that is the next step for this question.

The details are: once an image is inserted in Glance, Ceilometer starts collecting meters about the image. In the details that it collects about the image are all its properties. Ceilometer stores the meters in MongoDB. If a property name (not value) has a dot and/or dollar symbol in it, as you saw in case of Savanna, the insert of the meter into MongoDB fails.

Below is a document (called meter) that Ceilometer tries to insert into MongoDB for an image. Notice that it is a recursive set of key, value pairs, and in the hierarchy path resource_metadata.properties, there is a key (_savanna_...) that has a dot in it. MongoDB refuses to accept the insert when Ceilometer tries to save the data. It never will.

Actually any meter (not just for Glance images) that have a key name containing a dot or dollar will not get in to MongoDB. But, OS services themselves (like Glance) may accept key names with dots and dollars. That is the crux of the issue. The only way to solve it to not have property names with the forbidden MongoDB characters.

{
     'counter_name': 'image.size',
     'user_id': None,
     'message_signature': 'ff2881317978023f21b28710d6dbed18fd1ddb83af6d75cd3a601e2ec6719136',
     'timestamp': datetime.datetime(2013,10,29,6,50,15),
     'resource_id': '28cf4ad9-75ad-42e9-b230-cf98bfa64c70',
     'message_id': '5e32cef6-4066-11e3-b649-78e3b50a8129',
     'source': 'openstack',
     'counter_unit': 'B',
     'counter_volume': 729067520,
     'project_id': 'f1e1f2f638e54627aa0c86a3acd81d4a',
     'resource_metadata': {
          'status': 'active',
          'name': 'XXX.image',
          'deleted': False,
          'container_format': 'bare',
          'created_at': '2013-10-28T23: 57: 56',
          'disk_format': 'iso',
          'updated_at': '2013-10-29T00: 13: 14',
          'properties': {
               '_savanna_tag_1.2.1': 'false',
               'metadata_savanna_tag_1.2.1': 'false'
          },
          'protected': False,
          'checksum': 'e235b63c02644e219b7bf3668f479c9e',
          'min_disk': 8,
          'is_public': True,
          'deleted_at': None,
          'min_ram': 512,
          'size': 729067520
     },
     'counter_type': 'gauge'
}

Dina Belova (dbelova)
Changed in mos:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Dina Belova (dbelova) wrote :
tags: added: sahara
Changed in mos:
assignee: nobody → Igor Degtiarov (idegtiarov)
Changed in mos:
milestone: none → 6.0
tags: added: release-notes
Revision history for this message
Dina Belova (dbelova) wrote :

Known issue
==========

MongoDB does not support storing objects (dictionaries) with keys, containing '.' and '$' - these symbols are special characters for this database, that's why when Ceilometer is processing data samples, containing, for instance, resource metadata with dots in the tag names, that leads to the sample writing failure. That usually occurs if metric is collected from the images with special tags (like Sahara is creating images with tags like '_sahara_tag_1.2.1').

Workaround
==========

Don't create cloud resources (images, VMs, etc.) containing resource metadata keys with forbidden characters.

Revision history for this message
Dina Belova (dbelova) wrote :

#release-notes-new-variant

Known issue
==========

MongoDB does not support storing objects (dictionaries) with keys, containing '.' and '$' - these symbols are special characters for this database, that's why when Ceilometer is processing data samples, containing, for instance, resource metadata with dots in the tag names, that leads to the sample writing failure. That usually occurs if metric is collected from the images with special tags (like Sahara is creating images with tags like '_sahara_tag_1.2.1'). All data samples, that do not contain these forbidden symbols, will be processed as usual without any problems.

Workaround
==========

Don't create cloud resources (images, VMs, etc.) containing resource metadata keys with forbidden characters

tags: added: docs
Eoghan Glynn (eglynn)
Changed in mos:
status: Confirmed → In Progress
Changed in ceilometer:
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Igor Degtiarov (idegtiarov)
milestone: none → juno-rc1
Revision history for this message
Lianhao Lu (lianhao-lu) wrote :
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Ceilometer team: this bug is only a duplicate of bug #1246264 in Ceilometer project, but it's not a duplicate of that Ceilometer bug in MOS project. Please update Ceilometer milestone to Kilo to match bug #1246264, but please don't close it as duplicate again, we need to track its status separately for MOS. Apologies about the confusion.

Changed in ceilometer:
status: In Progress → Fix Released
Dina Belova (dbelova)
Changed in mos:
status: In Progress → Fix Released
Revision history for this message
Dmitry Sutyagin (dsutyagin) wrote :

Bug found by customer, not fixed in 6.0.

Changed in mos:
status: Fix Released → Confirmed
milestone: 6.0 → 6.0-updates
assignee: Igor Degtiarov (idegtiarov) → MOS Maintenance (mos-maintenance)
importance: Medium → High
tags: added: customer-found support
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/ceilometer (openstack-ci/fuel-6.0/2014.2)

Fix proposed to branch: openstack-ci/fuel-6.0/2014.2
Change author: Igor Degtiarov <email address hidden>
Review: https://review.fuel-infra.org/17690

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/ceilometer (openstack-ci/fuel-6.0-updates/2014.2)

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: Igor Degtiarov <email address hidden>
Review: https://review.fuel-infra.org/17693

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/ceilometer (openstack-ci/fuel-6.0/2014.2)

Fix proposed to branch: openstack-ci/fuel-6.0/2014.2
Change author: Igor Degtiarov <email address hidden>
Review: https://review.fuel-infra.org/17694

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/ceilometer (openstack-ci/fuel-6.0/2014.2)

Change abandoned by Igor Degtiarov <email address hidden> on branch: openstack-ci/fuel-6.0/2014.2
Review: https://review.fuel-infra.org/17690
Reason: It is duplicate with 17694

Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

Set this bug's status to Won't fix (wontfix-low) since 6.0 is now in Limited support status and only security fixes are accepted.

Changed in mos:
status: Confirmed → Won't Fix
tags: added: wontfix-low
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote :

Change abandoned by Igor Degtiarov <email address hidden> on branch: openstack-ci/fuel-6.0/2014.2
Review: https://review.fuel-infra.org/17694

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/ceilometer (openstack-ci/fuel-6.0-updates/2014.2)

Change abandoned by Igor Degtiarov <email address hidden> on branch: openstack-ci/fuel-6.0-updates/2014.2
Review: https://review.fuel-infra.org/17693

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.