[HBase] Error during recording metering data

[Nadya Privalova]

During metering data processing in HBase the following error occurs:

2013-10-16 13:06:37.703 15231 ERROR ceilometer.collector.dispatcher.database [req-04d52fa4-74e8-4af3-a41d-1bfe1aeed866 admin None] Failed to record metering data: expected string or Unicode object, dict found
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database Traceback (most recent call last):
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database File "/opt/stack/ceilometer/ceilometer/collector/dispatcher/database.py", line 65, in record_metering_data
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database self.storage_conn.record_metering_data(meter)
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database File "/opt/stack/ceilometer/ceilometer/storage/impl_hbase.py", line 217, in record_metering_data
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database resource_table.put(data['resource_id'], new_resource)
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database File "/usr/local/lib/python2.7/dist-packages/happybase/table.py", line 370, in put
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database batch.put(row, data)
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database File "/usr/local/lib/python2.7/dist-packages/happybase/batch.py", line 116, in __exit__
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database self.send()
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database File "/usr/local/lib/python2.7/dist-packages/happybase/batch.py", line 54, in send
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database self._table.connection.client.mutateRows(self._table.name, bms)
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database File "/usr/local/lib/python2.7/dist-packages/happybase/hbase/Hbase.py", line 1449, in mutateRows
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database self.send_mutateRows(tableName, rowBatches)
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database File "/usr/local/lib/python2.7/dist-packages/happybase/hbase/Hbase.py", line 1457, in send_mutateRows
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database args.write(self._oprot)
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database File "/usr/local/lib/python2.7/dist-packages/happybase/hbase/Hbase.py", line 6352, in write
2013-10-16 13:06:37.703 15231 TRACE ceilometer.collector.dispatcher.database oprot.trans.write(fastbinary.encode_binary(self, (self.__class__, self.thrift_spec)))

The same error about Boolean was hit too.
The problem here is that data which arrives to record_metering_data may be the following:

data = {u'f:r_deleted_at': None, u'f:r_checksum': u'c352f4e7121c6eae958bc1570324f17e', 'f:source': u'openstack', u'f:r_updated_at': u'2013-10-14T09:30:41', u'f:r_created_at': u'2013-10-14T09:30:40', u'f:r_name': u'cirros-0.3.1-x86_64-uec-kernel', 'f:user_id': None, u'f:r_properties': {}, 'f:resource_id': u'6df1edeb-863e-4647-bcb9-9b41e280548d', u'f:r_is_public': 'True', u'f:r_container_format': u'aki', u'f:r_protected': 'False', u'f:r_min_ram': 0, u'f:r_disk_format': u'aki', u'f:m_image!gauge!image': '1', 'f:project_id': u'e90c2ac83dd54e05afe8671a1108c6c2', u'f:r_deleted': 'False', u'f:r_status': u'active', u'f:r_size': 4955792, u'f:r_min_disk': 0}

Here we see u'f:r_properties': {}, but happybase cannot process {} as dict.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/52670

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/52670
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=ecd62e833a0ee58030467b98dad7b700c83f7556
Submitter: Jenkins
Branch: master

commit ecd62e833a0ee58030467b98dad7b700c83f7556
Author: Nadya Privalova <email address hidden>
Date: Fri Oct 18 20:42:21 2013 +0400

    Insertion in HBase should be fixed

    * All data inserted to HBase should be String or Unicode
    * Data stored in HBase is flatten

    Fixes bug 1240474

    Change-Id: Ic36f5460275de3c829a7b27932bde284d71bd806