ceilometer network.bw.in, network.bw.out and network.floating collector is not JSON serializable

[zhangguoqing]

The network.bw.in, network.bw.out and network.floating collector in cloudkitty/collector/ceilometer.py fails with an JSON serialization error. Those resources are not transformed in cloudkitty/transformer/ceilometer.py.
And they are similar with image collector at https://bugs.launchpad.net/cloudkitty/+bug/1610376.

[stack@zhangguoqing-dev devstack]$ /usr/bin/cloudkitty-processor --config-file /etc/cloudkitty/cloudkitty.conf & echo $! >/opt/stack/status/stack/ck-proc.pid; f|| echo "ck-proc failed to start" | tee "/opt/stack/status/stack/ck-proc.failure"
[1] 31373
/usr/bin/cloudkitty-processor --config-file /etc/cloudkitty/cloudkitty.conf
2016-08-22 21:47:12.972 31373 CRITICAL cloudkitty [-] TypeError: <Resource {u'user_id': u'40c2102f4a554b848d96b14f3eec39ed', u'links': [{u'href': u'http://172.16.40.152:8777/v2/resources/instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'self'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.outgoing.packets?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.outgoing.packets'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.incoming.packets?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.incoming.packets'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.incoming.bytes?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.incoming.bytes'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.outgoing.bytes?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.outgoing.bytes'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.incoming.packets.rate?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.incoming.packets.rate'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.incoming.bytes.rate?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.incoming.bytes.rate'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.outgoing.packets.rate?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.outgoing.packets.rate'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.outgoing.bytes.rate?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.outgoing.bytes.rate'}], u'resource_id': u'instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'source': u'openstack', u'last_sample_timestamp': u'2016-08-18T01:51:33.094243', u'first_sample_timestamp': u'2016-08-11T13:36:46.310846', u'project_id': u'1f5ce2b053d64b9c970100f1dc638c8e', u'metadata': {u'instance_host': u'zhangguoqing-dev-mitaka', u'ramdisk_id': u'None', u'flavor.vcpus': u'1', u'OS-EXT-AZ.availability_zone': u'nova', u'instance_id': u'17d03976-e40c-49c5-95ca-11233033fc61', u'task_state': u'None', u'display_name': u'cirros-001', u'state': u'active', u'flavor.id': u'42', u'status': u'active', u'ephemeral_gb': u'0', u'flavor.name': u'm1.nano', u'disk_gb': u'0', u'kernel_id': u'None', u'image.id': u'c2d16b10-cc49-4943-a348-c58ae868fff5', u'fref': u'None', u'flavor.ram': u'64', u'host': u'5b85dd1e7796f87bae6fea6dcb608cf907446eaa715de90d77dadb7b', u'flavor.ephemeral': u'0', u'image.name': u'cirros-test-img', u'image_ref_url': u'http://172.16.40.152:8774/images/c2d16b10-cc49-4943-a348-c58ae868fff5', u'image.links': u"[{'href': 'http://172.16.40.152:8774/images/c2d16b10-cc49-4943-a348-c58ae868fff5', 'rel': 'bookmark'}]", u'flavor.disk': u'0', u'root_gb': u'0', u'name': u'tap7115c5d8-74', u'mac': u'fa:16:3e:6f:de:5a', u'vnic_name': u'tap7115c5d8-74', u'memory_mb': u'64', u'instance_type': u'm1.nano', u'vcpus': u'1', u'image_ref': u'c2d16b10-cc49-4943-a348-c58ae868fff5', u'flavor.links': u"[{'href': 'http://172.16.40.152:8774/flavors/42', 'rel': 'bookmark'}]"}}> is not JSON serializable
2016-08-22 21:47:12.972 31373 ERROR cloudkitty Traceback (most recent call last):
2016-08-22 21:47:12.972 31373 ERROR cloudkitty File "/usr/bin/cloudkitty-processor", line 10, in <module>
2016-08-22 21:47:12.972 31373 ERROR cloudkitty sys.exit(main())
2016-08-22 21:47:12.972 31373 ERROR cloudkitty File "/opt/stack/cloudkitty/cloudkitty/cli/processor.py", line 28, in main
2016-08-22 21:47:12.972 31373 ERROR cloudkitty processor.process()
2016-08-22 21:47:12.972 31373 ERROR cloudkitty File "/opt/stack/cloudkitty/cloudkitty/orchestrator.py", line 295, in process
2016-08-22 21:47:12.972 31373 ERROR cloudkitty worker.run()
2016-08-22 21:47:12.972 31373 ERROR cloudkitty File "/opt/stack/cloudkitty/cloudkitty/orchestrator.py", line 214, in run
2016-08-22 21:47:12.972 31373 ERROR cloudkitty self._storage.append(data, self._tenant_id)
2016-08-22 21:47:12.972 31373 ERROR cloudkitty File "/opt/stack/cloudkitty/cloudkitty/storage/__init__.py", line 227, in append
2016-08-22 21:47:12.972 31373 ERROR cloudkitty self._dispatch(data, tenant_id)
2016-08-22 21:47:12.972 31373 ERROR cloudkitty File "/opt/stack/cloudkitty/cloudkitty/storage/sqlalchemy/__init__.py", line 69, in _dispatch
2016-08-22 21:47:12.972 31373 ERROR cloudkitty self._append_time_frame(service, frame, tenant_id)
2016-08-22 21:47:12.972 31373 ERROR cloudkitty File "/opt/stack/cloudkitty/cloudkitty/storage/sqlalchemy/__init__.py", line 154, in _append_time_frame
2016-08-22 21:47:12.972 31373 ERROR cloudkitty desc = json.dumps(frame['desc'])
2016-08-22 21:47:12.972 31373 ERROR cloudkitty File "/usr/lib64/python2.7/json/__init__.py", line 243, in dumps
2016-08-22 21:47:12.972 31373 ERROR cloudkitty return _default_encoder.encode(obj)
2016-08-22 21:47:12.972 31373 ERROR cloudkitty File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
2016-08-22 21:47:12.972 31373 ERROR cloudkitty chunks = self.iterencode(o, _one_shot=True)
2016-08-22 21:47:12.972 31373 ERROR cloudkitty File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
2016-08-22 21:47:12.972 31373 ERROR cloudkitty return _iterencode(o, 0)
2016-08-22 21:47:12.972 31373 ERROR cloudkitty File "/usr/lib64/python2.7/json/encoder.py", line 184, in default
2016-08-22 21:47:12.972 31373 ERROR cloudkitty raise TypeError(repr(o) + " is not JSON serializable")
2016-08-22 21:47:12.972 31373 ERROR cloudkitty TypeError: <Resource {u'user_id': u'40c2102f4a554b848d96b14f3eec39ed', u'links': [{u'href': u'http://172.16.40.152:8777/v2/resources/instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'self'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.outgoing.packets?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.outgoing.packets'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.incoming.packets?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.incoming.packets'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.incoming.bytes?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.incoming.bytes'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.outgoing.bytes?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.outgoing.bytes'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.incoming.packets.rate?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.incoming.packets.rate'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.incoming.bytes.rate?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.incoming.bytes.rate'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.outgoing.packets.rate?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.outgoing.packets.rate'}, {u'href': u'http://172.16.40.152:8777/v2/meters/network.outgoing.bytes.rate?q.field=resource_id&q.value=instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'rel': u'network.outgoing.bytes.rate'}], u'resource_id': u'instance-00000001-17d03976-e40c-49c5-95ca-11233033fc61-tap7115c5d8-74', u'source': u'openstack', u'last_sample_timestamp': u'2016-08-18T01:51:33.094243', u'first_sample_timestamp': u'2016-08-11T13:36:46.310846', u'project_id': u'1f5ce2b053d64b9c970100f1dc638c8e', u'metadata': {u'instance_host': u'zhangguoqing-dev-mitaka', u'ramdisk_id': u'None', u'flavor.vcpus': u'1', u'OS-EXT-AZ.availability_zone': u'nova', u'instance_id': u'17d03976-e40c-49c5-95ca-11233033fc61', u'task_state': u'None', u'display_name': u'cirros-001', u'state': u'active', u'flavor.id': u'42', u'status': u'active', u'ephemeral_gb': u'0', u'flavor.name': u'm1.nano', u'disk_gb': u'0', u'kernel_id': u'None', u'image.id': u'c2d16b10-cc49-4943-a348-c58ae868fff5', u'fref': u'None', u'flavor.ram': u'64', u'host': u'5b85dd1e7796f87bae6fea6dcb608cf907446eaa715de90d77dadb7b', u'flavor.ephemeral': u'0', u'image.name': u'cirros-test-img', u'image_ref_url': u'http://172.16.40.152:8774/images/c2d16b10-cc49-4943-a348-c58ae868fff5', u'image.links': u"[{'href': 'http://172.16.40.152:8774/images/c2d16b10-cc49-4943-a348-c58ae868fff5', 'rel': 'bookmark'}]", u'flavor.disk': u'0', u'root_gb': u'0', u'name': u'tap7115c5d8-74', u'mac': u'fa:16:3e:6f:de:5a', u'vnic_name': u'tap7115c5d8-74', u'memory_mb': u'64', u'instance_type': u'm1.nano', u'vcpus': u'1', u'image_ref': u'c2d16b10-cc49-4943-a348-c58ae868fff5', u'flavor.links': u"[{'href': 'http://172.16.40.152:8774/flavors/42', 'rel': 'bookmark'}]"}}> is not JSON serializable
2016-08-22 21:47:12.972 31373 ERROR cloudkitty
2016-08-22 21:47:13.037 31373 WARNING tooz.drivers.file [-] Unreleased lock /var/lib/cloudkitty/locks/cloudkitty-1f5ce2b053d64b9c970100f1dc638c8e garbage collected
ck-proc failed to start

[zhangguoqing (474751729-o)]

Pollster for network internal traffic (n1,n2) https://review.openstack.org/#/c/9977/
Fixes AttributeError of FloatingIPPollster https://review.openstack.org/#/c/27975/
According to the above two patches in ceilometer, we known that the three pollsters about network.bw.in, network.bw.out and network.floating have been updated. Then the name of sample metadata attribute is named 'resource_meatedata' rather than 'metadata'. And the resource_meatedata only has little useful information, almost of information are in sample's other attributes. So we had better to deal with them respectively.

[zhangguoqing (474751729-o)]

Fix proposed to branch: master
Review: https://review.openstack.org/#/c/355869/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/355869
Committed: https://git.openstack.org/cgit/openstack/cloudkitty/commit/?id=015546e44efa3e3ea0cd59d8a4130e3598fe6544
Submitter: Jenkins
Branch: master

commit 015546e44efa3e3ea0cd59d8a4130e3598fe6544
Author: zhangguoqing <[e-mail]>
Date: Tue Aug 16 19:36:32 2016 +0800

    ceilometer network.* collector are not JSON serializable

    Add network_tap and network_floating to CeilometerTransformer.
    Add corresponding test cases.

    In order to map the strip_func, replace the '.' to '_' in service name,
    by the line "res_type = res_type.replace('.', '_')".
    network.floating --> _strip_network_floating
    (network.bw.in and network.bw.out) network.tap --> _strip_network_tap

    Change-Id: Iee47f7693e8a37d8239ea52419bef1c2304d0a31
    Closes-bug: #1616738

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cloudkitty 0.6.1 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/431397