gnocchi-metricd stacktraces after 1/2 hours of processing w/CEPH as backend

[Alejandro Comisario]

gnocchi-metricd starts ok and process metrics (against CEPH) for about and hour or so, after that time, we observe this stacktraces from metricd with no error on CEPH nor gnocchi-api:

2015-09-24 21:35:52.221 1410 DEBUG gnocchi.storage._carbonara [-] Processing measures for 0e3f1994-768a-4808-b5ea-323d9409781e process_measures /usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py:148
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara [-] Error processing new measures
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara Traceback (most recent call last):
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py", line 184, in process_measures
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara ignore_too_old_timestamps=True)
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/gnocchi/carbonara.py", line 188, in set_values
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara before_truncate_callback(self)
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py", line 179, in _map_add_measures
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara for aggregation in agg_methods))
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py", line 237, in _map_in_thread
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara list_of_args))
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 579, in result_iterator
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara yield future.result()
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 403, in result
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara return self.__get_result()
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/thread.py", line 55, in run
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara result = self.fn(*self.args, **self.kwargs)
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py", line 236, in <lambda>
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara return list(self.executor.map(lambda args: method(*args),
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py", line 106, in _add_measures
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara archive = self._get_measures_archive(metric, aggregation)
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py", line 103, in _get_measures_archive
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara return carbonara.TimeSerieArchive.unserialize(contents)
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/gnocchi/carbonara.py", line 58, in unserialize
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara return cls.from_dict(msgpack.loads(data, encoding='utf-8'))
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara File "msgpack/_unpacker.pyx", line 145, in msgpack._unpacker.unpackb (msgpack/_unpacker.cpp:145)
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara UnpackValueError: Unpack failed: error = 0
2015-09-24 21:36:10.156 1410 ERROR gnocchi.storage._carbonara
2015-09-24 21:36:10.253 1410 DEBUG gnocchi.storage._carbonara [-] Processing measures for 0e8236e6-f44a-48b2-8146-0c62e71399ac process_measures /usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py:148

gnocchi.conf example:

[DEFAULT]
debug = True
verbose = True
log_file = /var/log/gnocchi/gnocchi.log
[api]
port = 8041
host = 0.0.0.0
workers = 2
[archive_policy]
[database]
[indexer]
url = mysql://gnocchi:NOTgnocchi@mysql/gnocchi?charset=utf8
[keystone_authtoken]
signing_dir = /var/cache/gnocchi
auth_uri = http://kstn:5000/v2.0
auth_url = http://kstn:35357/v2.0
project_domain_id = default
project_name = service
project_name = admin
password = MYSUPERPASSWD
username = cloudadmin
auth_plugin = password
memcached_servers = memcache2:11211,memcache1:11211
memcache_security_strategy = ENCRYPT
memcache_secret_key = LE9_s0kyh7Z_qNsmljOT
[metricd]
[oslo_policy]
[statsd]
[storage]
driver = ceph
metric_processing_delay = 5
ceph_pool = gnocchi
ceph_username = gnocchi
ceph_keyring = /etc/ceph/ceph.client.gnocchi.keyring
ceph_conffile = /etc/ceph/ceph.conf
file_basepath = /var/lib/gnocchi
file_basepath_tmp = ${file_basepath}/tmp

[Julien Danjou (jdanjou)]

This is actually a duplicate. For whatever reason you got a corrupted (more likely empty file) and metricd is going to yell about it for ever until deleted.