gnocchi-metricd is crashing often with that error:
2015-07-16 14:06:17.581 3062 DEBUG gnocchi.storage._carbonara [-] Processing measures for f101e90c-f789-4ea2-a01b-b1f047db2225 process_measures /opt/stack/gnocchi/gnocchi/storage/_carbonara.py:230
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara [-] Error processing new measures
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara Traceback (most recent call last):
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/storage/_carbonara.py", line 235, in process_measures
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara for aggregation in agg_methods))
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/storage/_carbonara.py", line 283, in _map_in_thread
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara list_of_args))
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 579, in result_iterator
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara yield future.result()
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 403, in result
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara return self.__get_result()
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/thread.py", line 55, in run
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara result = self.fn(*self.args, **self.kwargs)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/storage/_carbonara.py", line 282, in <lambda>
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara return list(self.executor.map(lambda args: method(*args),
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/storage/_carbonara.py", line 207, in _add_measures
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara archive.set_values(measures, ignore_too_old_timestamps=True)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/carbonara.py", line 394, in set_values
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara ignore_too_old_timestamps=ignore_too_old_timestamps)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/carbonara.py", line 168, in set_values
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara super(BoundTimeSerie, self).set_values(values)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/carbonara.py", line 81, in set_values
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara self.ts = t.combine_first(self.ts).sort_index()
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/series.py", line 1564, in combine_first
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara this = self.reindex(new_index, copy=False)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/series.py", line 2151, in reindex
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara return super(Series, self).reindex(index=index, **kwargs)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/generic.py", line 1773, in reindex
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara method, fill_value, copy).__finalize__(self)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/generic.py", line 1790, in _reindex_axes
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara fill_value=fill_value, copy=copy, allow_dups=False)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/generic.py", line 1876, in _reindex_with_indexers
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara copy=copy)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/internals.py", line 3150, in reindex_indexer
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara self.axes[axis]._can_reindex(indexer)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/index.py", line 1860, in _can_reindex
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara raise ValueError("cannot reindex from a duplicate axis")
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara ValueError: cannot reindex from a duplicate axis
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara
I'm getting this as well when using the 'file' backend.
It's stable, in the sense that you can shutdown metricd and start it back up and there error is still there, so presumably at some point something bad is being written and from that point forward, boom.