Exception in gnocchi-metricd: ValueError: cannot reindex from a duplicate axis

[Frederic Guillot]

gnocchi-metricd is crashing often with that error:

2015-07-16 14:06:17.581 3062 DEBUG gnocchi.storage._carbonara [-] Processing measures for f101e90c-f789-4ea2-a01b-b1f047db2225 process_measures /opt/stack/gnocchi/gnocchi/storage/_carbonara.py:230
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara [-] Error processing new measures
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara Traceback (most recent call last):
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/storage/_carbonara.py", line 235, in process_measures
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara for aggregation in agg_methods))
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/storage/_carbonara.py", line 283, in _map_in_thread
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara list_of_args))
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 579, in result_iterator
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara yield future.result()
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 403, in result
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara return self.__get_result()
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/thread.py", line 55, in run
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara result = self.fn(*self.args, **self.kwargs)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/storage/_carbonara.py", line 282, in <lambda>
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara return list(self.executor.map(lambda args: method(*args),
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/storage/_carbonara.py", line 207, in _add_measures
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara archive.set_values(measures, ignore_too_old_timestamps=True)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/carbonara.py", line 394, in set_values
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara ignore_too_old_timestamps=ignore_too_old_timestamps)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/carbonara.py", line 168, in set_values
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara super(BoundTimeSerie, self).set_values(values)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/opt/stack/gnocchi/gnocchi/carbonara.py", line 81, in set_values
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara self.ts = t.combine_first(self.ts).sort_index()
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/series.py", line 1564, in combine_first
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara this = self.reindex(new_index, copy=False)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/series.py", line 2151, in reindex
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara return super(Series, self).reindex(index=index, **kwargs)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/generic.py", line 1773, in reindex
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara method, fill_value, copy).__finalize__(self)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/generic.py", line 1790, in _reindex_axes
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara fill_value=fill_value, copy=copy, allow_dups=False)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/generic.py", line 1876, in _reindex_with_indexers
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara copy=copy)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/internals.py", line 3150, in reindex_indexer
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara self.axes[axis]._can_reindex(indexer)
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/pandas/core/index.py", line 1860, in _can_reindex
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara raise ValueError("cannot reindex from a duplicate axis")
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara ValueError: cannot reindex from a duplicate axis
2015-07-16 14:06:17.631 3062 ERROR gnocchi.storage._carbonara

[Chris Dent (cdent)]

I'm getting this as well when using the 'file' backend.

It's stable, in the sense that you can shutdown metricd and start it back up and there error is still there, so presumably at some point something bad is being written and from that point forward, boom.

[Chris Dent (cdent)]

There's a stackoverflow question about it, for reference: http://stackoverflow.com/questions/27236275/what-does-valueerror-cannot-reindex-from-a-duplicate-axis-mean

[Chris Dent (cdent)]

This happens when some of the measure awaiting processing by metricd have the same timestamp. That timestamp is the axis/index and pandas is not willing to have duplicates there.

That suggests some options, the last of which seems to be the least likely to cause havoc:

* raise granularity of timestamps
* ignore the entire current lump being processed when there is a ValueError that indicate
* drop measures of the same metric that have duplicate timestamps (presumably by using drop_duplicates: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html )

[Julien Danjou (jdanjou)]

You cannot raise the granularity of timestamp, they go up to the nanosecond. The thing is that the timestamp is provided by the user, and if it provides twice the same timestamp even with the same value, well the problem is going to occur.

I think we should simply ignore the oldest one.

Do you have a unit test case that reproduces this?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/204177

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/204177
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=262e8454f977b20862241c76dcfd203a8f1ef80a
Submitter: Jenkins
Branch: master

commit 262e8454f977b20862241c76dcfd203a8f1ef80a
Author: Chris Dent <email address hidden>
Date: Tue Jul 21 17:38:05 2015 +0000

    Don't allow duplicate timestamps in carbonara series

    If the calling code provides timestamp/value pairs with duplicate
    timestamps pandas will raise a ValueError when trying to operate on the
    index of the data set. We avoid this problem by removing duplicates in
    the timeseries. The assumption being made here is that some single
    metric can't have two different measurements at the same time, that's a
    violation of physics.

    The change is in place for both __init__ and set_values to insure
    that the internal representation of the timeseries is consistent no
    matter how it is created.

    Change-Id: Ib293714e26eca6b5de7cd4a2307569ab3d8a9c43
    Closes-Bug: #1475329