IndexError: index is out of bounds for axis with size

[Larry Lile]

I am receiving the following trace from gnocchi

2019-12-20 16:35:06,334 [31] ERROR gnocchi.chef: Error processing new measures
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/gnocchi/chef.py", line 165, in process_new_measures_for_sack
    for metric in metrics
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 714, in add_measures_to_metrics
    before_truncate_callback=_map_compute_splits_operations,
  File "/usr/lib/python2.7/site-packages/gnocchi/carbonara.py", line 364, in set_values
    return_value = before_truncate_callback(self)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 673, in _map_compute_splits_operations
    tstamp = max(bound_timeserie.first, measures['timestamps'][0])
IndexError: index 0 is out of bounds for axis 0 with size 0

Unfortunately there is not enough information here to determine what measurement is causing the issue.

Could gnocchi/chef.py be modified to try/catch the single failed self.storage.add_measures_to_metrics individually instead of dropping the remainder of the measurements?

        try:
            with self.incoming.process_measures_for_sack(sack) as measures:
                # process only active metrics. deleted metrics with unprocessed
                # measures will be skipped until cleaned by janitor.
                if not measures:
                    return 0

                metrics = self.index.list_metrics(
                    attribute_filter={
                        "in": {"id": measures.keys()}
                    })
-------- try:
                self.storage.add_measures_to_metrics({
                    metric: measures[metric.id]
                    for metric in metrics
                })
-------- except:
                Report failed metric and any details?

                LOG.debug("Measures for %d metrics processed",
                          len(metrics))
                return len(measures)
        except Exception:
            if sync:
                raise
            LOG.error("Error processing new measures", exc_info=True)
            return 0
        finally:
            lock.release()

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Rizky (ridz)]

I have the same problem. This error log begins showing once I set the archive-policy granularity to 3s as well as setting the polling interval in ceilometer to 3 seconds. This error never showing days before when the granularity and the polling interval is both at 3600s

I think the cause is because some of the metrics has zero value, or perhaps because of the error it became zero?

gnocchi aggregates '(/ (aggregate rate:mean (metric cpu mean)) 3000000000)' id=65592309-1e03-4c9e-8696-af7ebe30583c | tail

| aggregated | 2021-03-31T21:16:48+00:00 | 3.0 | 0.0 |
| aggregated | 2021-03-31T21:16:54+00:00 | 3.0 | 0.01 |
| aggregated | 2021-03-31T21:16:57+00:00 | 3.0 | 0.006666666666666667 |
| aggregated | 2021-03-31T21:17:00+00:00 | 3.0 | 0.006666666666666667 |
| aggregated | 2021-03-31T21:17:03+00:00 | 3.0 | 0.0033333333333333335 |
| aggregated | 2021-03-31T21:17:09+00:00 | 3.0 | 0.008333333333333333 |
| aggregated | 2021-03-31T21:17:12+00:00 | 3.0 | 0.008333333333333333 |
| aggregated | 2021-03-31T21:17:15+00:00 | 3.0 | 0.0 |
| aggregated | 2021-03-31T21:17:18+00:00 | 3.0 | 0.01 |
+------------+---------------------------+-------------+-----------------------+

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gnocchi/chef.py", line 163, in process_new_measures_for_sack
    self.storage.add_measures_to_metrics({
  File "/usr/lib/python3/dist-packages/gnocchi/storage/__init__.py", line 675, in add_measures_to_metrics
    keys_and_splits_to_store) = ts.set_values(
  File "/usr/lib/python3/dist-packages/gnocchi/carbonara.py", line 367, in set_values
    return_value = before_truncate_callback(self)
  File "/usr/lib/python3/dist-packages/gnocchi/storage/__init__.py", line 638, in _map_compute_splits_operations
    tstamp = max(bound_timeserie.first, measures['timestamps'][0])
IndexError: index 0 is out of bounds for axis 0 with size 0

My installed versions :

gnocchi-api/groovy,now 4.4.0-0ubuntu1 all [installed]
gnocchi-common/groovy,now 4.4.0-0ubuntu1 all [installed,automatic]
gnocchi-metricd/groovy,now 4.4.0-0ubuntu1 all [installed]
gnocchi-statsd/groovy 4.4.0-0ubuntu1 all