Gnocchi-upgrade fails when moving to latest master

[Nicolas Vila]

With a deploy made in mid-January (from branch master), gnocchi-upgrade fails when deploying gnocchi 2.0.1.dev1 (current master as of feb 22).

The stack is:

CRITI [gnocchi] TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'
Traceback (most recent call last):
  File "/usr/local/bin/gnocchi-upgrade", line 10, in <module>
    sys.exit(upgrade())
  File "/usr/local/lib/python2.7/dist-packages/gnocchi/cli.py", line 53, in upgrade
    s.upgrade(index)
  File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py", line 324, in upgrade
    ((metric,) for metric in index.list_metrics()))
  File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py", line 502, in _map_in_thread
    return list(executor.map(lambda args: method(*args), list_of_args))
  File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 581, in result_iterator
    yield future.result()
  File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 405, in result
    return self.__get_result()
  File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py", line 502, in <lambda>
    return list(executor.map(lambda args: method(*args), list_of_args))
  File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py", line 310, in _check_for_metric_upgrade
    archive = carbonara.TimeSerieArchive.unserialize(data)
  File "/usr/local/lib/python2.7/dist-packages/gnocchi/carbonara.py", line 67, in unserialize
    return cls.from_dict(msgpack.loads(data, encoding='utf-8'))
  File "/usr/local/lib/python2.7/dist-packages/gnocchi/carbonara.py", line 648, in from_dict
    return cls([AggregatedTimeSerie.from_dict(a) for a in d['archives']])
  File "/usr/local/lib/python2.7/dist-packages/gnocchi/carbonara.py", line 364, in from_dict
    prev_timestamp = pandas.Timestamp(d.get('first_timestamp') * 10e8)
TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'

I've fixed it by editing /usr/local/lib/python2.7/dist-packages/gnocchi/carbonara.py with the following:

    def from_dict(cls, d):
        """Build a time series from a dict.

        The dict format must be datetime as key and values as values.

        :param d: The dict.
        :returns: A TimeSerie object
        """
        sampling = d.get('sampling')
 -> prev_timestamp = pandas.Timestamp((d.get('first_timestamp') if d.get('first_timestamp') else 0) * 10e8)
        timestamps = []
 -> if d.get('timestamp'):
            for delta in d.get('timestamps'):
                prev_timestamp = datetime.timedelta(
                    seconds=delta * sampling) + prev_timestamp
                timestamps.append(prev_timestamp)

        return cls.from_data(
            timestamps=timestamps,
            values=d.get('values'),
            max_size=d.get('max_size'),
            sampling=sampling,
            aggregation_method=d.get('aggregation_method', 'mean'))

After editing those two lines, gnocchi-upgrade finished with no errors.

[gordon chung (chungg)]

upgrade path is broken from lz4 compression

[gordon chung (chungg)]

our migration test is broken. we write in new format, migrate and validate. we need to write in old format.

https://github.com/openstack/gnocchi/blob/master/gnocchi/carbonara.py#L639 need to patch that to write in old format.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/284528

[Nicolas Vila (nvlan)]

After applying the patch, gnocchi-upgrade compelted its run without errors: http://paste.openstack.org/show/488266/

The API is working as expected, however gnocchi-metricd is now unable to process new measures: http://paste.openstack.org/show/488260/

Gnocchi version on both API and metricd is 2.0.1.dev1.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/284528
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=6ca313dec3f4980defa3af9bd58921fe331849d0
Submitter: Jenkins
Branch: master

commit 6ca313dec3f4980defa3af9bd58921fe331849d0
Author: gordon chung <email address hidden>
Date: Wed Feb 24 22:56:52 2016 -0500

    fix migration from v1.3 to v2.0

    previous tests never really tested migration, it just took a v2.0
    TimeSerieArchive and changed to v2.0 AggregatedTimeSerie. this
    changes it so we take v1.3 TimeSerieArchive and convert it to V2.0
    AggregatedTimeSerie.

    Related-Bug: #1548367

    Change-Id: I8eea10912517b1b0a77ad7c1a3c9037d00bf1960

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2.0
Review: https://review.openstack.org/286749

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/286749
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=e7739cb4ed6cfc1a15169a6872b8b835d904b026
Submitter: Jenkins
Branch: stable/2.0

commit e7739cb4ed6cfc1a15169a6872b8b835d904b026
Author: gordon chung <email address hidden>
Date: Wed Feb 24 22:56:52 2016 -0500

    fix migration from v1.3 to v2.0

    previous tests never really tested migration, it just took a v2.0
    TimeSerieArchive and changed to v2.0 AggregatedTimeSerie. this
    changes it so we take v1.3 TimeSerieArchive and convert it to V2.0
    AggregatedTimeSerie.

    Related-Bug: #1548367

    Change-Id: I8eea10912517b1b0a77ad7c1a3c9037d00bf1960
    (cherry picked from commit 6ca313dec3f4980defa3af9bd58921fe331849d0)