gnocchi is eating all my cpu too :(

[Gabriele Cerami]

During HA ipv6 deployment, gnocchi-metricd is eating most of the overcloud node CPU

/var/log/gnocchi/metricd.log is full of these ERRORS

2016-09-30 11:05:07.839 2677 ERROR cotyledon [-] Unhandled exception
2016-09-30 11:05:07.839 2677 ERROR cotyledon Traceback (most recent call last):
2016-09-30 11:05:07.839 2677 ERROR cotyledon File "/usr/lib/python2.7/site-packages/cotyledon/__init__.py", line 52, in _exit_on_exception
2016-09-30 11:05:07.839 2677 ERROR cotyledon yield
2016-09-30 11:05:07.839 2677 ERROR cotyledon File "/usr/lib/python2.7/site-packages/cotyledon/__init__.py", line 398, in _start_service
2016-09-30 11:05:07.839 2677 ERROR cotyledon self._current_process = config.service(worker_id, *args, **kwargs)
2016-09-30 11:05:07.839 2677 ERROR cotyledon File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 155, in __init__
2016-09-30 11:05:07.839 2677 ERROR cotyledon self._enable_coordination(conf)
2016-09-30 11:05:07.839 2677 ERROR cotyledon File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 149, in _enable_coordination
2016-09-30 11:05:07.839 2677 ERROR cotyledon self._coord.start(start_heart=True)
2016-09-30 11:05:07.839 2677 ERROR cotyledon File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 383, in start
2016-09-30 11:05:07.839 2677 ERROR cotyledon self._start()
2016-09-30 11:05:07.839 2677 ERROR cotyledon File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 429, in _start
2016-09-30 11:05:07.839 2677 ERROR cotyledon self.timeout)
2016-09-30 11:05:07.839 2677 ERROR cotyledon File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 381, in _make_client
2016-09-30 11:05:07.839 2677 ERROR cotyledon if parsed_url.port:
2016-09-30 11:05:07.839 2677 ERROR cotyledon File "/usr/lib64/python2.7/urlparse.py", line 112, in port
2016-09-30 11:05:07.839 2677 ERROR cotyledon port = int(port, 10)
2016-09-30 11:05:07.839 2677 ERROR cotyledon ValueError: invalid literal for int() with base 10: 'fd00'

[Gabriele Cerami (gcerami)]

logs in CI

http://logs.openstack.org/74/363674/28/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha-ipv6/726be86/logs/overcloud-controller-1/var/log/gnocchi/metricd.txt.gz

[Gabriele Cerami (gcerami)]

apparently gnocchi puppet manifest passes the wrong coordination_url to gnocchi.conf. It does not contain brackets.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/380291

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/380310

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/380291
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b97c913f23a0c01a14e74384edc4f2d3a9e0ec1f
Submitter: Jenkins
Branch: master

commit b97c913f23a0c01a14e74384edc4f2d3a9e0ec1f
Author: Emilien Macchi <email address hidden>
Date: Fri Sep 30 09:28:06 2016 -0400

    Telemetry: add redis_password hiera parameter

    Add redis_password parameter in Hiera so we can re-use it from
    puppet-tripleo later for Aodh, Ceilometer and Gnocchi.

    Change-Id: I038e2bac22e3bfa5047d2e76e23cff664546464d
    Partial-Bug: #1629279

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/380310
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d720f22925d20f430fc64c8a809efc4bfc2b4038
Submitter: Jenkins
Branch: master

commit d720f22925d20f430fc64c8a809efc4bfc2b4038
Author: Emilien Macchi <email address hidden>
Date: Fri Sep 30 09:48:56 2016 -0400

    telemetry: remove coordination_url hiera settings

    They are now normalized and set in puppet-tripleo.

    Change-Id: I197481c577b85894178e7899a55869da47847755
    Closes-Bug: #1629279
    Depends-On: Ic6de09acf0d36ca90cc2041c0add1bc2b4a369a5

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/380546

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/380881

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/380546
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=cd133a01738fb25a1e307436ec691c2081d9f1be
Submitter: Jenkins
Branch: stable/newton

commit cd133a01738fb25a1e307436ec691c2081d9f1be
Author: Emilien Macchi <email address hidden>
Date: Fri Sep 30 09:28:06 2016 -0400

    Telemetry: add redis_password hiera parameter

    Add redis_password parameter in Hiera so we can re-use it from
    puppet-tripleo later for Aodh, Ceilometer and Gnocchi.

    Change-Id: I038e2bac22e3bfa5047d2e76e23cff664546464d
    Partial-Bug: #1629279
    (cherry picked from commit b97c913f23a0c01a14e74384edc4f2d3a9e0ec1f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/380881
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a4e53b87069729e8b5f94132bd69fc770ff10760
Submitter: Jenkins
Branch: stable/newton

commit a4e53b87069729e8b5f94132bd69fc770ff10760
Author: Emilien Macchi <email address hidden>
Date: Fri Sep 30 09:48:56 2016 -0400

    telemetry: remove coordination_url hiera settings

    They are now normalized and set in puppet-tripleo.

    Change-Id: I197481c577b85894178e7899a55869da47847755
    Closes-Bug: #1629279
    Depends-On: Ic6de09acf0d36ca90cc2041c0add1bc2b4a369a5
    (cherry picked from commit d720f22925d20f430fc64c8a809efc4bfc2b4038)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 5.0.0.0rc3 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 5.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 6.0.0.0b1 development milestone.