Ceilometer transformation is failing due to misconfiguration of memcache_servers endpoint

[Tee Ngo]

The following exception is observed in STX build 2018-09-03_20-18-01

Exception in ceilometer-agent-notification.log

2018-09-04T12:30:51.000 controller-1 ceilometer-agent-notification: err 2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi [-] Unable to parse connection string: "abba::4:11211": ValueError: Unable to parse connection string: "abba::4:11211"
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi Traceback (most recent call last):
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python2.7/site-packages/ceilometer/publisher/gnocchi.py", line 391, in publish_samples
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi resource_extra)
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python2.7/site-packages/ceilometer/publisher/gnocchi.py", line 461, in _if_not_cached
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi if self._resource_cache_diff(res_id, attribute_hash):
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python2.7/site-packages/ceilometer/publisher/gnocchi.py", line 483, in _resource_cache_diff
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi cached_hash = self.cache.get(key)
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 626, in get
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi value = self.backend.get(key)
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 161, in get
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi value = self.client.get(key)
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 151, in client
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi return self._clients.memcached
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python2.7/site-packages/dogpile/util/langhelpers.py", line 63, in __get__
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi obj.__dict__[self.__name__] = result = self.fget(obj)
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 139, in _clients
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi return ClientPool()
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 137, in __init__
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi self.memcached = backend._create_client()
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 290, in _create_client
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi return memcache.Client(self.url)
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python2.7/site-packages/memcache.py", line 228, in __init__
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi self.set_servers(servers)
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python2.7/site-packages/memcache.py", line 307, in set_servers
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi for s in servers]
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python2.7/site-packages/memcache.py", line 1357, in __init__
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi raise ValueError('Unable to parse connection string: "%s"' % host)
2018-09-04 12:30:51.687 9681 ERROR ceilometer.publisher.gnocchi ValueError: Unable to parse connection string: "abba::4:11211"

In ceilometer.conf
Memcache_servers='abba::4:11211'

Need prefix for IPv6 addressing.

Severity
--------
Major, only affecting IPv6

Steps to Reproduce
------------------
Launch a VM. List cpu, cpu_util, vcpu_util meters and measurements belonging to the VM

Expected Behavior
------------------
CPU related meters and measurements belongubg to the VM are displayed

Actual Behavior
----------------
Expected meters and measurements do not exist resulting in vcpu scaling test case failure.

Reproducibility
---------------
Always

System Configuration
--------------------
Issue was observed in a multi-node system with IPv6.

Timestamp/Logs
--------------
See logs above

[Angie Wang (angiewang)]

On the ipv6 system, should add prefix inet6 for the ip address, ex: memcache_servers='inet6:[abba::4]:11211'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/601418

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/601418
Committed: https://git.openstack.org/cgit/openstack/stx-config/commit/?id=11fe36dea0dc24acbd7bca4f0829939b132feda5
Submitter: Zuul
Branch: master

commit 11fe36dea0dc24acbd7bca4f0829939b132feda5
Author: Angie Wang <email address hidden>
Date: Mon Sep 10 15:30:56 2018 -0400

    Set the correct memcached server in puppet ceilometer

    The existing memcached server address configured in puppet ceilometer
    cannot be recognized by memcached in an ipv6 system, this causes the
    failure of caching ceilometer resource metadata.
    This update is to correct the format of memcached server configured in
    ceilometer puppet for ipv6 system.

    Closes-Bug: 1790662

    Change-Id: Icfdcb688cb0740d10c5d970e53564680e75058e9
    Signed-off-by: Angie Wang <email address hidden>