gnocchi-metricd container missconfigured on second and third nodes

[Gabriel Batir]

Hi,

I am testing kolla-ansible on three nodes.
I have an issue with the gnocchi_metricsd container on the second and third containers.

This is the error message in /var/lib/docker/volumes/kolla_logs/_data/gnocchi/gnocchi-metricd.log:

2018-07-05 08:12:36,365 [9755] ERROR cotyledon._utils: Unhandled exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cotyledon/_utils.py", line 84, in exit_on_exception
    yield
  File "/usr/lib/python2.7/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/usr/lib/python2.7/site-packages/gnocchi/cli/metricd.py", line 76, in run
    self._configure()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 214, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 295, in call
    start_time=start_time)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 252, in iter
    return fut.result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/gnocchi/cli/metricd.py", line 161, in _configure
    six.moves.range(self.incoming.NUM_SACKS))
  File "/usr/lib/python2.7/site-packages/gnocchi/incoming/__init__.py", line 53, in NUM_SACKS
    raise SackDetectionError(e)
SackDetectionError: [Errno 2] No such file or directory: '/var/lib/gnocchi/tmp/gnocchi-config'

Indeed, on the second and third nodes /var/lib/gnocchi/tmp/gnocchi-config doesn't exist while on the first node the file exists with the following content:

{"sacks": 128}

Also, ont he second and third nodes the gnocchi volumes are empty.

From what I could figure out the gnocchi volume is populated in the "Running gnocchi bootstrap container" ansible task that runs only on the first node.

Running gnocchi status returns a 500 Internal Server Error when the request hits the nodes with this problem.

kolla version is 6.0.0 running on Centos7 with centos binary containers.

[Björn Hagemeier (b-hagemeier)]

Hi there,

has there been any progress on this? I am having the exact same problem. What can I do to get around this problem?

Best regards,
Björn

[Björn Hagemeier (b-hagemeier)]

Hi all,

I just gave it a little try and did a 'gnocchi-upgrade' inside the container. That worked a deal and the missing gnocchi-conf has been created.

I found the idea for this in the comment of another bug: https://bugs.launchpad.net/kolla/+bug/1803522/comments/3

Best regards,
Björn

[Maciej Kucia (maciejkucia)]

gnocchi-upgrade is executed during bootstrapping:
 extended_start.sh:
 if [[ "${!KOLLA_BOOTSTRAP[@]}" ]]; then
     sudo -H -u gnocchi gnocchi-upgrade --log-file /var/log/kolla/gnocchi/gnocchi-upgrade.log
     exit 0
 fi

In kolla-ansible the bootstraping is done on single node.
 bootstrap_service.yml:
 - name: Running gnocchi bootstrap container
   kolla_docker:
     action: "start_container"
     environment:
       KOLLA_BOOTSTRAP:
       KOLLA_CONFIG_STRATEGY: "{{ config_strategy }}"
   [...]
   delegate_to: "{{ groups[gnocchi_api.group][0] }}"

[Mark Goddard (mgoddard)]

Looks like you need to remove run_once and delegate_to from that task.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/906625

[Maksim Malchuk (mmalchuk)]

Removed EOL releases.