can't call agent manager

[Alexander Chadin]

Trying to start ceilometer central I get the following traceback:
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics [-] Failed to call periodic 'ceilometer.agent.manager.task' (it runs every 60.00 seconds): AttributeError: 'AgentManager' object has no attribute 'hashrings'
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics Traceback (most recent call last):
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 290, in run
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics work()
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 64, in __call__
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs)
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 178, in decorator
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics return f(*args, **kwargs)
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics File "/usr/lib/python2.7/site-packages/ceilometer/agent/manager.py", line 407, in task
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics self.interval_task(running_task)
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics File "/usr/lib/python2.7/site-packages/ceilometer/agent/manager.py", line 434, in interval_task
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics task.poll_and_notify()
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics File "/usr/lib/python2.7/site-packages/ceilometer/agent/manager.py", line 157, in poll_and_notify
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics self.resources[key].get(discovery_cache))
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics File "/usr/lib/python2.7/site-packages/ceilometer/agent/manager.py", line 111, in get
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics self.agent_manager.hashrings[
2017-09-12 14:31:24.707 13100 ERROR futurist.periodics AttributeError: 'AgentManager' object has no attribute 'hashrings'

Ceilometer version 9.0

Polling file:
---
sources:
    - name: all_pollsters
      interval: 60
      meters:
          - "*"

    - name: meter_snmp
      interval: 60
      resources:
          - snmp://10.5.250.21
          - snmp://10.5.250.22
      meters:
          - "hardware.cpu*"
          - "hardware.memory*"
          - "hardware.system_stats*"
      sinks:
          - meter_sink

[gordon chung (chungg)]

seems valid. don't think it should be calling hashrings if partitioning not configured: https://github.com/openstack/ceilometer/blob/2f7c3aaf9ca0539ffe3a4425a7899da6741bd1cb/ceilometer/agent/manager.py#L111

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/505213

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/505213
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=e3db343535a546e2a8eb36f412219161b60c059d
Submitter: Jenkins
Branch: master

commit e3db343535a546e2a8eb36f412219161b60c059d
Author: Hanxi_Liu <email address hidden>
Date: Tue Sep 19 20:50:05 2017 +0800

    Initialize hashrings as AgentManager object

    If partition_coordinator value is valid, it will get hashrings. Before
    that the object hashrings must be initialized.

    Closes-Bug: #1716678
    Change-Id: I3d2b67a70a67e2176dccf1c90f23c16a334c0fdf

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/507127

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/507127
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=105788514dadcd881fc86d4b9a03d0d10e2e0874
Submitter: Jenkins
Branch: stable/pike

commit 105788514dadcd881fc86d4b9a03d0d10e2e0874
Author: Hanxi_Liu <email address hidden>
Date: Tue Sep 19 20:50:05 2017 +0800

    Initialize hashrings as AgentManager object

    If partition_coordinator value is valid, it will get hashrings. Before
    that the object hashrings must be initialized.

    Closes-Bug: #1716678
    Change-Id: I3d2b67a70a67e2176dccf1c90f23c16a334c0fdf
    (cherry picked from commit e3db343535a546e2a8eb36f412219161b60c059d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ceilometer 9.0.2 release.

[kunthar (kunthar)]

It seems 9.0.2 fix is not fixed the problem.

python-ceilometerclient version: 2.8.0
ceilometer version: 9.0.2

$ cat /etc/ceilometer/polling.yaml
---
sources:
    - name: all_pollsters
      interval: 15
      meters:
          - "*"
    - name: meter_snmp
      interval: 15
      resources:
          - snmp://compute1
          - snmp://compute2
      meters:
          - "hardware.cpu*"
          - "hardware.memory*"
          - "hardware.disk*"
          - "hardware.network*"
      sinks:
          - meter_sink

Aaaannnnd error log

2017-11-30 18:17:43.570 28529 ERROR futurist.periodics [-] Failed to call periodic 'ceilometer.agent.manager.task' (it runs every 15.00 seconds)
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics Traceback (most recent call last):
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics File "/usr/local/lib/python2.7/dist-packages/futurist/periodics.py", line 290, in run
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics work()
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics File "/usr/local/lib/python2.7/dist-packages/futurist/periodics.py", line 64, in __call__
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs)
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics File "/usr/local/lib/python2.7/dist-packages/futurist/periodics.py", line 178, in decorator
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics return f(*args, **kwargs)
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics File "/usr/local/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 412, in task
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics self.polling_periodics.add, task, polling_task)
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics File "/usr/local/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 439, in interval_task
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics def keystone(self):
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics File "/usr/local/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 161, in poll_and_notify
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics [pollster.obj.default_discovery], discovery_cache)
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics File "/usr/local/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 113, in get
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics static_resources_group].belongs_to_self(
2017-11-30 18:17:43.570 28529 ERROR futurist.periodics AttributeError: 'AgentManager' object has no attribute 'hashrings'

[Dmitriy Rabotyagov (noonedeadpunk)]

The same thing for me:
root@openstack-sandbox:~# dpkg -l| grep ceilometer
ii ceilometer-agent-central 1:9.0.2-0ubuntu1~cloud0 all ceilometer central agent
ii ceilometer-agent-notification 1:9.0.2-0ubuntu1~cloud0 all ceilometer notification agent
ii ceilometer-common 1:9.0.2-0ubuntu1~cloud0 all ceilometer common files
ii python-ceilometer 1:9.0.2-0ubuntu1~cloud0 all ceilometer python libraries
ii python-ceilometerclient 2.9.0-0ubuntu1~cloud0 all Client library for Openstack Ceilometer API server - Python 2.7
root@openstack-sandbox:~# cat /etc/ceilometer/polling.yaml
---
sources:
    - name: hardware
      interval: 60
      resources:
        - snmp://10.157.128.2
      meters:
        - hardware.cpu.util
        - hardware.memory.used
        - hardware.memory.total
        - hardware.memory.buffer
        - hardware.memory.cached
        - hardware.memory.swap.avail
        - hardware.memory.swap.total

In log:
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics [-] Failed to call periodic 'ceilometer.agent.manager.task' (it runs every 60.00 seconds): AttributeError: 'AgentManager' object has no attribute 'hashrings'
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics Traceback (most recent call last):
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics File "/usr/local/lib/python2.7/dist-packages/futurist/periodics.py", line 290, in run
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics work()
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics File "/usr/local/lib/python2.7/dist-packages/futurist/periodics.py", line 64, in __call__
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs)
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics File "/usr/local/lib/python2.7/dist-packages/futurist/periodics.py", line 178, in decorator
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics return f(*args, **kwargs)
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics File "/usr/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 409, in task
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics self.interval_task(running_task)
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics File "/usr/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 436, in interval_task
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics task.poll_and_notify()
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics File "/usr/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 158, in poll_and_notify
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics self.resources[key].get(discovery_cache))
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics File "/usr/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 110, in get
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics if self.agent_manager.hashrings:
2018-01-30 19:55:45.814 8516 ERROR futurist.periodics AttributeError: 'AgentManager' object has no attribut...

[Dmitriy Rabotyagov (noonedeadpunk)]

Actually, you've made pretty strange change:
what point to check generator one more time with if, while reverencing Attribute error? Probably it's worth to replace your new if with try\except?

[gordon chung (chungg)]

you need https://github.com/openstack/ceilometer/commit/b33a12e4665d49bc8112bb8b8517a66ac50c54dd

ceilometer 9.0.3+

[Dmitriy Rabotyagov (noonedeadpunk)]

Hi Gordon,

Yep, it works as well. Thank you for pointing to the right commit.

[gordon chung (chungg)]

:) np

[Dmitriy Rabotyagov (noonedeadpunk)]

Actually, there were no errors in log, but snmp check still didn't work. But update to 9.0.5 did the job and made snmp pollers work properly.

[gordon chung (chungg)]

that would be related to: https://bugs.launchpad.net/ceilometer/+bug/1716678

[gordon chung (chungg)]

we had a bad refactor in pike that broke stuff. hopefully it's all fixed.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ceilometer 10.0.0 release.