AttributeError in discovery while keystone unavailable

[Chris Dent]

If a pollster is attempting to do discovery with the keystone client and the keystone service is not yet ready, a confusing AttributeError is raised in the central agent logs:

2015-09-07 13:16:30.111 18838 ERROR ceilometer.agent.base [-] Unable to discover resources: 'AuthorizationFailure' object has no attribute 'service_catalog'
2015-09-07 13:16:30.111 18838 ERROR ceilometer.agent.base Traceback (most recent call last):
2015-09-07 13:16:30.111 18838 ERROR ceilometer.agent.base File "/home/opt/stack/ceilometer/ceilometer/agent/base.py", line 396, in discover
2015-09-07 13:16:30.111 18838 ERROR ceilometer.agent.base discovered = discoverer.discover(self, param)
2015-09-07 13:16:30.111 18838 ERROR ceilometer.agent.base File "/home/opt/stack/ceilometer/ceilometer/agent/discovery/endpoint.py", line 37, in discover
2015-09-07 13:16:30.111 18838 ERROR ceilometer.agent.base endpoints = manager.keystone.service_catalog.get_urls(
2015-09-07 13:16:30.111 18838 ERROR ceilometer.agent.base AttributeError: 'AuthorizationFailure' object has no attribute 'service_catalog'

Some sort of type or error checking that should be happening here is not.

This is happening early in a devstack, right after the central agent has started. Eventually, as keystone becomes ready, the problem goes away.

However with a lot of pollster plugins this can lead to a bunch of scary log messages that are not useful.

[Rohit Jaiswal (rohit-jaiswal-3)]

I've seen this too.

[ZhiQiang Fan (aji-zqfan)]

see: https://github.com/openstack/ceilometer/blob/f94a72ae2be4a97aced3b7c70ef982fab8ad1742/ceilometer/agent/manager.py#L51

    def interval_task(self, task):
        try:
            self.keystone = keystone_client.get_client()
        except Exception as e:
            self.keystone = e
        super(AgentManager, self).interval_task(task)

[Rohit Jaiswal (rohit-jaiswal-3)]

I think the attribute check should be owned by each discovery plugin which knows whether it uses keystone for discovery or not. Validating it anywhere else like at the manager level will restrict other pollsters who dont use keystone for discovery.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/221546

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/222495

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/222495
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=840feeab4feb4fe67b8f7bbf8dc264897e65615c
Submitter: Jenkins
Branch: master

commit 840feeab4feb4fe67b8f7bbf8dc264897e65615c
Author: Mehdi Abaakouk <email address hidden>
Date: Fri Sep 11 08:38:52 2015 +0200

    Refactor keystone handling in discovery manager

    The current code assumes the keytone attribute of the manager is a
    keystone client object but sometimes this is not true, this is
    the exception raised by keystoneclient.

    This is not sane to store two differents things in same attribute.
    This change fixes that, by storing only the keystone client object and
    just reraise the keystoneclient exception when keystone is unavailable.

    So the discovery process can detect that keystone is unavailable by
    catching the base exception of the keystoneclient library and
    just skip discovery plugin with an appropriate message.

    Another advantage is that we will not swallow anymore unexpected keystone
    client issue by a simple keystone authentication message.

    The check_keystone won't work as expected too. Usage of cfg.CONF into a
    decorator is broken. This refactor moves the code into the discover
    method.

    Change-Id: I6adabd7c09672af20e8fabb5016bd2978564e1b2
    Closes-Bug: 1493056
    Closes-Bug: 1494592

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rohit Jaiswal (<email address hidden>) on branch: master
Review: https://review.openstack.org/221546