openstack-ceilometer-api.service fails to start if event_connection db is not available

[Yurii Prokulevych]

Excerpt from ceilometer.conf

[database]
...
connection=mongodb://ceilometer:12345678@X.X.X.X:27017/ceilometer
event_connection=mongodb://ceilometer:SomePass@Y.Y.Y.Y:27017/eventsdb
...

When MongoDB instance at Y.Y.Y.Y host becomes unavailable and a try to restart openstack-ceilometer-api.service is given,
openstack-ceilometer-api.service fails to restart.

EXCERPT from api.log
...
2015-05-20 10:12:54.058 89605 WARNING ceilometer.storage.mongo.utils [-] Unable to connect to the database server: could not connect to Y.Y.Y.Y:27017: [Errno 111] ECONNREFUSED.
2015-05-20 10:12:54.058 89605 CRITICAL ceilometer [-] ConnectionFailure: could not connect to Y.Y.Y.Y:27017: [Errno 111] ECONNREFUSED
2015-05-20 10:12:54.058 89605 TRACE ceilometer Traceback (most recent call last):
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/bin/ceilometer-api", line 10, in <module>
2015-05-20 10:12:54.058 89605 TRACE ceilometer sys.exit(main())
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/cmd/api.py", line 23, in main
2015-05-20 10:12:54.058 89605 TRACE ceilometer app.build_server()
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/api/app.py", line 132, in build_server
2015-05-20 10:12:54.058 89605 TRACE ceilometer app = load_app()
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/api/app.py", line 128, in load_app
2015-05-20 10:12:54.058 89605 TRACE ceilometer return deploy.loadapp("config:" + cfg_file)
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2015-05-20 10:12:54.058 89605 TRACE ceilometer return loadobj(APP, uri, name=name, **kw)
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2015-05-20 10:12:54.058 89605 TRACE ceilometer return context.create()
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2015-05-20 10:12:54.058 89605 TRACE ceilometer return self.object_type.invoke(self)
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 203, in invoke
2015-05-20 10:12:54.058 89605 TRACE ceilometer app = context.app_context.create()
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2015-05-20 10:12:54.058 89605 TRACE ceilometer return self.object_type.invoke(self)
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2015-05-20 10:12:54.058 89605 TRACE ceilometer return fix_call(context.object, context.global_conf, **context.local_conf)
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
2015-05-20 10:12:54.058 89605 TRACE ceilometer val = callable(*args, **kw)
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/api/app.py", line 154, in app_factory
2015-05-20 10:12:54.058 89605 TRACE ceilometer return VersionSelectorApplication()
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/api/app.py", line 108, in __init__
2015-05-20 10:12:54.058 89605 TRACE ceilometer self.v2 = setup_app(pecan_config=pc)
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/api/app.py", line 68, in setup_app
2015-05-20 10:12:54.058 89605 TRACE ceilometer storage.get_connection_from_config(cfg.CONF, 'event'),
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py", line 116, in get_connection_from_config
2015-05-20 10:12:54.058 89605 TRACE ceilometer return _inner()
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f
2015-05-20 10:12:54.058 89605 TRACE ceilometer return Retrying(*dargs, **dkw).call(f, *args, **kw)
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call
2015-05-20 10:12:54.058 89605 TRACE ceilometer raise attempt.get()
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
2015-05-20 10:12:54.058 89605 TRACE ceilometer six.reraise(self.value[0], self.value[1], self.value[2])
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
2015-05-20 10:12:54.058 89605 TRACE ceilometer attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py", line 114, in _inner
2015-05-20 10:12:54.058 89605 TRACE ceilometer return get_connection(url, namespace)
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py", line 129, in get_connection
2015-05-20 10:12:54.058 89605 TRACE ceilometer return mgr.driver(url)
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/event/storage/impl_mongodb.py", line 38, in __init__
2015-05-20 10:12:54.058 89605 TRACE ceilometer self.conn = self.CONNECTION_POOL.connect(url)
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/storage/mongo/utils.py", line 247, in connect
2015-05-20 10:12:54.058 89605 TRACE ceilometer client = self._mongo_connect(url)
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/storage/mongo/utils.py", line 260, in _mongo_connect
2015-05-20 10:12:54.058 89605 TRACE ceilometer client = MongoProxy(pymongo.MongoClient(url))
2015-05-20 10:12:54.058 89605 TRACE ceilometer File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 337, in __init__
2015-05-20 10:12:54.02015-05-20 10:12:54.058 89605 TRACE ceilometer ConnectionFailure: could not connect to Y.Y.Y.Y:27017: [Errno 111] ECONNREFUSED
2015-05-20 10:12:54.058 89605 TRACE ceilometer
58 89605 TRACE ceilometer raise ConnectionFailure(str(e))

systemctl --failed -t service
UNIT LOAD ACTIVE SUB DESCRIPTION
openstack-ceilometer-api.service loaded failed failed OpenStack ceilometer API service

# ceilometer meter-list
('Connection aborted.', error(111, 'Connection refused'))

# ceilometer event-list
('Connection aborted.', error(111, 'Connection refused'))

event_connection, similarly to metering and alarm connections, allows event-related data be stored at dedicated db, I assume event db failure should just affect event specific api calls not the api service.

Since there might be up to 3 connection configured - it would be great if they can work independently.
In case some of them fails - just a part of functionally, specific to failed connection, would be inaccessible.

Thanks,
Yurii

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/184655
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=1f30839494c5b5002173a1f81eb960d735f39643
Submitter: Jenkins
Branch: master

commit 1f30839494c5b5002173a1f81eb960d735f39643
Author: Rohit Jaiswal <email address hidden>
Date: Thu May 21 00:05:38 2015 +0000

    Handle database failures on api startup

    Storage layer connections are initialized
    for metering, alarm and event in Pecan hook
    middleware.
    If any of the backends are not responding,
    the connections are retried till maximum
    retries and the exception propagated to the
    Ceilometer-api which fails to startup.

    This fix logs and handles the exception in
    the Pecan hook and allows all connection
    types to be initialized.

    Closes-Bug: 1456944

    Change-Id: Id70693dafc823927b3d72e0371e3e55d4e5588ce

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/196045

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/196045
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=6ea8dd113a3fe1f19b792a9012c17a3386893361
Submitter: Jenkins
Branch: stable/kilo

commit 6ea8dd113a3fe1f19b792a9012c17a3386893361
Author: Rohit Jaiswal <email address hidden>
Date: Thu May 21 00:05:38 2015 +0000

    Handle database failures on api startup

    Storage layer connections are initialized
    for metering, alarm and event in Pecan hook
    middleware.
    If any of the backends are not responding,
    the connections are retried till maximum
    retries and the exception propagated to the
    Ceilometer-api which fails to startup.

    This fix logs and handles the exception in
    the Pecan hook and allows all connection
    types to be initialized.

    Closes-Bug: 1456944

    Change-Id: Id70693dafc823927b3d72e0371e3e55d4e5588ce
    (cherry picked from commit 1f30839494c5b5002173a1f81eb960d735f39643)