gnocchi not able to connect to ceph cluster

[James Page]

Following stacktrace is seen when gnocchi attempts to sync its ceph storage:

unit-gnocchi-0: 11:06:57 DEBUG unit.gnocchi/0.identity-service-relation-changed 2019-03-13 11:06:57,684 [14307] INFO gnocchi.common.ceph: Ceph storage backend use 'rados' python library
unit-gnocchi-0: 11:06:57 DEBUG unit.gnocchi/0.identity-service-relation-changed 2019-03-13 11:06:57,704 [14307] ERROR gnocchi.utils: Unable to initialize storage driver
unit-gnocchi-0: 11:06:57 DEBUG unit.gnocchi/0.identity-service-relation-changed Traceback (most recent call last):
unit-gnocchi-0: 11:06:57 DEBUG unit.gnocchi/0.identity-service-relation-changed File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 298, in call
unit-gnocchi-0: 11:06:57 DEBUG unit.gnocchi/0.identity-service-relation-changed result = fn(*args, **kwargs)
unit-gnocchi-0: 11:06:57 DEBUG unit.gnocchi/0.identity-service-relation-changed File "/usr/lib/python3/dist-packages/gnocchi/storage/__init__.py", line 117, in get_driver
unit-gnocchi-0: 11:06:57 DEBUG unit.gnocchi/0.identity-service-relation-changed conf.storage, coord)
unit-gnocchi-0: 11:06:57 DEBUG unit.gnocchi/0.identity-service-relation-changed File "/usr/lib/python3/dist-packages/gnocchi/storage/ceph.py", line 48, in __init__
unit-gnocchi-0: 11:06:57 DEBUG unit.gnocchi/0.identity-service-relation-changed self.rados, self.ioctx = ceph.create_rados_connection(conf)
unit-gnocchi-0: 11:06:57 DEBUG unit.gnocchi/0.identity-service-relation-changed File "/usr/lib/python3/dist-packages/gnocchi/common/ceph.py", line 69, in create_rados_connection
unit-gnocchi-0: 11:06:57 DEBUG unit.gnocchi/0.identity-service-relation-changed conn.connect()
unit-gnocchi-0: 11:06:57 DEBUG unit.gnocchi/0.identity-service-relation-changed File "rados.pyx", line 884, in rados.Rados.connect
unit-gnocchi-0: 11:06:57 DEBUG unit.gnocchi/0.identity-service-relation-changed rados.PermissionError: [errno 1] error connecting to the cluster
unit-gnocchi-0: 11:07:57 DEBUG unit.gnocchi/0.identity-service-relation-changed 2019-03-13 11:07:57,745 [14307] INFO gnocchi.common.ceph: Ceph storage backend use 'rados' python library
unit-gnocchi-0: 11:07:57 DEBUG unit.gnocchi/0.identity-service-relation-changed 2019-03-13 11:07:57,760 [14307] ERROR gnocchi.utils: Unable to initialize storage driver

This is not actually due to the gnocchi charm; the ceph-mon charm executes the client_relation handler outside of the original hook context for the gnocchi/ceph-mon relation instance - in this case its done inside of the ceph-osd relation - the key on the side has an id of 'ceph-osd' but is created with an id of 'gnocchi' on the consuming side resulting in it being rejected.

[Frode Nordahl (fnordahl)]

https://review.openstack.org/#/c/643006/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/643006
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=fe3097f9d7940a74dc4635ce8d67db22426b3917
Submitter: Zuul
Branch: master

commit fe3097f9d7940a74dc4635ce8d67db22426b3917
Author: James Page <email address hidden>
Date: Wed Mar 13 11:05:10 2019 +0000

    Ensure relation id is passed to remote_service_name

    Its possible for the client_relation handler to be called from
    outside of the original hook context; as a result use of
    remote_service_name is not safe as this will evaluate in the
    current hook context, rather than the relation_id/unit context
    provided in the function parameters.

    Passing the relation_id to the helper ensures that the
    remote units for the intended relation are used to generate
    the remote service name.

    Change-Id: I2305a111c0a618156208e4f96acda5e63736051c
    Closes-Bug: 1819892

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/643340

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/643340
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=2dd7491a3e4ea78428a0b99e0a8c450cd820a95a
Submitter: Zuul
Branch: master

commit 2dd7491a3e4ea78428a0b99e0a8c450cd820a95a
Author: James Page <email address hidden>
Date: Thu Mar 14 13:54:17 2019 +0000

    Ensure remote service name is resolved

    In the event that no units have presented on a relation, the
    remote_service_name function will return 'None'.

    Ensure this situation is detected and skip broker request
    processing if this is the case.

    Change-Id: I210ed914326761b7b353a7b86303d8877a112ca5
    Related-Bug: 1819892