Only one monitor address is populated into the config
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Prometheus Ceph Exporter Charm |
Fix Released
|
Medium
|
Drew Freiberger |
Bug Description
Rocky/Bionic deployment
Ceph cluster consists of 3 monitors and a number of OSDs.
When adding prometheus-
However, the ceph config used by the exporter contains only one mon address:
root@juju-
#######
# [ WARNING ]
# glance configuration file maintained by Juju
# local changes may be overwritten.
#######
[global]
auth_supported = cephx
#keyring = /etc/ceph/
keyring = /var/snap/
mon host = 10.11.30.46:6789
At the same time, for example, nova-computes have all 3 mons populated in their config. This must be caused by different implementation of obtaining the addresses of the mons in the layer vs how it is done in Nova:
https:/
Related branches
- Paul Goins: Approve
- Joe Guo (community): Approve
- Drew Freiberger: Pending requested
- Canonical IS Reviewers: Pending requested
- Joel Sing: Pending requested
-
Diff: 49 lines (+16/-1)1 file modifiedreactive/prometheus_ceph_exporter.py (+16/-1)
- Joe Guo (community): Needs Fixing
- BootStack Reviewers: Pending requested
-
Diff: 4404 lines (+4265/-19)11 files modifiedREADME.md (+2/-16)
dev/null (+0/-1)
reactive/prometheus_ceph_exporter.py (+3/-1)
templates/dashboards/CephCluster.json (+2204/-0)
templates/dashboards/CephOSD.json (+1309/-0)
templates/dashboards/CephPools.json (+722/-0)
tests/bundles/bionic.yaml (+7/-0)
tests/bundles/focal.yaml (+7/-0)
tests/bundles/xenial.yaml (+7/-0)
tests/requirements.txt (+1/-1)
tests/tests.yaml (+3/-0)
- Drew Freiberger (community): Needs Resubmitting
- Joel Sing: Pending requested
- Canonical IS Reviewers: Pending requested
-
Diff: 49 lines (+16/-1)1 file modifiedreactive/prometheus_ceph_exporter.py (+16/-1)
- Joe Guo (community): Approve
- Xav Paice (community): Approve
- Drew Freiberger (community): Needs Resubmitting
- Paul Goins: Pending requested
-
Diff: 648 lines (+227/-79)14 files modified.gitignore (+24/-1)
Makefile (+1/-1)
README.md (+30/-39)
config.yaml (+6/-0)
dev/null (+0/-1)
layer.yaml (+8/-8)
metadata.yaml (+8/-0)
reactive/prometheus_ceph_exporter.py (+115/-15)
tests/requirements.txt (+7/-1)
tests/test_prometheus_ceph_exporter.py (+6/-5)
tests/tests.yaml (+3/-3)
tests/unit/test_reactive_ceph_exporter.py (+11/-1)
tox.ini (+7/-4)
wheelhouse.txt (+1/-0)
- Joel Sing (community): Needs Fixing
- Canonical IS Reviewers: Pending requested
-
Diff: 46 lines (+9/-6)1 file modifiedreactive/prometheus-ceph-exporter.py (+9/-6)
Changed in prometheus-ceph-exporter-charm: | |
importance: | Undecided → Medium |
Changed in prometheus-ceph-exporter-charm: | |
status: | New → Incomplete |
status: | Incomplete → In Progress |
assignee: | nobody → Vladimir Grevtsev (vlgrevtsev) |
Changed in charm-prometheus-ceph-exporter: | |
status: | In Progress → Fix Released |
Based on the output produced during a clean redeploy, I can see that the charm logic is incorrect in handling config changes.
Only the first config-changed event with all necessary data present is handled /github. com/openstack- charmers/ charm-interface -ceph-client/ blob/0e191799fc 9ee2d7662cd77f0 30790fa850c6dea /requires. py#L40- L48
https:/
at this time other units do not have ceph-public-address exposed via relation data yet, therefore, the following code simply ignores the lack of those:
https:/ /github. com/openstack- charmers/ charm-interface -ceph-client/ blob/0e191799fc 9ee2d7662cd77f0 30790fa850c6dea /requires. py#L108- L109
Then configure_exporter is executed and exporter.started is set which triggers ceph.conf rendering with only one monitor address present.
@when(' ceph.available' ) 'exporter. started' ) exporter( ceph_client) :
@when_not(
def configure_
This flag is never cleared and the reactive code of the exporter does not have any intelligent relation data change processing with regards to that.
This is confirmed by the timestamps:
# the first and the only rendering event happened at 02:46:45 d85cbd- 12-lxd- 2:/var/ lib/juju/ agents/ unit-prometheus -ceph-exporter- 0/charm# grep ceph.conf /var/log/ juju/unit- prometheus- ceph-exporter- 0.log prometheus- ceph-exporter/ current/ ceph.conf root:root 444
root@juju-
2018-12-07 02:46:45 DEBUG juju-log ceph:295: Writing file /var/snap/
# joined events - nothing rendered after that as expected d85cbd- 12-lxd- 2:/var/ lib/juju/ agents/ unit-prometheus -ceph-exporter- 0/charm# grep ceph-relation- joined /var/log/ juju/unit- prometheus- ceph-exporter- 0.log joined joined lxc joined joined lxc joined joined lxc
root@juju-
2018-12-07 02:28:11 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-
2018-12-07 02:28:12 DEBUG ceph-relation-
2018-12-07 02:28:13 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-
2018-12-07 02:28:13 DEBUG ceph-relation-
2018-12-07 02:29:29 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-
2018-12-07 02:29:29 DEBUG ceph-relation-
root@juju- d85cbd- 12-lxd- 2:/var/ lib/juju/ agents/ unit-prometheus -ceph-exporter- 0/charm# grep ceph-relation- changed /var/log/ juju/unit- prometheus- ceph-exporter- 0.log changed changed lxc changed changed lxc changed changed lxc changed changed lxc changed active
2018-12-07 02:28:12 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-
2018-12-07 02:28:12 DEBUG ceph-relation-
2018-12-07 02:28:13 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-
2018-12-07 02:28:13 DEBUG ceph-relation-
2018-12-07 02:30:10 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-
2018-12-07 02:30:10 DEBUG ceph-relation-
2018-12-07 02:46:45 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-
2018-12-07 02:46:45 DEBUG ceph-relation-
2018-12-07 02:46:55 DEBUG ceph-relation-
^^^ this corresponds to the rendering event and no rendering attempts happen for other relation-changed events which results in the incorrect file content changed changed lxc
2018-12-07 02:47:26 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-
2018-12-07 02:47:26 DEBUG ceph-relation-
2018-12-07 02:47:...