With multiple instances of prometheus-ceph-exporter, the Ceph graphs report multiple series error

[Xav Paice]

We have a site where there are two Ceph clusters (regrettably both named 'ceph'). The Ceph applications in Juju are named, e.g. CephA and CephB, and the exporter applications are named cephA-prometheus-ceph-exporter and cephB-prometheus-ceph-exporter.

These both send metrics to the same Prometheus, which aggregates both sets of data. When the Ceph Cluster dashboard loads, we want to know the stats for each Ceph application cephA and CephB, separately, however the queries are not written to do that. The various panels simply report N/A and "Multiple Series Error".

We can edit the dash to get a good result, using queries such as:
ceph_osd_utilization{osd="$osd", job="cephA-prometheus-ceph-exporter"}

This affects all the Ceph dashboards in Grafana.

It would be good if we could split this out, to have a set of Dashboards for each Ceph in use - i.e. in this example a list of dashboards rather than "[Juju] Ceph - Cluster" we have "[Juju] Ceph - Cluster - CephA" and "[Juju] Ceph - Cluster - CephB"

[Yoshi Kadokawa (yoshikadokawa)]

I have an environment with 3 ceph clusters, and I have applied the patch that was in the MP, and successfully the Ceph dashboards are now showing multi-ceph cluster metrics.
I have attached the screenshots of ceph-cluster, ceph-osd and ceph-pools.

[Nobuto Murata (nobuto)]

https://git.launchpad.net/charm-grafana/commit/?id=67f0cdbca9032ebc570cc23f99474bc72246030f

[David Coronel (davecore)]

Is this commited fix available in a charm somewhere in the charm store? (ie. cs:~llama-charmers-next/grafana)

[David Coronel (davecore)]

It looks like cs:~llama-charmers-next/grafana-9 has the commit:

~/charms/next$ charm pull cs:~llama-charmers-next/grafana-9
cs:~llama-charmers-next/grafana-9

~/charms/next$ cd grafana/

~/charms/next/grafana$ grep -ri ceph_monitor_quorum_count *
templates/dashboards/prometheus/CephCluster.json.j2: "expr": "ceph_monitor_quorum_count{job=\"$job\"}",

~/charms/next/grafana$ grep -ri ceph_stuck_degraded_pgs *
templates/dashboards/prometheus/CephCluster.json.j2: "expr": "ceph_stuck_degraded_pgs{job=\"$job\"} + ceph_stuck_stale_pgs{job=\"$job\"} + ceph_stuck_unclean_pgs{job=\"$job\"} + ceph_stuck_undersized_pgs{job=\"$job\"}",
templates/dashboards/prometheus/CephCluster.json.j2: "expr": "ceph_stuck_degraded_pgs{job=\"$job\"}",

[David Coronel (davecore)]

I know it's a little late, but I subscribed ~field-medium for awareness

[David Coronel (davecore)]

I'm looking in the cs:~llama-charmers-next/grafana charms revisions 9 and 11 and it looks like the CephOSD.json.j2 templates don't have the job="$job" fix from https://git.launchpad.net/charm-grafana/commit/?id=67f0cdbca9032ebc570cc23f99474bc72246030f

The CephPools.json.j2 and CephCluster.json.j2 templates have the fix, but not CephOSD.json.j2

Was CephOSD just missed?