Some metrics are not collected with Ceph Nautilus

[Yoshi Kadokawa]

With Ceph Nautilus, some of the Ceph Template dashboard for Grafana fails to load.
In Grafana, I see the following logs in /var/log/juju/unit-grafana-0.log

2020-03-12 05:42:36 DEBUG juju-log Skipping Dashboard Template: CephCluster.json.j2 missing 2 metrics.Missing: ceph_osd_perf_apply_latency_seconds, ceph_osd_perf_commit_latency_seconds
2020-03-12 05:42:36 DEBUG juju-log Skipping Dashboard Template: CephOSD.json.j2 missing 2 metrics.Missing: ceph_osd_perf_apply_latency_seconds, ceph_osd_perf_commit_latency_seconds

It looks like the following metrics are not available or not collected.
- ceph_osd_perf_apply_latency_seconds
- ceph_osd_perf_commit_latency_seconds

According to this[0], there is a fix for this in ceph-exporter for Nautilus.
However, the prometheus-ceph-exporter snap is built from the default branch(luminous),
so this fix is not available from snap package.

[0] https://github.com/digitalocean/ceph_exporter/commit/6622c2f7a3f44be47738c02270c2ec479d82ecff#diff-1536b81b5897f95267830a7c215ad5ab

[Nobuto Murata (nobuto)]

The github project doesn't have a release or a tag from the nautilus branch.
https://github.com/digitalocean/ceph_exporter/tags

So a separate snap channel would be required for nautilus.

[Yoshi Kadokawa (yoshikadokawa)]

I have tested by using the snap build from the nautilus branch, but did not work.
It looks like that the latest fix in nautilus branch in https://github.com/digitalocean/ceph_exporter has reverted the fix for OSD Latency metrics.

However, I could confirm with the snap build by specifying the commit that has the fix.
https://github.com/digitalocean/ceph_exporter/pull/125/files

This is the snapcraft.yaml that I used.
https://git.launchpad.net/~yoshikadokawa/snap-prometheus-ceph-exporter/commit/?h=nautilus-support

[Xav Paice (xavpaice)]

This is a snap issue, following up on that, have marked invalid for the charm since there's nothing we can change in the charm to help.

[Nobuto Murata (nobuto)]

@Xav, even if the snap could be updated, it probably needs multiple channels to support multiple releases of Ceph. Thus, the ceph-exporter charm would also need a config option to specify a channel for the snap like Graylog:
https://jaas.ai/graylog#charm-config-channel

[Xav Paice (xavpaice)]

Related: https://bugs.launchpad.net/snap-prometheus-ceph-exporter/+bug/1884076 (request for an update to the snap)

[Xav Paice (xavpaice)]

https://forum.snapcraft.io/t/request-for-tracks-for-prometheus-ceph-exporter/19407 requesting tracks in the snap.

For the snap itself, I've uploaded a v3.0.0 (nautilus) snap to the latest/edge channel if we need it.

The latest release of the prometheus-ceph-exporter charm will have the ability to switch channels, but changing that will need manual intervention (see LP:#1891582).

[David O Neill (dmzoneill)]

need to promote -next to production.

[Vladimir Grevtsev (vlgrevtsev)]

Can confirm that latest/edge works fine, the metrics are in-place and the dashboards has been imported successfully.

However, with the edge snap we're having another issue: https://bugs.launchpad.net/charm-prometheus-ceph-exporter/+bug/1899030

[Drew Freiberger (afreiberger)]

Upstream bug filed against the project. The ceph_exporter team is working on a one-version-to-rule-them-all rewrite. I believe we should hold until that is ready. We could write up something in the charm to detect if Nautilus+ is running and force the snap channel to edge.
https://github.com/digitalocean/ceph_exporter/issues/182

I'd hate to unwind the separate snapstore tracks across many clouds back to a single track once the upstream project finishes their work.

[Jose Guedez (jfguedez)]

As both the snap (edge channel) [0], and the charm change to enable changing channels have been released [1]. Setting status to fix released.

[0] https://bugs.launchpad.net/charm-prometheus-ceph-exporter/+bug/1867100/comments/6
[1] https://bugs.launchpad.net/charm-prometheus-ceph-exporter/+bug/1891582