alert.yaml.rules won't be passed and set up

[Nobuto Murata]

ceph-mon quincy/stable 193

By reading the following patchset that landed to both the master branch and quincy.2 branch, there seems to be a function to send the definition of Ceph alert rules over the relation.
https://review.opendev.org/c/openstack/charm-ceph-mon/+/858940

However, even after relating ceph-mon:metrics-endpoint with prometheus, the alerts were not defined in prometheus.

No "alert_rules" in the relation data either.

  - relation-id: 35
    endpoint: configurable-scrape-jobs
    cross-model: true
    related-endpoint: metrics-endpoint
    application-data:
      scrape_jobs: '[{"metrics_path": "/metrics", "static_configs": [{"targets": ["*:9283"]}]}]'
      scrape_metadata: '{"model": "ceph", "model_uuid": "42cc6513-d213-4d0e-8ddd-db0723ccc68e",
        "application": "ceph-mon", "unit": "ceph-mon/0", "charm_name": "ceph-mon"}'
    related-units:
      remote-a273a88c75ca426a81e9f2b44afc9c22/0:

[Nobuto Murata (nobuto)]

By looking into the code, it looks like the charm is supposed to download the charm resource from the Juju controller into:
/var/lib/juju/agents/unit-ceph-mon-*/charm/files/prometheus_alert_rules/alert.yaml.rules

But there was no file there in any of the units.

However, explicitly running the upgrade-charm hook will call self.metrics_endpoint.update_alert_rules() and the file "alert.yaml.rules" will show up and the relation data is populated.

$ juju exec -u ceph-mon/leader "JUJU_DISPATCH_PATH=hooks/upgrade-charm ./dispatch"

I'm not sure why the initial update_alert_rules() is not called or processed.

[Nobuto Murata (nobuto)]

It's this: https://review.opendev.org/c/openstack/charm-ceph-mon/+/866139

It's been fixed a year ago but never been backported.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/901426

[Nobuto Murata (nobuto)]

Subscribing ~field-high to get the stable backport done.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/901426
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/010a6e278673b66eaec222a0855746d45f7a9279
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit 010a6e278673b66eaec222a0855746d45f7a9279
Author: Peter Sabaini <email address hidden>
Date: Wed Nov 30 12:41:41 2022 +0100

    Fix: init alert rules on rel change

    Check for alert rules early, on first metrics-endpoint rel change

    Closes-Bug: #2043758

    Change-Id: Iea39c33c614d204ee39ad39da68c31d213ed19e6
    (cherry picked from commit 1cf9d4d228ca4df801094755d8247d8a0d8ff6d9)