Comment 3 for bug 2042891

This part of the code will never be retried if the Ceph is not bootstrapped at that point.

    def _on_relation_changed(self, event):
        """Enable prometheus on relation change"""
        if self._charm.unit.is_leader() and ceph_utils.is_bootstrapped():
            logger.debug(
                "is_leader and is_bootstrapped, running rel changed: %s", event
            )
            mgr_config_set_rbd_stats_pools()
            ceph_utils.mgr_enable_module("prometheus")
            logger.debug("module_enabled")
            self.update_alert_rules()
            super()._on_relation_changed(event)

[when the relation is added at the deployment time (via bundle, etc.)]

# cat /var/log/juju/unit-ceph-mon-0.log | egrep 'metrics-endpoint-relation-|leader-|not bootstrapped'; ll /var/lib/ceph/mon/ceph-ceph-mon-1/done
2024-02-28 10:08:11 INFO juju.worker.uniter.operation runhook.go:186 ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
2024-02-28 10:08:16 INFO juju.worker.uniter resolver.go:165 found queued "leader-elected" hook
2024-02-28 10:08:35 INFO juju.worker.uniter.operation runhook.go:186 ran "leader-elected" hook (via hook dispatching script: dispatch)
2024-02-28 10:08:39 INFO unit.ceph-mon/0.juju-log server.go:325 Ceph is not bootstrapped, skipping upgrade checks.
2024-02-28 10:08:40 INFO unit.ceph-mon/0.juju-log server.go:325 Ceph is not bootstrapped, skipping upgrade checks.
2024-02-28 10:08:50 INFO juju.worker.uniter.operation runhook.go:186 ran "metrics-endpoint-relation-joined" hook (via hook dispatching script: dispatch)
2024-02-28 10:09:18 INFO juju.worker.uniter.operation runhook.go:186 ran "metrics-endpoint-relation-changed" hook (via hook dispatching script: dispatch)

-rw-r--r-- 1 root root 0 Feb 28 10:10 /var/lib/ceph/mon/ceph-ceph-mon-1/done

^^^ bootstrap was complete *after* running metrics-endpoint-relation-changed

[when the relation is added after the Ceph model settled]

# cat /var/log/juju/unit-ceph-mon-*.log | egrep 'metrics-endpoint-relation-|leader-|not bootstrapped'; ll /var/lib/ceph/mon/ceph-*/done
2024-02-28 04:43:43 INFO juju.worker.uniter resolver.go:165 found queued "leader-elected" hook
2024-02-28 04:43:46 INFO juju.worker.uniter.operation runhook.go:186 ran "leader-elected" hook (via hook dispatching script: dispatch)
2024-02-28 04:43:48 INFO unit.ceph-mon/1.juju-log server.go:325 Ceph is not bootstrapped, skipping upgrade checks.
2024-02-28 04:43:49 INFO unit.ceph-mon/1.juju-log server.go:325 Ceph is not bootstrapped, skipping upgrade checks.
2024-02-28 04:54:05 INFO juju.worker.uniter.operation runhook.go:186 ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
2024-02-28 04:54:07 INFO juju.worker.uniter.operation runhook.go:186 ran "metrics-endpoint-relation-joined" hook (via hook dispatching script: dispatch)
2024-02-28 04:54:15 INFO juju.worker.uniter.operation runhook.go:186 ran "metrics-endpoint-relation-changed" hook (via hook dispatching script: dispatch)

-rw-r--r-- 1 root root 0 Feb 28 04:44 /var/lib/ceph/mon/ceph-juju-704761-1-lxd-1/done

^^^ bootstrap was complete *before* running metrics-endpoint-relation-changed