COS integration doesn't support remote_write

[Nobuto Murata]

By following the charm documentation for COS integration:

> Alternatively, integration with the [COS Lite][cos-lite] observability
> stack is available via the metrics-endpoint relation.
>
> Relating to prometheus-k8s via the metrics-endpoint interface (as is
> found in the [COS Lite][cos-lite] bundle) will send metrics to
> prometheus.

It will add scrape_configs to scrape the ceph-mon endpoints from Prometheus so the behavior is different from what is stated in the documentation ("will send metrics to prometheus").

In any case, traffic in the ceph-mon -> prometheus direction is important in the COS world. Since grafana agent is leveraging prometheus remote_write so the direction is always grafana-agent -> prometheus and that makes the network design and firewall policy simpler to reduce the headache of operations.

====
$ kubectl -n cos exec -it prometheus-0 -c prometheus -- cat /etc/prometheus/prometheus.yml

...

scrape_configs:

...

- honor_labels: true
  job_name: juju_ceph_3cfee97d_ceph-mon_prometheus_scrape-2
  metrics_path: /metrics
  relabel_configs:
  - &id001
    regex: (.*)
    separator: _
    source_labels:
    - juju_model
    - juju_model_uuid
    - juju_application
    - juju_unit
    target_label: instance
  static_configs:
  - labels:
      juju_application: ceph-mon
      juju_charm: ceph-mon
      juju_model: ceph
      juju_model_uuid: 3cfee97d-57fe-4312-8b17-49d9a03924f6
      juju_unit: ceph-mon/2
    targets:
    - 192.168.151.107:9283
- honor_labels: true
  job_name: juju_ceph_3cfee97d_ceph-mon_prometheus_scrape-1
  metrics_path: /metrics
  relabel_configs:
  - *id001
  static_configs:
  - labels:
      juju_application: ceph-mon
      juju_charm: ceph-mon
      juju_model: ceph
      juju_model_uuid: 3cfee97d-57fe-4312-8b17-49d9a03924f6
      juju_unit: ceph-mon/1
    targets:
    - 192.168.151.112:9283
- honor_labels: true
  job_name: juju_ceph_3cfee97d_ceph-mon_prometheus_scrape-0
  metrics_path: /metrics
  relabel_configs:
  - *id001
  static_configs:
  - labels:
      juju_application: ceph-mon
      juju_charm: ceph-mon
      juju_model: ceph
      juju_model_uuid: 3cfee97d-57fe-4312-8b17-49d9a03924f6
      juju_unit: ceph-mon/0
    targets:
    - 192.168.151.109:9283
====

[Nobuto Murata (nobuto)]

My gut feeling is that new relation with grafana-agent would be required instead of using the existing juju-info relation.

  juju-info:
    description: |
      `juju-info` provides basic compatibility with all charms.
      If all you want is /var/log logs and node_exporter metrics,
      this relation will be enough.
    interface: juju-info
    scope: container
  cos-agent:
    description: |
      `cos-agent` is a dedicated relation for the grafana agent machine
      charm. It will allow you to set up custom scrape jobs, fetch files
      from arbitrary locations, send alert rules, dashboards, etc.
    interface: cos_agent
    scope: container

[Nobuto Murata (nobuto)]

One advantage of leveraging cos-agent is that we can limit the listen address of the exporter to localhost.

Currently the port is open to everywhere so anybody who has network connectivity can get all the metrics of Ceph.

$ sudo ss -tlnp | grep mgr
LISTEN 0 512 192.168.151.109:6800 0.0.0.0:* users:(("ceph-mgr",pid=14161,fd=27))
LISTEN 0 512 192.168.151.109:6801 0.0.0.0:* users:(("ceph-mgr",pid=14161,fd=28))
LISTEN 0 5 192.168.151.109:8443 0.0.0.0:* users:(("ceph-mgr",pid=14161,fd=36))
LISTEN 0 5 *:9283 *:* users:(("ceph-mgr",pid=14161,fd=33))

ref: https://docs.ceph.com/en/quincy/mgr/prometheus/#configuration

[Nobuto Murata (nobuto)]

https://github.com/canonical/grafana-agent-operator/issues/25