create and replace target:http relation with a new prometheus target interface

[Zachary Zehring]

Due to the migration of charms to use interface-grafana-dashboard to import dashboards to Grafana, prometheus needs to have a way to communicate to these charms various details needed for rendering the details of the dashboard. Grafana currently gets this information from interface-grafana-source, where prometheus provides the url to query metrics.

This new interface would need to relay the following information:

- datasource name: comprised of the service name, or the name of the application, and the description, currently set to Juju generated source)
- url: prometheus's URL used to query metrics. For example, in telegraf charm, there's logic around rendering the dashboard template regarding these metrics

With this interface, a provider for interface-grafana-dashboard can generate a template correctly, without any hardcoding of values, by grabbing them from the prometheus relation and passing the resulting dashboard along the relation to Grafana.

[Alvaro Uria (aluria)]

Thank you for the bug. Could you describe a little more what you would expect (ie. share what a bundle.yaml would look like).

Exported dashboards can have an "__input" key to convert a variable used across the JSON file (ie. "<< datasource >>") to the input shared.

More information can be found at:
* Data source API [1]
* Dashboard API [2]
* Export/Import [3]
* Provisioning (Datasources) [4]

Some questions:
1) What is the expectation in terms of Grafana relations to datasources?
2) Is the goal to have a single datasource (Prometheus) and make any dashboard use such datasource? Or should Grafana be able to relate to multiple datasources and configure dashboards using different datasources?
3) In case of multiple datasources, shouldn't the dashboard JSON file have some information on the type of datasource to use (ie. Prometheus, or Elasticsearch...)?
4) In case of multiple datasources, what if there are multiple of the same type (2 Prometheus datasources)

When grafana relates to prometheus via the grafana-source interface, Grafana already gets information about the datasource (which could be used for any dashboard).

OTOH, the other side of the grafana-dashboard relation could check if it also has a target endpoint available, and share the datasource pointing to that target. However, it could happen that the same prometheus exporter (managing its own dashboards) is related to multiple scrapers (although unlikely and we could state that such approach is not supported).

1. https://grafana.com/docs/grafana/latest/http_api/data_source/
2. https://grafana.com/docs/grafana/latest/http_api/dashboard/
3. https://grafana.com/docs/grafana/latest/reference/export_import/
4. https://grafana.com/docs/grafana/latest/administration/provisioning/#datasources

[Zachary Zehring (zzehring)]

So I think my description is a bit unclear. I think we need to create an interface to replace the existing http interface Prometheus2 currently uses to communicate with targets. This relation does not provide enough details for the target charm to import dashboards to grafana. The two pieces of data it needs (so far) is the datasource name and prometheus endpoint to query for metrics. Right now, charms consuming grafana-dashboard to import dashboard have to hardcode in the datasource, as there's no reliable way to get the datasource (ie how kubernetes-master does it).

To answer your questions:

1. No change. It should still relate to datasources. The change is on prometheus communicating it's datasource name to target/dashboard provider charms.

2. Grafana should be able to relate to multiple datasource, and those datasources are communicated to the charm creating the grafana-dashboard over relation data (newly proposed prom2 interface).

3. Exactly, the problem today is that it doesn't. It's hardcoded either in the charm code itself or config option.

4. Wouldn't be an issue. Even in the instance where a target would have 2 prometheus applications it's relaying data to, it would be able to iterate through available datasources and create dashboards for each.

Provider charm usage could look something like this:

@when('endpoint.prometheus-client.has_datasources',
      'endpoint.grafana.ready_for_dashes')
@when_not('grafana.configured')
def register_grafana_dash():
    prometheus = endpoint_from_flag('endpoint.prometheus-client.has_datasources')
    grafana = endpoint_from_flag('endpoint.grafana.ready_for_dashes')
    for prometheus_source in prometheus.prometheus_sources:
        # prom_src.url is for some metric evaluations that can be passed to dashboard rendering
        dash_dict = load_grafana_dash(prometheus_source.datasource_name, prometheus_source.url)
        grafana.register_dashboard('$charm - {datasource_name}', dashboard=dash_dict)
    set_flag('grafana.configured')

Let me know if that cleared some things up.

- Zack

[Alvaro Uria (aluria)]

I have discussed this bug with Zack and here are a few clarifications:
* the http interface (used between prometheus and the exporters such as prometheus-libvirt-exporter or telegraf) only shared the hostname and port
* prometheus exporters would need the datasource name so they can build a correct dashboard from the template, to be pushed via the grafana-dashboard interface
* the grafana-dashboard interface relates grafana and the exporters, to gather specific dashboards
* Grafana creates a datasource name when related to a datasource (ie. prometheus) by naming it "{{juju-appname}} - Juju generated source"

Other comments:
* if the exporters are related to more than one datasource (ie. prometheus1 and prometheus2 apps), then it would push 2 dashboards which are the same but with different configured datasources (one for prometheus1 and another one for prometheus2 datasource)

* for backward compatibility, the "prometheus:target <-> exporter" relation should keep using the http interface. An extra interface should be shared if more information needs to be shared (ie. datasource type and name).

My suggestion is to:
0)
juju add-relation grafana:grafana-dashboard exporter
juju add-relation grafana:grafana-source prometheus1
juju add-relation prometheus1:target exporter

1) Make exporters infer the datasource by "{{juju-appname}} - Juju generated source" (caution with cross-model relations, which could have 2 apps with the same name in different models

2) Extend the grafana-dashboard interface to not only share "dashboard" (serialized string) and "name" (dashboard title), but also share a "datasources" (list of datasources, in case the scrape endpoint [uses the http interface] is related to more than one Juju app). If the "datasources" attribute is empty, the dashboard will be imported into Grafana as is (otherwise, Jinja2 rendering would be used).

2.a) As a side note, "datasources" may contain multiple data sources but this instance of Grafana may only be related to a single datasource. In such case, the rest of the datasources would be skipped (a single dashboard would be imported with the correct datasource label).