scrape target 'hostname' is not guaranteed to be reachable

[Andrea Ieri]

Prometheus will render a scrape target configuration based on the value of relation data key 'hostname', which cannot be guaranteed to be a valid, reachable IP.

This field is provided by the related application, and doesn't seem to serve any other purpose than duplicating the ingress address.

Prometheus should ignore the hostname relation data key and configure scrape targets based on the ingress address of their relation endpoint.

Example of a buggy scrape target:

[assume the prometheus scrape endpoint is bound to the oam-space, and cloudstats:prometheus-scrape-target is related to prometheus:scrape]

$ juju show-application cloudstats
cloudstats:
  charm: cloudstats
  series: bionic
  constraints:
    spaces:
    - default-space
    - internal-space
    - oam-space
  principal: true
  exposed: false
  remote: false
  endpoint-bindings:
    "": internal-space
    grafana-dashboard: internal-space
    openstack-admin: internal-space
    prometheus-client: internal-space
    prometheus-scrape-target: oam-space

$ juju run -u cloudstats/0 -- network-get prometheus-scrape-target --ingress-address
10.17.101.27

$ juju run -u prometheus/0 -- relation-get -r376 - cloudstats/0
egress-subnets: 10.17.101.27/32
hostname: 10.62.0.189 <--- address on the internal-space
ingress-address: 10.17.101.27 <--- the correct address on the oam-space
labels: '{"host": "<ops.model.Unit cloudstats-0>"}'
metrics_path: /metrics
port: "9748"
private-address: 10.17.101.27
scrape_interval: 2m

The target will be rendered in prometheus.yml as follows:

# services related to the 'scrape' endpoint
  - job_name: 'cloudstats'
    metrics_path: '/metrics'
    scrape_interval: 2m
    static_configs:
      - targets:
          - 10.62.0.189:9748

Although most applications providing a scrape endpoint will correctly populate the hostname key with their ingress address, this seems redundant when using the ingress-address key would be more reliable.