| 2020-09-10 23:41:02 |
Andrea Ieri |
description |
If an application providing a scrape target endpoint bound to a non-default space is related to prometheus, the charm will fail to configure the correct ingress address for that scraping job.
Example:
$ juju show-application cloudstats
cloudstats:
charm: cloudstats
series: bionic
constraints:
spaces:
- default-space
- internal-space
- oam-space
principal: true
exposed: false
remote: false
endpoint-bindings:
"": internal-space
grafana-dashboard: internal-space
openstack-admin: internal-space
prometheus-client: internal-space
prometheus-scrape-target: oam-space
$ juju run -u cloudstats/0 -- network-get prometheus-scrape-target --ingress-address
10.17.101.27
$ juju run -u prometheus/0 -- relation-get -r376 - cloudstats/0
egress-subnets: 10.17.101.27/32
hostname: 10.62.0.189 <--- this is cloudstats/0 address on the internal-space
ingress-address: 10.17.101.27
labels: '{"host": "<ops.model.Unit cloudstats-0>"}'
metrics_path: /metrics
port: "9748"
private-address: 10.17.101.27
scrape_interval: 2m
The target will be rendered in prometheus.yml as follows:
# services related to the 'scrape' endpoint
- job_name: 'cloudstats'
metrics_path: '/metrics'
scrape_interval: 2m
static_configs:
- targets:
- 10.62.0.189:9748
The prometheus charm appears to be using the 'hostname' value instead of the ingress-address. |
Prometheus will render a scrape target configuration based on the value of relation data key 'hostname', which cannot be guaranteed to be a valid, reachable IP.
This field is provided by the related application, and doesn't seem to serve any other purpose than duplicating the ingress address.
Prometheus should ignore the hostname relation data key and configure scrape targets based on the ingress address of their relation endpoint.
Example of a buggy scrape target:
[assume the prometheus scrape endpoint is bound to the oam-space, and cloudstats:prometheus-scrape-target is related to prometheus:scrape]
$ juju show-application cloudstats
cloudstats:
charm: cloudstats
series: bionic
constraints:
spaces:
- default-space
- internal-space
- oam-space
principal: true
exposed: false
remote: false
endpoint-bindings:
"": internal-space
grafana-dashboard: internal-space
openstack-admin: internal-space
prometheus-client: internal-space
prometheus-scrape-target: oam-space
$ juju run -u cloudstats/0 -- network-get prometheus-scrape-target --ingress-address
10.17.101.27
$ juju run -u prometheus/0 -- relation-get -r376 - cloudstats/0
egress-subnets: 10.17.101.27/32
hostname: 10.62.0.189 <--- address on the internal-space
ingress-address: 10.17.101.27 <--- the correct address on the oam-space
labels: '{"host": "<ops.model.Unit cloudstats-0>"}'
metrics_path: /metrics
port: "9748"
private-address: 10.17.101.27
scrape_interval: 2m
The target will be rendered in prometheus.yml as follows:
# services related to the 'scrape' endpoint
- job_name: 'cloudstats'
metrics_path: '/metrics'
scrape_interval: 2m
static_configs:
- targets:
- 10.62.0.189:9748
Although most applications providing a scrape endpoint will correctly populate the hostname key with their ingress address, this seems redundant when using the ingress-address key would be more reliable. |
|
