prometheus restart needed when new target relations appear

[Adam Dyess]

Many times I do not see prometheus scraping the endpoints which have been provided via the target relation and which have been written to the config file /var/snap/prometheus/*/prometheus.yml.

I will check the relation data is correct

$ juju run -u telegraf/<unit> "relation-get -r prometheus-client:$rid - <unit>"

then check the config file

$ juju ssh prometheus/0 "cat /var/snap/prometheus/*/prometheus.yml" | less

to find the IP and PORT of the target prometheus should scrape and it's in the config

But it requires the prometheus service to be restarted

$ juju run -u prometheus/0 "service snap.prometheus.prometheus restart"

Shouldn't this be done automatically when a new target relation is changed -- or at least anytime the config file is updated

[Stephan Pampel (stephanpampel)]

In general Prometheus does not need to restart to reload its' config: "Prometheus can reload its configuration at runtime." https://prometheus.io/docs/prometheus/latest/configuration/configuration/

A reload can be triggered via SIGHUP process signal or http request to '<ip>:<port>/-/reload' if prometheus runs with --web.enable-lifecycle.

[Stephan Pampel (stephanpampel)]

The charm uses SIGHUP by default whenever the flag set_state("prometheus.do-reload") is present (e.g. after config change https://git.launchpad.net/charm-prometheus2/tree/src/reactive/prometheus.py#n425)

It looks to me that the code is correctly sending the signal to prometheus. https://git.launchpad.net/charm-prometheus2/tree/src/reactive/prometheus.py#n580

There was a previous bug about the reloading: https://bugs.launchpad.net/charm-prometheus2/+bug/1891942
and it go fixed:
https://git.launchpad.net/charm-prometheus2/commit/src/reactive/prometheus.py?id=798b492c33bc2a3c83ac0c096aaf8d812ef1d997