prometheus doesn't restart when adding a new relation

[Garrett Neugent]

When adding the prometheus-libvirt-exporter relation to a customer cloud, we found that the prometheus config file was correctly updated with the new endpoints for pulling data, but the snap was not restarted as part of config-changed:

$ service snap.prometheus.prometheus status
● snap.prometheus.prometheus.service - Service for snap application prometheus.prometheus
   Loaded: loaded (/etc/systemd/system/snap.prometheus.prometheus.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2020-09-28 19:02:18 UTC; 8 months 29 days ago

$ ls -al /var/snap/prometheus/32/prometheus.yml
-r--r----- 1 root root 8025 Jun 28 15:59 /var/snap/prometheus/32/prometheus.yml

Charm info:

  prometheus:
    charm: cs:prometheus2-19
    series: bionic
    os: ubuntu
    charm-origin: jujucharms
    charm-name: prometheus2
    charm-rev: 19
    charm-version: ae30894
    can-upgrade-to: cs:prometheus2-22
    exposed: false
    application-status:
      current: active
      message: Ready
      since: 28 Jun 2021 15:59:41Z
    relations:
      grafana-source:
      - grafana
      juju-info:
      - canonical-livepatch
      - filebeat
      - landscape-client
      - logrotate
      - ntp
      - telegraf-prometheus
      nrpe-external-master:
      - nrpe-host
      target:
      - prometheus-ceph-exporter
      - prometheus-libvirt-exporter
      - prometheus-openstack-exporter
      - telegraf
      - telegraf-maas-infra
      - telegraf-prometheus

$ juju config keystone openstack-origin
cloud:bionic-ussuri

As a workaround, restarting the prometheus snap worked:

$ sudo service snap.prometheus.prometheus restart

[Garrett Neugent (thogarre)]

Note, here were the commands used to deploy and relate the charm:

juju deploy prometheus-libvirt-exporter
juju relate prometheus-libvirt-exporter nova-compute-kvm
juju relate prometheus-libvirt-exporter:scrape prometheus:target
juju relate prometheus-libvirt-exporter:dashboards grafana:dashboards
juju relate prometheus-libvirt-exporter:nrpe-external-master nrpe-host:nrpe-external-master

[Stephan Pampel (stephanpampel)]

In general Prometheus does not need to restart to reload its' config: "Prometheus can reload its configuration at runtime." https://prometheus.io/docs/prometheus/latest/configuration/configuration/

There is already an bug for this (if you are using prometheus2 charm): https://bugs.launchpad.net/charm-prometheus2/+bug/1940565