Newly added targets don't show up without restarting prometheus

Bug #1891942 reported by Drew Freiberger
34
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Prometheus2 charm
Fix Released
Critical
Unassigned

Bug Description

This merge added code to use the web hook to reload the configuration file via web hook /-/reload, but the daemon argument --web.enable-lifecycle is not enabled in the charm or default daemon_args config to allow for this.

Ultimately, newly related targets don't cause config file reloads, so data doesn't start flowing into prometheus without the operator restarting the service or running kill -HUP on the service.

Recommend adding daemon arg --web.enable-lifecycle to charm code.

https://code.launchpad.net/~afreiberger/charm-prometheus2/+git/charm-prometheus2/+merge/387967

The runtime_args code doesn't seem like it would support value-less binary runtime args, however, so this may require a slight refactor.

Tags: sts

Related branches

Revision history for this message
Drew Freiberger (afreiberger) wrote :

Suggest functional test that adds a new "target" relation and then checks that the new target exists in running prometheus via api endpoint /targets.

Changed in charm-prometheus2:
importance: Undecided → Critical
Edin S (exsdev)
Changed in charm-prometheus2:
assignee: nobody → Edin S (exsdev)
Revision history for this message
Liam Young (gnuoy) wrote :

Pretty sure we are hitting this in our ceph-mon tests

Revision history for this message
Liam Young (gnuoy) wrote :

Confirmed. Reverting our test to use cs:prometheus2-18 fixes the missing ceph data issue

Jose Guedez (jfguedez)
Changed in charm-prometheus2:
status: New → In Progress
Andrea Ieri (aieri)
Changed in charm-prometheus2:
assignee: Edin S (exsdev) → Andrea Ieri (aieri)
status: In Progress → Fix Committed
Revision history for this message
Andrea Ieri (aieri) wrote :

available as cs:~llama-charmers-next/prometheus2-6

Changed in charm-prometheus2:
assignee: Andrea Ieri (aieri) → nobody
Revision history for this message
David Coronel (davecore) wrote :

subscribed ~field-medium for awareness

Felipe Reyes (freyes)
tags: added: sts
Jeremy Lounder (jldev)
Changed in charm-prometheus2:
status: Fix Committed → Fix Released
milestone: none → 20.10
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.