Prometheus Alertmanager Charm

alert not clearing in PagerDuty

Bug #1861740 reported by Joe Guo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Prometheus Alertmanager Charm
Invalid
Undecided
Unassigned

Bug Description

We got alert from PD:

[FIRING:3] BBProbeFailure (LG_Schiphol_ICMP_Probes prometheus-monitor bootstack-monitoring-2)

However, after checking Alert Manager for this cloud, there is no alert there. It seems like the alert disappeared, but didn't clear in PagerDuty.

The problem is gone and I don't know how to re-produce it for now. This bug is a record for future reference.

Revision history for this message
Joe Guo (guoqiao) wrote :

Just after a few minutes, the alert is back, but Alert Manger is still empty. So maybe there is some integration issue. Will update here once I get more info.

Revision history for this message
James Hebden (ec0) wrote :

Multiple Prometheus instances were in use here, and this alert was coming from a different instance, where the alert was indeed still active. With the alert active, resolving the alert in PD should have resulted in a new alert being raised in PD for the still-active Alert Manager entry, so it looks like the behavior was as intended.

Changed in prometheus-alertmanager-charm:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.