Where's alertmanager / breaking change

Bug #1942926 reported by Steven R. Loomis
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
prometheus-alertmanager-snap
Undecided
Unassigned
prometheus-snap
Undecided
Unassigned

Bug Description

Is there an official alertmanager snap?

Our installation was recently broken (and missed alerts from a critical outage this weekend) because of the upgrade to 2.28.1 from 2.20.1 which is incompatible (v2 API?) with https://snapcraft.io/prometheus-alertmanager-is

Symptom is like https://discuss.istio.io/t/issue-with-http-communication-from-prometheus-to-alertmanager/10796

"level=error ts=2021-05-31T07:16:45.210Z caller=notifier.go:527 component=notifier alertmanager=http://100.96.9.151:9093/api/v2/alerts count=1 msg="Error sending alert" err="bad response status 404 Not Found"

Revision history for this message
Steven R. Loomis (srl-icu-project) wrote :

prometheus-alertmanager-is only has v0.21.0

Revision history for this message
Haw Loeung (hloeung) wrote :

FWIW, we're using prometheus-alertmanager v0.21.0 together with prometheus v2.28.1 without any issues (that I know of).

Revision history for this message
Paul Collins (pjdc) wrote :

We have the following installed and they work together:

prometheus 2.28.1 53 2/candidate canonical* -
prometheus-alertmanager 0.21.0 11 latest/stable jldev -

Looks like the current release is 0.23.0, so it's not that far behind.

I'm not familiar with how they are wired up, but this might be the relevant part of prometheus.yml:

```
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - [REDACTED]:9093
```

Also, the 404 URL you list above is working here:

$ curl -sv http://127.0.0.1:9093/api/v2/alerts >/dev/null
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9093 (#0)
> GET /api/v2/alerts HTTP/1.1
> Host: 127.0.0.1:9093
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: Origin
< Date: Fri, 17 Sep 2021 04:20:37 GMT
< Transfer-Encoding: chunked
<
{ [16254 bytes data]
* Connection #0 to host 127.0.0.1 left intact
$ _

Revision history for this message
Steven R. Loomis (srl-icu-project) wrote : Re: [Bug 1942926] Re: Where's alertmanager / breaking change

I’ll try “ prometheus-alertmanager”

El El jue, sep. 16, 2021 a la(s) 11:30 p. m., Paul Collins <
<email address hidden>> escribió:

> We have the following installed and they work together:
>
> prometheus 2.28.1 53 2/candidate
> canonical* -
> prometheus-alertmanager 0.21.0 11 latest/stable jldev
> -
>
>
> Looks like the current release is 0.23.0, so it's not that far behind.
>
> I'm not familiar with how they are wired up, but this might be the
> relevant part of prometheus.yml:
>
> ```
> alerting:
> alertmanagers:
> - static_configs:
> - targets:
> - [REDACTED]:9093
> ```
>
> Also, the 404 URL you list above is working here:
>
> $ curl -sv http://127.0.0.1:9093/api/v2/alerts >/dev/null
> * Trying 127.0.0.1...
> * Connected to 127.0.0.1 (127.0.0.1) port 9093 (#0)
> > GET /api/v2/alerts HTTP/1.1
> > Host: 127.0.0.1:9093
> > User-Agent: curl/7.47.0
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Content-Type: application/json
> < Vary: Origin
> < Date: Fri, 17 Sep 2021 04:20:37 GMT
> < Transfer-Encoding: chunked
> <
> { [16254 bytes data]
> * Connection #0 to host 127.0.0.1 left intact
> $ _
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1942926
>
> Title:
> Where's alertmanager / breaking change
>
> Status in prometheus-alertmanager-snap:
> New
> Status in prometheus-snap:
> New
>
> Bug description:
> Is there an official alertmanager snap?
>
> Our installation was recently broken (and missed alerts from a
> critical outage this weekend) because of the upgrade to 2.28.1 from
> 2.20.1 which is incompatible (v2 API?) with
> https://snapcraft.io/prometheus-alertmanager-is
>
> Symptom is like https://discuss.istio.io/t/issue-with-http-
> communication-from-prometheus-to-alertmanager/10796
>
> "level=error ts=2021-05-31T07:16:45.210Z caller=notifier.go:527
> component=notifier alertmanager=http://100.96.9.151:9093/api/v2/alerts
> count=1 msg="Error sending alert" err="bad response status 404 Not
> Found"
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/prometheus-alertmanager-snap/+bug/1942926/+subscriptions
>
>

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers