broken webhooks inflate launchpad_job queue count
Bug #1886738 reported by
Paul Collins
This bug affects 2 people
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Launchpad itself |
New
|
Undecided
|
Unassigned | ||
Bug Description
Today we noticed the launchpad_job celery queue was stuck at around 200-300 messages.
After some investigation, this was found to be due to retries for a webhook that was returning 503. Said webhook was configured on a few hundred repositories.
It might make sense for Launchpad to somehow serialize dispatch by hostname or otherwise limit the number of in-flight webhook messages for a given destination.
It might also be that we need to figure out a better way to monitor this queue than a simple limit. :)
To post a comment you must log in.
