Ceilometer is bombing logs with connection retries when redis is down
Bug #1666163 reported by
Sagi (Sergey) Shnaidman
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ceilometer |
Invalid
|
Undecided
|
Unassigned | ||
tripleo |
Expired
|
Undecided
|
Unassigned |
Bug Description
When redis is down, the ceilometer tries to reconnect and dumps this to logs a few times in a second. This makes its logs huge size and overload our logs servers.
Changed in tripleo: | |
importance: | Undecided → High |
Changed in tripleo: | |
milestone: | none → pike-1 |
status: | New → Triaged |
Changed in tripleo: | |
milestone: | pike-1 → pike-2 |
Changed in tripleo: | |
milestone: | pike-2 → pike-3 |
Changed in tripleo: | |
milestone: | pike-3 → pike-rc1 |
Changed in tripleo: | |
milestone: | pike-rc1 → pike-rc2 |
Changed in tripleo: | |
milestone: | pike-rc2 → queens-1 |
Changed in tripleo: | |
milestone: | queens-1 → queens-2 |
Changed in tripleo: | |
milestone: | queens-2 → queens-3 |
Changed in tripleo: | |
milestone: | queens-3 → queens-rc1 |
Changed in tripleo: | |
milestone: | queens-rc1 → rocky-1 |
Changed in tripleo: | |
milestone: | rocky-1 → rocky-2 |
Changed in tripleo: | |
milestone: | rocky-2 → rocky-3 |
Changed in tripleo: | |
milestone: | rocky-3 → rocky-rc1 |
Changed in tripleo: | |
milestone: | rocky-rc1 → stein-1 |
Changed in tripleo: | |
milestone: | stein-1 → stein-2 |
To post a comment you must log in.
there already exists configurable retry logic: https:/ /github. com/openstack/ ceilometer/ commit/ 3459bc59f2014af d98865f8dfd93a0 197424d518
we're switching to tooz partitioning and that also has configurable retry logic (according to jd). will close if this works (or you can reraise if broken)