Ceilometer is bombing logs with connection retries when redis is down

Bug #1666163 reported by Sagi (Sergey) Shnaidman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Invalid
Undecided
Unassigned
tripleo
Expired
Undecided
Unassigned

Bug Description

When redis is down, the ceilometer tries to reconnect and dumps this to logs a few times in a second. This makes its logs huge size and overload our logs servers.

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/56232ef/logs/overcloud-controller-0/var/log/ceilometer/central.txt.gz

Changed in tripleo:
importance: Undecided → High
Changed in tripleo:
milestone: none → pike-1
status: New → Triaged
Revision history for this message
gordon chung (chungg) wrote :

there already exists configurable retry logic: https://github.com/openstack/ceilometer/commit/3459bc59f2014afd98865f8dfd93a0197424d518

we're switching to tooz partitioning and that also has configurable retry logic (according to jd). will close if this works (or you can reraise if broken)

Changed in ceilometer:
status: New → Incomplete
Changed in tripleo:
milestone: pike-1 → pike-2
Changed in tripleo:
milestone: pike-2 → pike-3
Revision history for this message
gordon chung (chungg) wrote :

we use tooz now completely for coordination[1]. please use those options and target tooz if it don't work for you.

https://github.com/openstack/ceilometer/commit/27604abd461d7dbf8098c7cc794dfcc2686c4527

Changed in ceilometer:
status: Incomplete → Invalid
Changed in tripleo:
milestone: pike-3 → pike-rc1
Changed in tripleo:
milestone: pike-rc1 → pike-rc2
Changed in tripleo:
milestone: pike-rc2 → queens-1
Changed in tripleo:
milestone: queens-1 → queens-2
Changed in tripleo:
milestone: queens-2 → queens-3
Changed in tripleo:
milestone: queens-3 → queens-rc1
Changed in tripleo:
milestone: queens-rc1 → rocky-1
Changed in tripleo:
milestone: rocky-1 → rocky-2
Changed in tripleo:
milestone: rocky-2 → rocky-3
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Changed in tripleo:
milestone: stein-1 → stein-2
Revision history for this message
Emilien Macchi (emilienm) wrote : Cleanup EOL bug report

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (FUTURE, PIKE, QUEENS, ROCKY, STEIN).
  Valid example: CONFIRMED FOR: FUTURE

Changed in tripleo:
importance: High → Undecided
status: Triaged → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.