commit 97e5c0e9b1906f2993b4c12820ac3cb9ddcfe821
Author: Mark Goddard <email address hidden>
Date: Fri Jan 5 11:02:39 2024 +0000
cadvisor: Set housekeeping interval to Prometheus scrape interval
The prometheus_cadvisor container has high CPU usage. On various
production systems I checked it sits around 13-16% on controllers,
averaged over the prometheus 1m scrape interval. When viewed with top we
can see it is a bit spikey and can jump over 100%.
There are various bugs about this, but I found https://github.com/google/cadvisor/issues/2523 which suggests reducing
the per-container housekeeping interval. This defaults to 1s, which
provides far greater granularity than we need with the default
prometheus scrape interval of 60s.
Reducing the housekeeping interval to 60s on a production controller
reduced the CPU usage from 13% to 3.5% average. This still seems high,
but is more reasonable.
Reviewed: https:/ /review. opendev. org/c/openstack /kolla- ansible/ +/904823 /opendev. org/openstack/ kolla-ansible/ commit/ 97e5c0e9b1906f2 993b4c12820ac3c b9ddcfe821
Committed: https:/
Submitter: "Zuul (22348)"
Branch: master
commit 97e5c0e9b1906f2 993b4c12820ac3c b9ddcfe821
Author: Mark Goddard <email address hidden>
Date: Fri Jan 5 11:02:39 2024 +0000
cadvisor: Set housekeeping interval to Prometheus scrape interval
The prometheus_cadvisor container has high CPU usage. On various
production systems I checked it sits around 13-16% on controllers,
averaged over the prometheus 1m scrape interval. When viewed with top we
can see it is a bit spikey and can jump over 100%.
There are various bugs about this, but I found /github. com/google/ cadvisor/ issues/ 2523 which suggests reducing
https:/
the per-container housekeeping interval. This defaults to 1s, which
provides far greater granularity than we need with the default
prometheus scrape interval of 60s.
Reducing the housekeeping interval to 60s on a production controller
reduced the CPU usage from 13% to 3.5% average. This still seems high,
but is more reasonable.
Change-Id: I89c62a45b1f358 aafadcc0317ce88 2f4609543e7
Closes-Bug: #2048223