Newton - gnocchi-metricd is hammering redis

Bug #1639842 reported by Michele Baldessari on 2016-11-07
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Undecided
Unassigned

Bug Description

Note that this is different than https://bugs.launchpad.net/tripleo/+bug/1621164

I can see that metricd is hammering redis quite a bit and using a lot of CPU.
So we have:
# grep -irE "Connect from.*redis" /var/log/swift/swift.log |wc -l
25966
# grep -irE "Connect from.*redis" /var/log/swift/swift.log | head -n1
Nov 7 14:52:19 localhost haproxy[18671]: Connect from 172.16.2.5:48742 to 172.16.2.5:6379 (redis/TCP)
# grep -irE "Connect from.*redis" /var/log/swift/swift.log | tail -n1
Nov 7 15:42:38 localhost haproxy[24643]: Connect from 172.16.2.5:49824 to 172.16.2.5:6379 (redis/TCP)

So in the timespan of ~50 minutes we have ~26000 requests. Which makes it ~520 requests per minute.
Which turns into ~ *8.5* requests per second.

(Not sure yet why haproxy ends up in swift.log, have not investigated that yet)

Michele Baldessari (michele) wrote :

So this might have been due to https://review.openstack.org/#/c/394387/ not being merged yet and redis was temporarily down. Still investigating

Steven Hardy (shardy) on 2016-11-11
Changed in tripleo:
milestone: ocata-1 → ocata-2
Changed in tripleo:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers