Newton - gnocchi-metricd is hammering redis

[Michele Baldessari]

Note that this is different than https://bugs.launchpad.net/tripleo/+bug/1621164

I can see that metricd is hammering redis quite a bit and using a lot of CPU.
So we have:
# grep -irE "Connect from.*redis" /var/log/swift/swift.log |wc -l
25966
# grep -irE "Connect from.*redis" /var/log/swift/swift.log | head -n1
Nov 7 14:52:19 localhost haproxy[18671]: Connect from 172.16.2.5:48742 to 172.16.2.5:6379 (redis/TCP)
# grep -irE "Connect from.*redis" /var/log/swift/swift.log | tail -n1
Nov 7 15:42:38 localhost haproxy[24643]: Connect from 172.16.2.5:49824 to 172.16.2.5:6379 (redis/TCP)

So in the timespan of ~50 minutes we have ~26000 requests. Which makes it ~520 requests per minute.
Which turns into ~ *8.5* requests per second.

(Not sure yet why haproxy ends up in swift.log, have not investigated that yet)

[Michele Baldessari (michele)]

So this might have been due to https://review.openstack.org/#/c/394387/ not being merged yet and redis was temporarily down. Still investigating