gnocchi api get's blocked causing monitoring alerts

[Wouter van Bommel]

In a cloud we are getting near hourly notification that the monitoring is reporting timeout's with the gnocchi api.

This could be cause by some process that the customer is running in the cloud, but not sure that this should cause the api to be blocked for such a long period that the monitoring gets tripped.

The cloud is running gnocchi charm release 23, and reports version 4.2.5 in juju status. If relevant the hacluster is running version 55

The cloud itself is Queens running on Xenial

[Wouter van Bommel (woutervb)]

Subscribed field medium

[James Page (james-page)]

Log files from gnocchi from impacted periods would be useful here - also may require some more in-depth monitoring of what gnocchi is actually doing during when the timeouts occur.

[Wouter van Bommel (woutervb)]

The logging shows lot's of lines like the ones below in the gnocchi-metricd.log.

This might be relevant, the memcached unit is running on a host that also has designate-bind installed, could be related.

2019-10-16 06:25:04,857 [2669] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-48-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:04,857 [2669] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-49-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:04,977 [2681] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-7-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:04,977 [2681] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-8-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:05,237 [2690] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-5-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:05,369 [2702] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-120-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:05,609 [2675] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-27-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:05,617 [2693] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-54-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:05,825 [2687] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-11-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:05,825 [2687] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-12-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:06,325 [2705] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-0-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:07,377 [2669] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-48-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:07,377 [2669] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-49-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:07,497 [2681] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-7-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:07,497 [2681] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-8-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:07,757 [2690] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-5-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:07,889 [2702] WARNING tooz.drivers.memcached: Unable to heartbeat by updating key 'b'__TOOZ_LOCK_gnocchi-sack-120-lock'' with extended expiry of 5 seconds
2019-10-16 06:25:08,137 [2693] WARNING tooz.drivers.memcached: Unable to he...

[Wouter van Bommel (woutervb)]

Hi,

If I can get some clue's on how to get a listing what gnocchi is doing during those peak moments, I am happy to add them to the ticket.

[Alex Kavanagh (ajkavanagh)]

I still think this bug is incomplete; there's not enough information to understand what is going on. It's not clear why the tooz memcached driver is throwing errors. Is the host IO or CPU bound? Memory issues? Networking?

[Launchpad Janitor (janitor)]

[Expired for Gnocchi Charm because there has been no activity for 60 days.]