Comment 1 for bug 1848049

Revision history for this message
Steven Parker (sbparke) wrote :

We found the following on a production cloud.
This causes gnocchi to alarm on our cloud and it seems on similar clouds that have this charm deployed.

The last dataset we collected was from ~20:30-11:30. On 21:55:48, all designate servers got MySQL errors. From the MySQL slow query log file, its possible to see that from 21:04 to 21:16 MySQL received 40 queries from gnocchi, each query [1]doing joins and returning ~500MB each and taking as much as 16min to finish, in a total of 20GB of data. The next thing logged on this file is 21:54 with a lot of slow
queries from other services specially designate. Also around this time, MySQL logs several warning messages saying "InnoDB: Warning: difficult to find free blocks in the buffer pool (338 search iterations)!". So, I believe this is swamping the IOs from the disk, consuming almost all CPUs from the server and causing other queries to timeout.