Memory usage grows beyond configured maximums

[Ken Zalewski]

I have Percona Server 5.7.19 configured with a 2GB buffer pool on a VM with 8GB. Table sizes (heap and tmp) are both 256MB, and the key buffer is 64M. All of the per-connection parameters (join_buffer_size, sort_buffer_size, thread_stack, etc) have low values. Both mysqltuner.pl and tuning-primer.sh indicate that the max memory used by MySQL should be 3GB. That's 2.3GB of global memory, plus 12.6M per connection with 50 connections = 600MB.

However, the mysqld process consistently exceeds the configured maximums, especially when running our nightly backups (using mysqldump). If I do not want to wait for the nightly backups, I can simply run a bunch of mysqldump processes during the day. Eventually, memory hits the 8GB limit, and the kernel OOM killer kills the process.

Even as I type this bug report, I have watched a sequence of mysqldump commands eat up memory, with RSS at 7.4GB and VSS at 9.2GB. In a short while, all available memory will be gone. I will not be able to run any new processes on the server, and eventually, the kernel OOM killer will terminate mysqld.

I have read multiple forum posts about this issue, and I am certain that it is a bug in Percona Server. There is a memory leak somewhere, and it really needs to be fixed. The server needs to respect the configured maximums for buffer sizes, and it must free up memory that is no longer being used.

When the kernel OOM killer terminates the process, it is not a clean termination, which results in a slow restart as table recovery must be run.

I'd really like to put an end to this daily ordeal.

[Sveta Smirnova (svetasmirnova)]

Thank you for the report.

When you configure memory buffers for MySQL/Percona Servers you only limit their sizes. But mysqld can allocate memory for other needs as well. High memory usage by itself does not mean you hit a bug. Also, if it is freed after job is done, it indicates what memory used properly. MySQL/Percona Servers do not have higher memory limits and any tool can tell only lower limit estimates, not higher.

If you want to investigate further where your memory gone use version 5.7, database sys and view memory_by_thread_by_current_bytes. It will show you how memory is used. You must enable memory instrumentation prior selecting from it.

If you see high memory usage with mysqldump you may try running FLUSH TABLES after job is done.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3759