pt-query-digest is very slow when processing long lines

[monty solomon]

pt-query-digest was taking a long time to run and could not be killed using the TERM signal.

I determined that the cause of the issue is that pt-query-digest doesn't handle long lines well.

Some of the lines in the slow query log file are longer than 20 MB.

Please add an option to pt-query-digest to ignore lines over a specified length.

[Sveta Smirnova (svetasmirnova)]

Thank you for the reasonable feature request: "add an option to pt-query-digest to ignore lines over a specified length"

[monty solomon (monty+launchpad)]

While processing very long queries in the slow log files, pt-query-digest consumed most of the memory on the server and caused OOM for various processes.

Here are some sample header entries from the output of pt-query-digest

# 324.4s user time, 2.4s system time, 5.27G rss, 5.44G vsz
# 260.5s user time, 2s system time, 1.96G rss, 2.13G vsz
# 297.6s user time, 2.3s system time, 4.80G rss, 4.97G vsz
# 13.7s user time, 160ms system time, 351.22M rss, 528.03M vsz
# 160ms user time, 10ms system time, 28.69M rss, 205.57M vsz
# 199.6s user time, 2.1s system time, 4.61G rss, 4.78G vsz
# 506.7s user time, 3.2s system time, 6.92G rss, 7.09G vsz
# 288.7s user time, 2.1s system time, 4.39G rss, 4.56G vsz
# 340s user time, 2.9s system time, 6.26G rss, 6.43G vsz
# 257.6s user time, 1.8s system time, 3.77G rss, 3.95G vsz
# 461.7s user time, 3.9s system time, 8.06G rss, 8.23G vsz
# 514.6s user time, 4.8s system time, 9.98G rss, 10.15G vsz
# 384.5s user time, 2.8s system time, 6.01G rss, 6.18G vsz
# 565.5s user time, 3.6s system time, 7.17G rss, 7.34G vsz
# 1188.2s user time, 7s system time, 15.88G rss, 16.05G vsz
# 1.2s user time, 20ms system time, 30.92M rss, 207.75M vsz

[monty solomon (monty+launchpad)]

Here is an example where pt-query-digest was killed due to OOM

Out of memory: Kill process 32635 (perl) score 885 or sacrifice child
Killed process 32635 (perl) total-vm:14709848kB, anon-rss:14521664kB, file-rss:44kB

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PT-721