pt-query-digest --type tcpdump memory usage keeps increasing
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Percona Toolkit moved to https://jira.percona.com/projects/PT |
Fix Released
|
Medium
|
Daniel Nichter |
Bug Description
I'm streaming tcpdump captures to another host to get a list of queries for later review like so:
tcpdump -i eth2 -Z <username> -s 65535 -x -n -q -tttt 'port 3306 and tcp[0:2] & 15 == 10 and tcp[2:4] & 15 == 10 ' | ssh <username>
pt-query-digest memory use grows steadily until it crashes or makes the server unusable.
I can reproduce this easily by doing a short tcpdump capture and then running these commands:
while :; do cat tcpdump.log ; done | ./pt-query-digest --no-report --type tcpdump --print
If I monitor memory usage by pt-query-digest while this runs, it just keeps growing.
If I do a similar short slow log capture and run the same command but with this capture instead:
while :; do cat slow.log ; done | ./pt-query-digest --no-report --print 2
Memory use remains constant.
Related branches
- Daniel Nichter: Approve
-
Diff: 13 lines (+3/-0)1 file modifiedbin/pt-query-digest (+3/-0)
Changed in percona-toolkit: | |
status: | New → In Progress |
assignee: | nobody → Daniel Nichter (daniel-nichter) |
tags: | added: memory-usage pt-query-digest |
summary: |
- pt-query-digest memory increases steadily when reading from a stream - with --type tcpdump, --no-report and --print + pt-query-digest --type tcpdump memory usage keeps increasing |
Changed in percona-toolkit: | |
importance: | Undecided → Medium |
milestone: | none → 2.1.4 |
Changed in percona-toolkit: | |
status: | In Progress → Fix Committed |
Changed in percona-toolkit: | |
status: | Fix Committed → Fix Released |
I can reproduce this, and I'm pretty sure it's related to MySQLProtocolPa rser's session tracking. Unlike slowlogs which have no sessions, tcpdump logs have sessions which span the lifetime of a client connection, e.g. from handshake to COM_QUIT. That's the only thing that's saved, so that's why I suspect it's causing increasing memory usage. It's been awhile since I've worked with MySQLProtocolParser so I'll have to study the code a bit to remind myself how and why it handles sessions and, relatedly, when it deletes sessions.