playback crashes on LInux with 'Out of memory'

[ska]

[In:Percona Playback Documentation]
Hi,

I am currently trying to run Playback tool but every time it gets killed by OOM due to memory exhaustion. I have tried the version, which comes with Debian Squeeze(0.6-179.squeeze) and the source code:
http://www.percona.com/downloads/Percona-Playback/Percona-Playback-0.6/

The result is the same. Moreover, if I run Playback on the server itself, sometimes MariaDB gets killed by OOM which is pretty bad.

OS details
Linux wkwvdb-fra2 2.6.32-5-amd64 #1 SMP Fri May 10 08:43:19 UTC 2013 x86_64 GNU/Linux

and I was testing it against MariaDB 5.5:
whatever@my_hidden_ip:~$ dpkg -l | grep mari
ii libmariadbclient18 5.5.31+maria-1~squeeze MariaDB database client library
ii libmysqlclient18 5.5.31+maria-1~squeeze Virtual package to satisfy external depends
ii mariadb-client-5.5 5.5.31+maria-1~squeeze MariaDB database client binaries
ii mariadb-client-core-5.5 5.5.31+maria-1~squeeze MariaDB database core client binaries
ii mariadb-common 5.5.31+maria-1~squeeze MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf)
ii mariadb-server 5.5.31+maria-1~squeeze MariaDB database server (metapackage depending on the latest version)
ii mariadb-server-5.5 5.5.31+maria-1~squeeze MariaDB database server binaries
ii mariadb-server-core-5.5 5.5.31+maria-1~squeeze MariaDB database core server files

I used the following line to start playback:

./percona-playback --input-plugin tcpdump --tcpdump-file /data/db1.tcpdump --tcpdump-mode accurate --db-plugin libmysqlclient --mysql-host my_hidden_ip --mysql-username root --mysql-schema imdb --mysql-port 3306

The syslog has the following:

Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.002307] lt-percona-play invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.005245] lt-percona-play cpuset=/ mems_allowed=0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.007048] Pid: 5047, comm: lt-percona-play Not tainted 2.6.32-5-amd64 #1
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.009311] Call Trace:
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.010304] [<ffffffff810b6708>] ? oom_kill_process+0x7f/0x23f
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.012217] [<ffffffff810b6c2c>] ? __out_of_memory+0x12a/0x141
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.014165] [<ffffffff810b6d83>] ? out_of_memory+0x140/0x172
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.016026] [<ffffffff810baae8>] ? __alloc_pages_nodemask+0x4ec/0x5fc
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.018278] [<ffffffff810cd034>] ? handle_mm_fault+0x27a/0x80f
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.020411] [<ffffffff810d2372>] ? do_brk+0x227/0x307
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.022230] [<ffffffff81300186>] ? do_page_fault+0x2e0/0x2fc
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.024201] [<ffffffff812fe025>] ? page_fault+0x25/0x30
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.026074] Mem-Info:
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.027076] Node 0 DMA per-cpu:
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.028456] CPU 0: hi: 0, btch: 1 usd: 0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.030230] Node 0 DMA32 per-cpu:
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.031598] CPU 0: hi: 186, btch: 31 usd: 30
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.033402] active_anon:124262 inactive_anon:124371 isolated_anon:0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.033403] active_file:69 inactive_file:174 isolated_file:0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.033403] unevictable:0 dirty:1 writeback:1 unstable:0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.033404] free:1998 slab_reclaimable:634 slab_unreclaimable:1087
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.033405] mapped:61 shmem:0 pagetables:1814 bounce:0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.043148] Node 0 DMA free:4036kB min:60kB low:72kB high:88kB active_anon:5744kB inactive_anon:5988kB active_file:16kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15368kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:56kB slab_unreclaimable:32kB kernel_stack:8kB pagetables:48kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.055187] lowmem_reserve[]: 0 994 994 994
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.057034] Node 0 DMA32 free:3956kB min:4000kB low:5000kB high:6000kB active_anon:491304kB inactive_anon:491496kB active_file:260kB inactive_file:696kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1018060kB mlocked:0kB dirty:4kB writeback:4kB mapped:236kB shmem:0kB slab_reclaimable:2480kB slab_unreclaimable:4316kB kernel_stack:528kB pagetables:7208kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.069915] lowmem_reserve[]: 0 0 0 0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.071390] Node 0 DMA: 1*4kB 6*8kB 9*16kB 10*32kB 7*64kB 4*128kB 2*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4036kB
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.075570] Node 0 DMA32: 477*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3964kB
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.107716] 2501 total pagecache pages
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.109426] 2243 pages in swap cache
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.110934] Swap cache stats: add 2608278, delete 2606035, find 60756/96530
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.113557] Free swap = 0kB
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.114871] Total swap = 1044216kB
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.120817] 262139 pages RAM
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.122226] 5393 pages reserved
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.123611] 125 pages shared
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.124970] 254553 pages non-shared
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.126417] Out of memory: kill process 5047 (lt-percona-play) score 42951 or a child
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.129494] Killed process 5047 (lt-percona-play)

I also ran playback with strace. It sends queries to server but seems not to properly free the memory.

Do I do something wrong or is it a bug?

Have a nice weekend,
Sergey

[ska (kasmanaft)]

and the dump has been created using

tcpdump -i any port 3306 -w example.dump

as explained in http://www.percona.com/doc/percona-playback/plugins/tcpdump/index.html