Out of memory error when running min_free_kbytes testcase on precise-updates on armadaxp

[Michael Reed]

Impact: After running the testcase min_free_kbytes, the system becomes unusable

Build Version and Date: ubuntu@ubuntu:~$ uname -a
Linux ubuntu 3.2.0-1605-armadaxp #8-Ubuntu SMP Sat Jul 7 22:36:46 UTC 2012 armv7l armv7l armv7l GNU/Linux

Steps to reproduce:
1. Download, build and Install LTP testsuite
2. echo 0 > /proc/sys/kernel/hung_task_timeout_secs
3. cd /ltp-full-20120614/testcases/kernel/mem/tunable
4. sudo ./min_free_kbytes

This is the output when I ran this from a script, so I could get the oom* info
sudo ./runtest.sh
[sudo] password for ubuntu:
cat /proc/1409/task/1409/oom*
0
1
0
ubuntu@ubuntu:~/ltp-full-20120614/testcases/kernel/mem/tunable$ min_free_kbytes 0 TINFO : set overcommit_memory to 2
min_free_kbytes 0 TINFO : set min_free_kbytes to 3624
memfree is 2970220 kB before eatup mem
memfree is 81228 kB after eatup mem
min_free_kbytes 0 TINFO : set min_free_kbytes to 1513802
[ 1643.106611] Out of memory: Kill process 264 (upstart-udev-br) score 1 or sacrifice child
[ 1643.114754] Killed process 264 (upstart-udev-br) total-vm:1960kB, anon-rss:0kB, file-rss:0kB
[ 1643.267693] Out of memory: Kill process 424 (upstart-socket-) score 1 or sacrifice child
[ 1643.275814] Killed process 424 (upstart-socket-) total-vm:1892kB, anon-rss:0kB, file-rss:0kB
[ 1643.424805] Out of memory: Kill process 460 (dhclient3) score 1 or sacrifice child
[ 1643.432462] Killed process 460 (dhclient3) total-vm:1884kB, anon-rss:0kB, file-rss:0kB
[ 1643.635470] Out of memory: Kill process 657 (dbus-daemon) score 1 or sacrifice child
[ 1643.643286] Killed process 657 (dbus-daemon) total-vm:2324kB, anon-rss:0kB, file-rss:0kB
[ 1644.085534] Out of memory: Kill process 670 (rsyslogd) score 1 or sacrifice child
[ 1644.093079] Killed process 670 (rsyslogd) total-vm:30204kB, anon-rss:112kB, file-rss:140kB

The output that shows the kernel panic is in the following link:
https://pastebin.canonical.com/71710/

A description and the expected output of the testcase can be found here:
http://<email address hidden>/msg13978.html

The system is n2 and is currently available for debugging

[Michael Reed (mreed8855)]

Output when running the testcase from the command line.

sudo ./min_free_k
min_free_kbytes 0 TINFO : set overcommit_memory to 2
min_free_kbytes 0 TINFO : set min_free_kbytes to 3624
memfree is 2976072 kB before eatup mem
memfree is 80376 kB after eatup mem
min_free_kbytes 0 TINFO : set min_free_kbytes to 1513804
[ 386.395436] Out of memory: Kill process 266 (upstart-udev-br) score 1 or sacrifice child
[ 386.403632] Killed process 266 (upstart-udev-br) total-vm:1960kB, anon-rss:0kB, file-rss:0kB
[ 386.555048] Out of memory: Kill process 424 (upstart-socket-) score 1 or sacrifice child
[ 386.563195] Killed process 424 (upstart-socket-) total-vm:1892kB, anon-rss:0kB, file-rss:0kB
[ 386.724271] Out of memory: Kill process 469 (dhclient3) score 1 or sacrifice child
[ 386.731900] Killed process 469 (dhclient3) total-vm:1884kB, anon-rss:0kB, file-rss:0kB
[ 387.914502] Out of memory: Kill process 669 (dbus-daemon) score 1 or sacrifice child
[ 387.922307] Killed process 669 (dbus-daemon) total-vm:2324kB, anon-rss:0kB, file-rss:0kB
[ 388.913183] Out of memory: Kill process 677 (rsyslogd) score 1 or sacrifice child
[ 388.920719] Killed process 677 (rsyslogd) total-vm:30204kB, anon-rss:40kB, file-rss:100kB
[ 388.984618] Out of memory: Kill process 690 (rs:main Q:Reg) score 1 or sacrifice child
[ 388.992595] Killed process 690 (rs:main Q:Reg) total-vm:30204kB, anon-rss:240kB, file-rss:160kB
[ 389.332725] Out of memory: Kill process 729 (getty) score 1 or sacrifice child
[ 389.340002] Killed process 729 (getty) total-vm:3836kB, anon-rss:0kB, file-rss:8kB
[ 389.912061] Out of memory: Kill process 738 (getty) score 1 or sacrifice child
[ 389.919332] Killed process 738 (getty) total-vm:3836kB, anon-rss:0kB, file-rss:8kB
[ 390.862432] Out of memory: Kill process 748 (getty) score 1 or sacrifice child
[ 390.869715] Killed process 748 (getty) total-vm:3836kB, anon-rss:0kB, file-rss:8kB
[ 391.033211] Out of memory: Kill process 751 (getty) score 1 or sacrifice child
[ 391.040482] Killed process 751 (getty) total-vm:3836kB, anon-rss:0kB, file-rss:8kB
[ 391.156255] Out of memory: Kill process 756 (getty) score 1 or sacrifice child
[ 391.163504] Killed process 756 (getty) total-vm:3836kB, anon-rss:0kB, file-rss:8kB
[ 391.276823] Out of memory: Kill process 768 (cron) score 1 or sacrifice child
[ 391.283986] Killed process 768 (cron) total-vm:1836kB, anon-rss:0kB, file-rss:0kB
[ 391.378061] Out of memory: Kill process 769 (atd) score 1 or sacrifice child
[ 391.385146] Killed process 769 (atd) total-vm:1708kB, anon-rss:0kB, file-rss:0kB
[ 391.496240] Out of memory: Kill process 772 (acpid) score 1 or sacrifice child
[ 391.503489] Killed process 772 (acpid) total-vm:1328kB, anon-rss:0kB, file-rss:4kB
[ 392.063055] Out of memory: Kill process 774 (irqbalance) score 1 or sacrifice child
[ 392.070760] Killed process 774 (irqbalance) total-vm:2600kB, anon-rss:52kB, file-rss:172kB
[ 392.132300] Out of memory: Kill process 781 (whoopsie) score 1 or sacrifice child
[ 392.139831] Killed process 781 (whoopsie) total-vm:20500kB, anon-rss:252kB, file-rss:680kB
[ 392.901843] Out of memory: Kill process 785 (getty) score 1 or sac...

[Jani Monoses (jani)]

This occurs in the released version as well not only in updates.
Tested with 3.2.0-1602.5

[Michael Reed (mreed8855)]

This error also occurs on highbank on precise, this is probably a kernel bug, and not architecture specific. I will test it on x86 to verify.

ubuntu@ubuntu:~$ uname -a
Linux ubuntu 3.5.0-9-highbank #9-Ubuntu SMP PREEMPT Thu Aug 9 17:12:46 UTC 2012 armv7l armv7l armv7l GNU/Linux

[Michael Reed (mreed8855)]

This error is also seen on an x86 system on precise.

       [ 6239.736670] Out of memory: Kill process 867 (xinetd) score 1 or sacrifice chi
       ld
       [ 6239.736746] Killed process 867 (xinetd) total-vm:2544kB, anon-rss:156kB, file
       -rss:768kB
       [ 6239.740701] Out of memory: Kill process 868 (atd) score 1 or sacrifice child
       [ 6239.740770] Killed process 868 (atd) total-vm:2452kB, anon-rss:128kB, file-rs
       s:224kB
       [ 6239.744692] Out of memory: Kill process 869 (cron) score 1 or sacrifice child
       [ 6239.744760] Killed process 869 (cron) total-vm:2600kB, anon-rss:148kB, file-r
       ss:600kB
       [ 6239.748720] Out of memory: Kill process 871 (acpid) score 1 or sacrifice chil
       d
       [ 6239.748794] Killed process 871 (acpid) total-vm:2156kB, anon-rss:96kB, file-r
       ss:500kB
       [ 6239.752530] Out of memory: Kill process 883 (whoopsie) score 1 or sacrifice c
       hild
       [ 6239.752606] Killed process 883 (whoopsie) total-vm:24448kB, anon-rss:840kB, f
       ile-rss:2716kB
       [ 6360.804206] INFO: task init:1 blocked for more than 120 seconds.
       [ 6360.804272] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
       message.
       [ 6360.805565] INFO: task whoopsie:883 blocked for more than 120 seconds.
       [ 6360.805611] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
       message.

[Gavin Guo (mimi0213kimo)]

Having tested on n1 and the kernel oops not happened. Don't know if the problem is the hardware issue and will try to test on n2 if it's available.

[Michael Reed (mreed8855)]

This bug is actually not architecture specific, it happens on x86 and x86_64. It should probably move over to the kernel team