crash after kswapd page allocation failure

[Bernhard Seibold]

This seems to be the same issue as debbugs #576838
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=576838

[770660.316022] __ratelimit: 80 callbacks suppressed
[770660.316027] kswapd0: page allocation failure. order:0, mode:0x20
[770660.316031] Pid: 36, comm: kswapd0 Not tainted 2.6.32-25-server #44-Ubuntu
[770660.316034] Call Trace:
[770660.316036] <IRQ> [<ffffffff810f9a3e>] __alloc_pages_slowpath+0x56e/0x580
[770660.316052] [<ffffffff810f9bc1>] __alloc_pages_nodemask+0x171/0x180
[770660.316057] [<ffffffff8112cb07>] alloc_pages_current+0x87/0xd0
[770660.316063] [<ffffffff813daa72>] try_fill_recv+0x182/0x200
[770660.316067] [<ffffffff813dac9d>] virtnet_poll+0x10d/0x160
[770660.316073] [<ffffffff81472fcf>] net_rx_action+0x10f/0x250
[770660.316078] [<ffffffff8106d487>] __do_softirq+0xb7/0x1e0
[770660.316082] [<ffffffff810c4040>] ? handle_IRQ_event+0x60/0x170
[770660.316087] [<ffffffff810132ec>] call_softirq+0x1c/0x30
[770660.316090] [<ffffffff81014cb5>] do_softirq+0x65/0xa0
[770660.316093] [<ffffffff8106d325>] irq_exit+0x85/0x90
[770660.316098] [<ffffffff8155f235>] do_IRQ+0x75/0xf0
[770660.316101] [<ffffffff81012b13>] ret_from_intr+0x0/0x11
[770660.316103] <EOI> [<ffffffff810ffea8>] ? __remove_mapping+0x88/0x110
[770660.316110] [<ffffffff8110151d>] ? shrink_page_list+0x2ad/0x600
[770660.316114] [<ffffffff81101b7e>] ? shrink_inactive_list+0x30e/0x810
[770660.316124] [<ffffffff81156e68>] ? d_kill+0x68/0x80
[770660.316128] [<ffffffff81156edc>] ? prune_one_dentry+0x5c/0x110
[770660.316131] [<ffffffff811571d4>] ? __shrink_dcache_sb+0x244/0x2c0
[770660.316135] [<ffffffff810fbd6a>] ? determine_dirtyable_memory+0x1a/0x30
[770660.316139] [<ffffffff810fbe17>] ? get_dirty_limits+0x27/0x2f0
[770660.316142] [<ffffffff81102111>] ? shrink_list+0x91/0xf0
[770660.316145] [<ffffffff81102307>] ? shrink_zone+0x197/0x240
[770660.316148] [<ffffffff81103399>] ? balance_pgdat+0x659/0x6d0
[770660.316152] [<ffffffff811003c0>] ? isolate_pages_global+0x0/0x50
[770660.316155] [<ffffffff8110350e>] ? kswapd+0xfe/0x150
[770660.316159] [<ffffffff81084250>] ? autoremove_wake_function+0x0/0x40
[770660.316162] [<ffffffff81103410>] ? kswapd+0x0/0x150
[770660.316165] [<ffffffff81083ed6>] ? kthread+0x96/0xa0
[770660.316169] [<ffffffff810131ea>] ? child_rip+0xa/0x20
[770660.316172] [<ffffffff81083e40>] ? kthread+0x0/0xa0
[770660.316175] [<ffffffff810131e0>] ? child_rip+0x0/0x20
[770660.316176] Mem-Info:
[770660.316178] Node 0 DMA per-cpu:
[770660.316181] CPU 0: hi: 0, btch: 1 usd: 0
[770660.316183] CPU 1: hi: 0, btch: 1 usd: 0
[770660.316185] Node 0 DMA32 per-cpu:
[770660.316188] CPU 0: hi: 186, btch: 31 usd: 162
[770660.316190] CPU 1: hi: 186, btch: 31 usd: 156
[770660.316195] active_anon:2751 inactive_anon:11810 isolated_anon:0
[770660.316196] active_file:33607 inactive_file:182301 isolated_file:35
[770660.316197] unevictable:0 dirty:21611 writeback:500 unstable:0
[770660.316198] free:1369 slab_reclaimable:11657 slab_unreclaimable:2756
[770660.316205] mapped:1927 shmem:64 pagetables:518 bounce:0
[770660.316208] Node 0 DMA free:3988kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:132kB inactive_file:11080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:1564kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:192kB slab_unreclaimable:152kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[770660.316218] lowmem_reserve[]: 0 994 994 994
[770660.316222] Node 0 DMA32 free:1488kB min:4000kB low:5000kB high:6000kB active_anon:11004kB inactive_anon:47240kB active_file:134296kB inactive_file:718124kB unevictable:0kB isolated(anon):0kB isolated(file):140kB present:1018060kB mlocked:0kB dirty:84880kB writeback:2000kB mapped:7708kB shmem:256kB slab_reclaimable:46436kB slab_unreclaimable:10872kB kernel_stack:1504kB pagetables:2072kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:99 all_unreclaimable? no
[770660.316233] lowmem_reserve[]: 0 0 0 0
[770660.316236] Node 0 DMA: 1*4kB 2*8kB 2*16kB 1*32kB 11*64kB 1*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3988kB
[770660.316247] Node 0 DMA32: 94*4kB 5*8kB 1*16kB 1*32kB 6*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1488kB
[770660.316257] 216001 total pagecache pages
[770660.316258] 0 pages in swap cache
[770660.316261] Swap cache stats: add 0, delete 0, find 0/0
[770660.316262] Free swap = 0kB
[770660.316264] Total swap = 0kB
[770660.320351] 262139 pages RAM
[770660.320353] 6263 pages reserved
[770660.320355] 195097 pages shared
[770660.320356] 62534 pages non-shared
[770660.320455] kswapd0: page allocation failure. order:0, mode:0x20
[770660.320459] Pid: 36, comm: kswapd0 Not tainted 2.6.32-25-server #44-Ubuntu
[770660.320462] Call Trace:
[770660.320465] <IRQ> [<ffffffff810f9a3e>] __alloc_pages_slowpath+0x56e/0x580
[770660.320476] [<ffffffff810f9bc1>] __alloc_pages_nodemask+0x171/0x180
[770660.320481] [<ffffffff8112cb07>] alloc_pages_current+0x87/0xd0
[770660.320485] [<ffffffff813daa72>] try_fill_recv+0x182/0x200
[770660.320489] [<ffffffff813dac9d>] virtnet_poll+0x10d/0x160
[770660.320494] [<ffffffff81472fcf>] net_rx_action+0x10f/0x250
[770660.320498] [<ffffffff8106d487>] __do_softirq+0xb7/0x1e0
[770660.320502] [<ffffffff810c4040>] ? handle_IRQ_event+0x60/0x170
[770660.320506] [<ffffffff810132ec>] call_softirq+0x1c/0x30
[770660.320509] [<ffffffff81014cb5>] do_softirq+0x65/0xa0
[770660.320512] [<ffffffff8106d325>] irq_exit+0x85/0x90
[770660.320516] [<ffffffff8155f235>] do_IRQ+0x75/0xf0
[770660.320519] [<ffffffff81012b13>] ret_from_intr+0x0/0x11
[770660.320521] <EOI> [<ffffffff810ffea8>] ? __remove_mapping+0x88/0x110
[770660.320528] [<ffffffff8110151d>] ? shrink_page_list+0x2ad/0x600
[770660.320531] [<ffffffff81101b7e>] ? shrink_inactive_list+0x30e/0x810
[770660.320536] [<ffffffff81156e68>] ? d_kill+0x68/0x80
[770660.320540] [<ffffffff81156edc>] ? prune_one_dentry+0x5c/0x110
[770660.320543] [<ffffffff811571d4>] ? __shrink_dcache_sb+0x244/0x2c0
[770660.320547] [<ffffffff810fbd6a>] ? determine_dirtyable_memory+0x1a/0x30
[770660.320551] [<ffffffff810fbe17>] ? get_dirty_limits+0x27/0x2f0
[770660.320554] [<ffffffff81102111>] ? shrink_list+0x91/0xf0
[770660.320557] [<ffffffff81102307>] ? shrink_zone+0x197/0x240
[770660.320560] [<ffffffff81103399>] ? balance_pgdat+0x659/0x6d0
[770660.320563] [<ffffffff811003c0>] ? isolate_pages_global+0x0/0x50
[770660.320566] [<ffffffff8110350e>] ? kswapd+0xfe/0x150
[770660.320570] [<ffffffff81084250>] ? autoremove_wake_function+0x0/0x40
[770660.320573] [<ffffffff81103410>] ? kswapd+0x0/0x150
[770660.320576] [<ffffffff81083ed6>] ? kthread+0x96/0xa0
[770660.320580] [<ffffffff810131ea>] ? child_rip+0xa/0x20
[770660.320583] [<ffffffff81083e40>] ? kthread+0x0/0xa0
[770660.320586] [<ffffffff810131e0>] ? child_rip+0x0/0x20
[770660.320598] Mem-Info:
[770660.320600] Node 0 DMA per-cpu:
[770660.320603] CPU 0: hi: 0, btch: 1 usd: 0
[770660.320605] CPU 1: hi: 0, btch: 1 usd: 0
[770660.320607] Node 0 DMA32 per-cpu:
[770660.320610] CPU 0: hi: 186, btch: 31 usd: 162
[770660.320612] CPU 1: hi: 186, btch: 31 usd: 156
[770660.320617] active_anon:2751 inactive_anon:11810 isolated_anon:0
[770660.320618] active_file:33607 inactive_file:182301 isolated_file:35
[770660.320619] unevictable:0 dirty:21611 writeback:500 unstable:0
[770660.320620] free:1369 slab_reclaimable:11657 slab_unreclaimable:2756
[770660.320622] mapped:1927 shmem:64 pagetables:518 bounce:0
[770660.320624] Node 0 DMA free:3988kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:132kB inactive_file:11080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:1564kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:192kB slab_unreclaimable:152kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[770660.320634] lowmem_reserve[]: 0 994 994 994
[770660.320638] Node 0 DMA32 free:1488kB min:4000kB low:5000kB high:6000kB active_anon:11004kB inactive_anon:47240kB active_file:134296kB inactive_file:718124kB unevictable:0kB isolated(anon):0kB isolated(file):140kB present:1018060kB mlocked:0kB dirty:84880kB writeback:2000kB mapped:7708kB shmem:256kB slab_reclaimable:46436kB slab_unreclaimable:10872kB kernel_stack:1504kB pagetables:2072kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:99 all_unreclaimable? no
[770660.320648] lowmem_reserve[]: 0 0 0 0
[770660.320652] Node 0 DMA: 1*4kB 2*8kB 2*16kB 1*32kB 11*64kB 1*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3988kB
[770660.320662] Node 0 DMA32: 94*4kB 5*8kB 1*16kB 1*32kB 6*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1488kB
[770660.320672] 216001 total pagecache pages
[770660.320674] 0 pages in swap cache
[770660.320676] Swap cache stats: add 0, delete 0, find 0/0
[770660.320678] Free swap = 0kB
[770660.320679] Total swap = 0kB
[770660.324197] 262139 pages RAM
[770660.324199] 6263 pages reserved
[770660.324201] 195041 pages shared
[770660.324202] 62533 pages non-shared

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: linux-image-2.6.32-25-virtual 2.6.32-25.44 [modified: lib/modules/2.6.32-25-server/modules.dep lib/modules/2.6.32-25-server/modules.pcimap lib/modules/2.6.32-25-server/modules.usbmap lib/modules/2.6.32-25-server/modules.seriomap lib/modules/2.6.32-25-server/modules.alias lib/modules/2.6.32-25-server/modules.alias.bin lib/modules/2.6.32-25-server/modules.symbols]
Regression: No
Reproducible: Yes
ProcVersionSignature: Ubuntu 2.6.32-25.44-server 2.6.32.21+drm33.7
Uname: Linux 2.6.32-25-server x86_64
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access /dev/snd/: No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
Date: Fri Oct 15 14:49:43 2010
InstallationMedia: Ubuntu-Server 10.04.1 LTS "Lucid Lynx" - Release amd64 (20100816.2)
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Bochs Bochs
PciMultimedia:

ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.32-25-server root=UUID=af1632c7-88b9-4a06-ae05-ae2647965da3 ro quiet
ProcEnviron:
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: linux
dmi.bios.date: 01/01/2007
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2007:svnBochs:pnBochs:pvr:cvnBochs:ct1:cvr:
dmi.product.name: Bochs
dmi.sys.vendor: Bochs

[Jeremy Foshee (jeremyfoshee)]

Hi Bernhard,

If you could also please test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

    [This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

[Bernhard Seibold (blubb8128)]

Running vanilla 2.6.35.7 now, and waiting for the bug to occur.

[Clayton Smith (argilo)]

I just had this occur today on a KVM guest running Ubuntu Server 10.04.1 64-bit. During a period of heavy network traffic, networking stopped working and I was unable to log into the virtual machine remotely. I was forced to kill and restart it using virsh on the KVM host machine (also running Ubuntu Server 10.04.1 64-bit).

My kernel is:

2.6.32-26-server #47-Ubuntu SMP Wed Nov 17 17:05:29 UTC 2010 x86_64 GNU/Linux

Here's what appeared in my syslog at the moment of the failure:

Nov 29 17:51:01 virtuala2 kernel: [316382.936157] kswapd0: page allocation failure. order:0, mode:0x4020
Nov 29 17:51:01 virtuala2 kernel: [316382.936162] Pid: 36, comm: kswapd0 Not tainted 2.6.32-26-server #47-Ubuntu
Nov 29 17:51:01 virtuala2 kernel: [316382.936165] Call Trace:
Nov 29 17:51:01 virtuala2 kernel: [316382.936167] <IRQ> [<ffffffff810f9a69>] __alloc_pages_slowpath+0x4a9/0x590
Nov 29 17:51:01 virtuala2 kernel: [316382.936178] [<ffffffff810f9cc1>] __alloc_pages_nodemask+0x171/0x180
Nov 29 17:51:01 virtuala2 kernel: [316382.936184] [<ffffffff8112ccf7>] alloc_pages_current+0x87/0xd0
Nov 29 17:51:01 virtuala2 kernel: [316382.936188] [<ffffffff81132d37>] new_slab+0x2f7/0x310
Nov 29 17:51:01 virtuala2 kernel: [316382.936191] [<ffffffff811355e1>] __slab_alloc+0x201/0x2d0
Nov 29 17:51:01 virtuala2 kernel: [316382.936196] [<ffffffff81469796>] ? __netdev_alloc_skb+0x36/0x60
Nov 29 17:51:01 virtuala2 kernel: [316382.936200] [<ffffffff811365bf>] __kmalloc_node_track_caller+0xaf/0x160
Nov 29 17:51:01 virtuala2 kernel: [316382.936204] [<ffffffff81469796>] ? __netdev_alloc_skb+0x36/0x60
Nov 29 17:51:01 virtuala2 kernel: [316382.936207] [<ffffffff81468fb0>] __alloc_skb+0x80/0x190
Nov 29 17:51:01 virtuala2 kernel: [316382.936210] [<ffffffff81469796>] __netdev_alloc_skb+0x36/0x60
Nov 29 17:51:01 virtuala2 kernel: [316382.936215] [<ffffffff813dacc4>] try_fill_recv+0x54/0x200
Nov 29 17:51:01 virtuala2 kernel: [316382.936218] [<ffffffff813db01d>] virtnet_poll+0x10d/0x160
Nov 29 17:51:01 virtuala2 kernel: [316382.936222] [<ffffffff8147356f>] net_rx_action+0x10f/0x250
Nov 29 17:51:01 virtuala2 kernel: [316382.936226] [<ffffffff8106d487>] __do_softirq+0xb7/0x1e0
Nov 29 17:51:01 virtuala2 kernel: [316382.936230] [<ffffffff810c4060>] ? handle_IRQ_event+0x60/0x170
Nov 29 17:51:01 virtuala2 kernel: [316382.936233] [<ffffffff8106d4e5>] ? __do_softirq+0x115/0x1e0
Nov 29 17:51:01 virtuala2 kernel: [316382.936237] [<ffffffff810132ec>] call_softirq+0x1c/0x30
Nov 29 17:51:01 virtuala2 kernel: [316382.936240] [<ffffffff81014cb5>] do_softirq+0x65/0xa0
Nov 29 17:51:01 virtuala2 kernel: [316382.936243] [<ffffffff8106d325>] irq_exit+0x85/0x90
Nov 29 17:51:01 virtuala2 kernel: [316382.936248] [<ffffffff8155f905>] do_IRQ+0x75/0xf0
Nov 29 17:51:01 virtuala2 kernel: [316382.936272] [<ffffffff81012b13>] ret_from_intr+0x0/0x11
Nov 29 17:51:01 virtuala2 kernel: [316382.936274] <EOI> [<ffffffff810fb1ac>] ? free_hot_cold_page+0xdc/0x240
Nov 29 17:51:01 virtuala2 kernel: [316382.936297] [<ffffffff8121ab47>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150
Nov 29 17:51:01 virtuala2 kernel: [316382.936302] [<ffffffff810fb369>] ? __pagevec_free+0x59/0xb0
Nov 29 17:51:01 virtuala2 ...

[Bernhard Seibold (blubb8128)]

Vanilla 2.6.35.7 still up and running, no crashes. Switching back to 2.6.32-26-virtual now.