Comment 19 for bug 164018

Revision history for this message
Mark Bergsma (mark-wikimedia) wrote :

We are seeing this bug often with Ubuntu 8.04 LTS on most of our servers, with a variety of network cards/drivers (bnx2, tg3, etc).

Here is an example trace:

Dec 28 00:43:42 brewster kernel: [5113787.548225] swapper: page allocation failure. order:1, mode:0x4020
Dec 28 00:43:42 brewster kernel: [5113787.548231] Pid: 0, comm: swapper Not tainted 2.6.24-25-server #1
Dec 28 00:43:42 brewster kernel: [5113787.548233]
Dec 28 00:43:42 brewster kernel: [5113787.548233] Call Trace:
Dec 28 00:43:42 brewster kernel: [5113787.548235] <IRQ> [<ffffffff8028c54d>] __alloc_pages+0x2fd/0x3d0
Dec 28 00:43:42 brewster kernel: [5113787.548261] [<ffffffff802aeea0>] new_slab+0x220/0x260
Dec 28 00:43:42 brewster kernel: [5113787.548266] [<ffffffff802af1d5>] __slab_alloc+0x2f5/0x410
Dec 28 00:43:42 brewster kernel: [5113787.548269] [<ffffffff803eb4fb>] __netdev_alloc_skb+0x2b/0x50
Dec 28 00:43:42 brewster kernel: [5113787.548273] [<ffffffff803eb4fb>] __netdev_alloc_skb+0x2b/0x50
Dec 28 00:43:42 brewster kernel: [5113787.548276] [<ffffffff802b0461>] __kmalloc_node_track_caller+0x121/0x130
Dec 28 00:43:42 brewster kernel: [5113787.548281] [<ffffffff803ea75b>] __alloc_skb+0x7b/0x160
Dec 28 00:43:42 brewster kernel: [5113787.548285] [<ffffffff803eb4fb>] __netdev_alloc_skb+0x2b/0x50
Dec 28 00:43:42 brewster kernel: [5113787.548293] [<ffffffff880f1a46>] :tg3:tg3_alloc_rx_skb+0xb6/0x190
Dec 28 00:43:42 brewster kernel: [5113787.548300] [<ffffffff880fae79>] :tg3:tg3_poll+0x8a9/0xa60
Dec 28 00:43:42 brewster kernel: [5113787.548314] [<ffffffff803f27e8>] net_rx_action+0x128/0x230
Dec 28 00:43:42 brewster kernel: [5113787.548319] [<ffffffff802440a5>] __do_softirq+0x75/0xe0
Dec 28 00:43:42 brewster kernel: [5113787.548323] [<ffffffff8020d52c>] call_softirq+0x1c/0x30
Dec 28 00:43:42 brewster kernel: [5113787.548325] [<ffffffff8020ed45>] do_softirq+0x35/0x90
Dec 28 00:43:42 brewster kernel: [5113787.548327] [<ffffffff80244028>] irq_exit+0x88/0x90
Dec 28 00:43:42 brewster kernel: [5113787.548329] [<ffffffff8020ef90>] do_IRQ+0x80/0x100
Dec 28 00:43:42 brewster kernel: [5113787.548331] [<ffffffff8020b390>] default_idle+0x0/0x40
Dec 28 00:43:42 brewster kernel: [5113787.548333] [<ffffffff8020c8b1>] ret_from_intr+0x0/0xa
Dec 28 00:43:42 brewster kernel: [5113787.548334] <EOI> [<ffffffff803dcbd0>] menu_reflect+0x0/0xb0
Dec 28 00:43:42 brewster kernel: [5113787.548343] [<ffffffff8803c234>] :processor:acpi_safe_halt+0x28/0x39
Dec 28 00:43:42 brewster kernel: [5113787.548348] [<ffffffff8803c2ac>] :processor:acpi_idle_enter_c1+0x67/0x70
Dec 28 00:43:42 brewster kernel: [5113787.548349] [<ffffffff8020b390>] default_idle+0x0/0x40
Dec 28 00:43:42 brewster kernel: [5113787.548352] [<ffffffff803dbf62>] cpuidle_idle_call+0xa2/0xe0
Dec 28 00:43:42 brewster kernel: [5113787.548354] [<ffffffff8020b390>] default_idle+0x0/0x40
Dec 28 00:43:42 brewster kernel: [5113787.548355] [<ffffffff803dbec0>] cpuidle_idle_call+0x0/0xe0
Dec 28 00:43:42 brewster kernel: [5113787.548357] [<ffffffff8020b418>] cpu_idle+0x48/0xe0
Dec 28 00:43:42 brewster kernel: [5113787.548361] [<ffffffff80632885>] start_kernel+0x2c5/0x350
Dec 28 00:43:42 brewster kernel: [5113787.548364] [<ffffffff8063212e>] _sinittext+0x12e/0x140
Dec 28 00:43:42 brewster kernel: [5113787.548366]
Dec 28 00:43:42 brewster kernel: [5113787.548367] Mem-info:
Dec 28 00:43:42 brewster kernel: [5113787.548368] Node 0 DMA per-cpu:
Dec 28 00:43:42 brewster kernel: [5113787.548369] CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Dec 28 00:43:42 brewster kernel: [5113787.548371] CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Dec 28 00:43:42 brewster kernel: [5113787.548372] Node 0 DMA32 per-cpu:
Dec 28 00:43:42 brewster kernel: [5113787.548374] CPU 0: Hot: hi: 186, btch: 31 usd: 116 Cold: hi: 62, btch: 15 usd: 59
Dec 28 00:43:42 brewster kernel: [5113787.548376] CPU 1: Hot: hi: 186, btch: 31 usd: 45 Cold: hi: 62, btch: 15 usd: 57
Dec 28 00:43:42 brewster kernel: [5113787.548377] Node 0 Normal per-cpu:
Dec 28 00:43:42 brewster kernel: [5113787.548378] CPU 0: Hot: hi: 186, btch: 31 usd: 165 Cold: hi: 62, btch: 15 usd: 50
Dec 28 00:43:42 brewster kernel: [5113787.548380] CPU 1: Hot: hi: 186, btch: 31 usd: 143 Cold: hi: 62, btch: 15 usd: 54
Dec 28 00:43:42 brewster kernel: [5113787.548382] Active:154755 inactive:661784 dirty:18126 writeback:151 unstable:0
Dec 28 00:43:42 brewster kernel: [5113787.548383] free:4238 slab:183996 mapped:2062 pagetables:621 bounce:0
Dec 28 00:43:42 brewster kernel: [5113787.548384] Node 0 DMA free:10940kB min:20kB low:24kB high:28kB active:0kB inactive:0kB present:10468kB pages_scanned:0 all_unreclaimable? yes
Dec 28 00:43:42 brewster kernel: [5113787.548387] lowmem_reserve[]: 0 3250 4008 4008
Dec 28 00:43:42 brewster kernel: [5113787.548389] Node 0 DMA32 free:5460kB min:6560kB low:8200kB high:9840kB active:462416kB inactive:2222256kB present:3328864kB pages_scanned:0 all_unreclaimable? no
Dec 28 00:43:42 brewster kernel: [5113787.548391] lowmem_reserve[]: 0 0 757 757
Dec 28 00:43:42 brewster kernel: [5113787.548393] Node 0 Normal free:552kB min:1528kB low:1908kB high:2292kB active:156604kB inactive:424880kB present:775680kB pages_scanned:0 all_unreclaimable? no
Dec 28 00:43:42 brewster kernel: [5113787.548396] lowmem_reserve[]: 0 0 0 0
Dec 28 00:43:42 brewster kernel: [5113787.548398] Node 0 DMA: 3*4kB 4*8kB 7*16kB 7*32kB 5*64kB 2*128kB 3*256kB 2*512kB 2*1024kB 1*2048kB 1*4096kB = 10940kB
Dec 28 00:43:42 brewster kernel: [5113787.548403] Node 0 DMA32: 867*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 5372kB
Dec 28 00:43:42 brewster kernel: [5113787.548407] Node 0 Normal: 14*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 544kB
Dec 28 00:43:42 brewster kernel: [5113787.548412] Swap cache: add 130, delete 130, find 44/66, race 0+0
Dec 28 00:43:42 brewster kernel: [5113787.548413] Free swap = 2097016kB
Dec 28 00:43:42 brewster kernel: [5113787.548414] Total swap = 2097144kB
Dec 28 00:43:42 brewster kernel: [5113787.548415] Free swap: 2097016kB
Dec 28 00:43:42 brewster kernel: [5113787.567311] 1245184 pages of RAM
Dec 28 00:43:42 brewster kernel: [5113787.567313] 232933 reserved pages
Dec 28 00:43:42 brewster kernel: [5113787.567314] 511256 pages shared
Dec 28 00:43:42 brewster kernel: [5113787.567314] 0 pages swap cached