We are seeing this bug often with Ubuntu 8.04 LTS on most of our servers, with a variety of network cards/drivers (bnx2, tg3, etc).

Here is an example trace:

Dec 28 00:43:42 brewster kernel: [5113787.548225] swapper: page allocation failure. order:1, mode:0x4020
Dec 28 00:43:42 brewster kernel: [5113787.548231] Pid: 0, comm: swapper Not tainted 2.6.24-25-server #1
Dec 28 00:43:42 brewster kernel: [5113787.548233] 
Dec 28 00:43:42 brewster kernel: [5113787.548233] Call Trace:
Dec 28 00:43:42 brewster kernel: [5113787.548235]  <IRQ>  [<ffffffff8028c54d>] __alloc_pages+0x2fd/0x3d0
Dec 28 00:43:42 brewster kernel: [5113787.548261]  [<ffffffff802aeea0>] new_slab+0x220/0x260
Dec 28 00:43:42 brewster kernel: [5113787.548266]  [<ffffffff802af1d5>] __slab_alloc+0x2f5/0x410
Dec 28 00:43:42 brewster kernel: [5113787.548269]  [<ffffffff803eb4fb>] __netdev_alloc_skb+0x2b/0x50
Dec 28 00:43:42 brewster kernel: [5113787.548273]  [<ffffffff803eb4fb>] __netdev_alloc_skb+0x2b/0x50
Dec 28 00:43:42 brewster kernel: [5113787.548276]  [<ffffffff802b0461>] __kmalloc_node_track_caller+0x121/0x130
Dec 28 00:43:42 brewster kernel: [5113787.548281]  [<ffffffff803ea75b>] __alloc_skb+0x7b/0x160
Dec 28 00:43:42 brewster kernel: [5113787.548285]  [<ffffffff803eb4fb>] __netdev_alloc_skb+0x2b/0x50
Dec 28 00:43:42 brewster kernel: [5113787.548293]  [<ffffffff880f1a46>] :tg3:tg3_alloc_rx_skb+0xb6/0x190
Dec 28 00:43:42 brewster kernel: [5113787.548300]  [<ffffffff880fae79>] :tg3:tg3_poll+0x8a9/0xa60
Dec 28 00:43:42 brewster kernel: [5113787.548314]  [<ffffffff803f27e8>] net_rx_action+0x128/0x230
Dec 28 00:43:42 brewster kernel: [5113787.548319]  [<ffffffff802440a5>] __do_softirq+0x75/0xe0
Dec 28 00:43:42 brewster kernel: [5113787.548323]  [<ffffffff8020d52c>] call_softirq+0x1c/0x30
Dec 28 00:43:42 brewster kernel: [5113787.548325]  [<ffffffff8020ed45>] do_softirq+0x35/0x90
Dec 28 00:43:42 brewster kernel: [5113787.548327]  [<ffffffff80244028>] irq_exit+0x88/0x90
Dec 28 00:43:42 brewster kernel: [5113787.548329]  [<ffffffff8020ef90>] do_IRQ+0x80/0x100
Dec 28 00:43:42 brewster kernel: [5113787.548331]  [<ffffffff8020b390>] default_idle+0x0/0x40
Dec 28 00:43:42 brewster kernel: [5113787.548333]  [<ffffffff8020c8b1>] ret_from_intr+0x0/0xa
Dec 28 00:43:42 brewster kernel: [5113787.548334]  <EOI>  [<ffffffff803dcbd0>] menu_reflect+0x0/0xb0
Dec 28 00:43:42 brewster kernel: [5113787.548343]  [<ffffffff8803c234>] :processor:acpi_safe_halt+0x28/0x39
Dec 28 00:43:42 brewster kernel: [5113787.548348]  [<ffffffff8803c2ac>] :processor:acpi_idle_enter_c1+0x67/0x70
Dec 28 00:43:42 brewster kernel: [5113787.548349]  [<ffffffff8020b390>] default_idle+0x0/0x40
Dec 28 00:43:42 brewster kernel: [5113787.548352]  [<ffffffff803dbf62>] cpuidle_idle_call+0xa2/0xe0
Dec 28 00:43:42 brewster kernel: [5113787.548354]  [<ffffffff8020b390>] default_idle+0x0/0x40
Dec 28 00:43:42 brewster kernel: [5113787.548355]  [<ffffffff803dbec0>] cpuidle_idle_call+0x0/0xe0
Dec 28 00:43:42 brewster kernel: [5113787.548357]  [<ffffffff8020b418>] cpu_idle+0x48/0xe0
Dec 28 00:43:42 brewster kernel: [5113787.548361]  [<ffffffff80632885>] start_kernel+0x2c5/0x350
Dec 28 00:43:42 brewster kernel: [5113787.548364]  [<ffffffff8063212e>] _sinittext+0x12e/0x140
Dec 28 00:43:42 brewster kernel: [5113787.548366] 
Dec 28 00:43:42 brewster kernel: [5113787.548367] Mem-info:
Dec 28 00:43:42 brewster kernel: [5113787.548368] Node 0 DMA per-cpu:
Dec 28 00:43:42 brewster kernel: [5113787.548369] CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Dec 28 00:43:42 brewster kernel: [5113787.548371] CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Dec 28 00:43:42 brewster kernel: [5113787.548372] Node 0 DMA32 per-cpu:
Dec 28 00:43:42 brewster kernel: [5113787.548374] CPU    0: Hot: hi:  186, btch:  31 usd: 116   Cold: hi:   62, btch:  15 usd:  59
Dec 28 00:43:42 brewster kernel: [5113787.548376] CPU    1: Hot: hi:  186, btch:  31 usd:  45   Cold: hi:   62, btch:  15 usd:  57
Dec 28 00:43:42 brewster kernel: [5113787.548377] Node 0 Normal per-cpu:
Dec 28 00:43:42 brewster kernel: [5113787.548378] CPU    0: Hot: hi:  186, btch:  31 usd: 165   Cold: hi:   62, btch:  15 usd:  50
Dec 28 00:43:42 brewster kernel: [5113787.548380] CPU    1: Hot: hi:  186, btch:  31 usd: 143   Cold: hi:   62, btch:  15 usd:  54
Dec 28 00:43:42 brewster kernel: [5113787.548382] Active:154755 inactive:661784 dirty:18126 writeback:151 unstable:0
Dec 28 00:43:42 brewster kernel: [5113787.548383]  free:4238 slab:183996 mapped:2062 pagetables:621 bounce:0
Dec 28 00:43:42 brewster kernel: [5113787.548384] Node 0 DMA free:10940kB min:20kB low:24kB high:28kB active:0kB inactive:0kB present:10468kB pages_scanned:0 all_unreclaimable? yes
Dec 28 00:43:42 brewster kernel: [5113787.548387] lowmem_reserve[]: 0 3250 4008 4008
Dec 28 00:43:42 brewster kernel: [5113787.548389] Node 0 DMA32 free:5460kB min:6560kB low:8200kB high:9840kB active:462416kB inactive:2222256kB present:3328864kB pages_scanned:0 all_unreclaimable? no
Dec 28 00:43:42 brewster kernel: [5113787.548391] lowmem_reserve[]: 0 0 757 757
Dec 28 00:43:42 brewster kernel: [5113787.548393] Node 0 Normal free:552kB min:1528kB low:1908kB high:2292kB active:156604kB inactive:424880kB present:775680kB pages_scanned:0 all_unreclaimable? no
Dec 28 00:43:42 brewster kernel: [5113787.548396] lowmem_reserve[]: 0 0 0 0
Dec 28 00:43:42 brewster kernel: [5113787.548398] Node 0 DMA: 3*4kB 4*8kB 7*16kB 7*32kB 5*64kB 2*128kB 3*256kB 2*512kB 2*1024kB 1*2048kB 1*4096kB = 10940kB
Dec 28 00:43:42 brewster kernel: [5113787.548403] Node 0 DMA32: 867*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 5372kB
Dec 28 00:43:42 brewster kernel: [5113787.548407] Node 0 Normal: 14*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 544kB
Dec 28 00:43:42 brewster kernel: [5113787.548412] Swap cache: add 130, delete 130, find 44/66, race 0+0
Dec 28 00:43:42 brewster kernel: [5113787.548413] Free swap  = 2097016kB
Dec 28 00:43:42 brewster kernel: [5113787.548414] Total swap = 2097144kB
Dec 28 00:43:42 brewster kernel: [5113787.548415] Free swap:       2097016kB
Dec 28 00:43:42 brewster kernel: [5113787.567311] 1245184 pages of RAM
Dec 28 00:43:42 brewster kernel: [5113787.567313] 232933 reserved pages
Dec 28 00:43:42 brewster kernel: [5113787.567314] 511256 pages shared
Dec 28 00:43:42 brewster kernel: [5113787.567314] 0 pages swap cached