No, I am completely wrong.  It still happens all the time.  There was a period when it seemed stable.  Here is my kernel panic fwiw on lucid linux-image-2.6.32-25-server       2.6.32-25.45:
[747393.713739] swapper: page allocation failure. order:0, mode:0x4020
[747393.713743] Pid: 0, comm: swapper Not tainted 2.6.32-25-server #45-Ubuntu
[747393.713745] Call Trace:
[747393.713746]  <IRQ>  [<ffffffff810f9a2e>] __alloc_pages_slowpath+0x56e/0x580
[747393.713756]  [<ffffffff810f9bb1>] __alloc_pages_nodemask+0x171/0x180
[747393.713760]  [<ffffffff8112cae7>] alloc_pages_current+0x87/0xd0
[747393.713763]  [<ffffffff81132b27>] new_slab+0x2f7/0x310
[747393.713766]  [<ffffffff811353d1>] __slab_alloc+0x201/0x2d0
[747393.713769]  [<ffffffff81468de6>] ? __netdev_alloc_skb+0x36/0x60
[747393.713772]  [<ffffffff811363af>] __kmalloc_node_track_caller+0xaf/0x160
[747393.713774]  [<ffffffff81468de6>] ? __netdev_alloc_skb+0x36/0x60
[747393.713776]  [<ffffffff81468aa0>] __alloc_skb+0x80/0x190
[747393.713778]  [<ffffffff81468de6>] __netdev_alloc_skb+0x36/0x60
[747393.713788]  [<ffffffffa00063c5>] e1000_alloc_rx_buffers+0x1c5/0x420 [e1000]
[747393.713792]  [<ffffffffa0004bae>] e1000_clean_rx_irq+0x3fe/0x530 [e1000]
[747393.713795]  [<ffffffff8106d2d8>] ? irq_exit+0x48/0x90
[747393.713799]  [<ffffffffa00032c1>] e1000_clean+0x51/0x230 [e1000]
[747393.713802]  [<ffffffff8147300f>] net_rx_action+0x10f/0x250
[747393.713806]  [<ffffffff81019103>] ? native_sched_clock+0x13/0x60
[747393.713808]  [<ffffffff8106d477>] __do_softirq+0xb7/0x1e0
[747393.713811]  [<ffffffff81030b22>] ? ack_apic_level+0x82/0x1f0
[747393.713813]  [<ffffffff810132ec>] call_softirq+0x1c/0x30
[747393.713815]  [<ffffffff81014cb5>] do_softirq+0x65/0xa0
[747393.713817]  [<ffffffff8106d315>] irq_exit+0x85/0x90
[747393.713820]  [<ffffffff8155f2a5>] do_IRQ+0x75/0xf0
[747393.713822]  [<ffffffff81012b13>] ret_from_intr+0x0/0x11
[747393.713823]  <EOI>  [<ffffffff81037adb>] ? native_safe_halt+0xb/0x10
[747393.713829]  [<ffffffff8155ce86>] ? notifier_call_chain+0x16/0x80
[747393.713831]  [<ffffffff8101a6ad>] ? default_idle+0x3d/0x90
[747393.713833]  [<ffffffff8101a763>] ? c1e_idle+0x63/0x120
[747393.713836]  [<ffffffff81010e63>] ? cpu_idle+0xb3/0x110
[747393.713839]  [<ffffffff81541c8b>] ? rest_init+0x6b/0x80
[747393.713843]  [<ffffffff8187edcc>] ? start_kernel+0x368/0x371
[747393.713845]  [<ffffffff8187e33a>] ? x86_64_start_reservations+0x125/0x129
[747393.713847]  [<ffffffff8187e438>] ? x86_64_start_kernel+0xfa/0x109
[747393.713848] Mem-Info:
[747393.713850] Node 0 DMA per-cpu:
[747393.713851] CPU    0: hi:    0, btch:   1 usd:   0
[747393.713853] CPU    1: hi:    0, btch:   1 usd:   0
[747393.713854] Node 0 DMA32 per-cpu:
[747393.713856] CPU    0: hi:  186, btch:  31 usd: 186
[747393.713857] CPU    1: hi:  186, btch:  31 usd:  65
[747393.713858] Node 0 Normal per-cpu:
[747393.713860] CPU    0: hi:  186, btch:  31 usd:  98
[747393.713861] CPU    1: hi:  186, btch:  31 usd: 156
[747393.713865] active_anon:466544 inactive_anon:140501 isolated_anon:0
[747393.713865]  active_file:61663 inactive_file:135115 isolated_file:0
[747393.713866]  unevictable:0 dirty:75800 writeback:35617 unstable:1118
[747393.713867]  free:4512 slab_reclaimable:17297 slab_unreclaimable:12299
[747393.713868]  mapped:5623 shmem:228 pagetables:2808 bounce:0
[747393.713869] Node 0 DMA free:13688kB min:32kB low:40kB high:48kB active_anon:120kB inactive_anon:360kB active_file:72kB inactive_file:1492kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15300kB mlocked:0kB dirty:28kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:56kB slab_unreclaimable:60kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[747393.713876] lowmem_reserve[]: 0 3000 3422 3422
[747393.713879] Node 0 DMA32 free:4052kB min:6548kB low:8184kB high:9820kB active_anon:1739864kB inactive_anon:435020kB active_file:213976kB inactive_file:494808kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:294352kB writeback:121532kB mapped:16836kB shmem:604kB slab_reclaimable:61964kB slab_unreclaimable:37148kB kernel_stack:4000kB pagetables:7496kB unstable:4472kB bounce:0kB writeback_tmp:0kB pages_scanned:256 all_unreclaimable? no
[747393.713886] lowmem_reserve[]: 0 0 422 422
[747393.713888] Node 0 Normal free:308kB min:920kB low:1148kB high:1380kB active_anon:126192kB inactive_anon:126624kB active_file:32604kB inactive_file:44160kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:432280kB mlocked:0kB dirty:8820kB writeback:20936kB mapped:5648kB shmem:308kB slab_reclaimable:7168kB slab_unreclaimable:11988kB kernel_stack:1624kB pagetables:3736kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no
[747393.713895] lowmem_reserve[]: 0 0 0 0
[747393.713897] Node 0 DMA: 2*4kB 4*8kB 3*16kB 7*32kB 7*64kB 3*128kB 3*256kB 3*512kB 2*1024kB 2*2048kB 1*4096kB = 13688kB
[747393.713903] Node 0 DMA32: 833*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4052kB
[747393.713909] Node 0 Normal: 21*4kB 16*8kB 2*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 308kB
[747393.713914] 200977 total pagecache pages
[747393.713916] 3939 pages in swap cache
[747393.713917] Swap cache stats: add 1523025, delete 1519086, find 539681/705157
[747393.713918] Free swap  = 4159756kB
[747393.713919] Total swap = 4194296kB
[747393.723663] 895984 pages RAM
[747393.723663] 32065 pages reserved
[747393.723663] 167230 pages shared
[747393.723663] 705863 pages non-shared
[747393.723663] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[747393.723663]   cache: kmalloc-4096, object size: 4096, buffer size: 4096, default order: 3, min order: 0
[747393.723663]   node 0: slabs: 593, objs: 1363, free: 0