I only seem to get this problem as soon as I hit a memory spike (usually with Firefox), then I get a bunch of swap built up, iwlagn will start saying:
[181706.508110] iwlagn 0000:03:00.0: Can not allocate SKB buffers
[181706.508110] swapper: page allocation failure. order:2, mode:0x4020
[181706.508110] Pid: 0, comm: swapper Tainted: G         C 2.6.31-14-generic #48-Ubuntu
[181706.508110] Call Trace:
[181706.508110]  <IRQ>  [<ffffffff810e083c>] __alloc_pages_slowpath+0x4cc/0x4e0
[181706.508110]  [<ffffffff810e099e>] __alloc_pages_nodemask+0x14e/0x150
[181706.508110]  [<ffffffff81111dfa>] kmalloc_large_node+0x5a/0xb0
[181706.508110]  [<ffffffff81115fa5>] __kmalloc_node_track_caller+0x135/0x180
[181706.508110]  [<ffffffffa0286899>] ? iwl_rx_allocate+0x1a9/0x230 [iwlcore]
[181706.508110]  [<ffffffff8142e41b>] __alloc_skb+0x7b/0x180
[181706.508110]  [<ffffffffa0286899>] iwl_rx_allocate+0x1a9/0x230 [iwlcore]
[181706.508110]  [<ffffffff8106b933>] ? mod_timer+0x103/0x160
[181706.508110]  [<ffffffffa0287c36>] iwl_rx_replenish_now+0x16/0x30 [iwlcore]
[181706.508110]  [<ffffffffa02a44f8>] iwl_rx_handle+0x288/0x2f0 [iwlagn]
[181706.508110]  [<ffffffffa02a4767>] iwl_irq_tasklet_legacy+0x207/0x440 [iwlagn]
[181706.508110]  [<ffffffff8107c29b>] ? hrtimer_get_next_event+0xbb/0xf0
[181706.508110]  [<ffffffff81063ee0>] tasklet_action+0xd0/0xe0
[181706.508110]  [<ffffffff8106549d>] __do_softirq+0xbd/0x200
[181706.508110]  [<ffffffff810131ec>] call_softirq+0x1c/0x30
[181706.508110]  [<ffffffff81014bc5>] do_softirq+0x55/0x90
[181706.508110]  [<ffffffff81065205>] irq_exit+0x85/0x90
[181706.508110]  [<ffffffff81014100>] do_IRQ+0x70/0xe0
[181706.508110]  [<ffffffff81012a13>] ret_from_intr+0x0/0x11
[181706.508110]  <EOI>  [<ffffffff812d7ed9>] ? acpi_idle_enter_bm+0x28b/0x2bf
[181706.508110]  [<ffffffff812d7ed2>] ? acpi_idle_enter_bm+0x284/0x2bf
[181706.508110]  [<ffffffff813fe40b>] ? cpuidle_idle_call+0x9b/0xf0
[181706.508110]  [<ffffffff81010e12>] ? cpu_idle+0xb2/0x100
[181706.508110]  [<ffffffff81514c56>] ? rest_init+0x66/0x70
[181706.508110]  [<ffffffff8183a047>] ? start_kernel+0x352/0x35b
[181706.508110]  [<ffffffff8183959a>] ? x86_64_start_reservations+0x125/0x129
[181706.508110]  [<ffffffff81839698>] ? x86_64_start_kernel+0xfa/0x109
[181706.508110] Mem-Info:
[181706.508110] Node 0 DMA per-cpu:
[181706.508110] CPU    0: hi:    0, btch:   1 usd:   0
[181706.508110] CPU    1: hi:    0, btch:   1 usd:   0
[181706.508110] Node 0 DMA32 per-cpu:
[181706.508110] CPU    0: hi:  186, btch:  31 usd: 104
[181706.508110] CPU    1: hi:  186, btch:  31 usd:  77
[181706.508110] Active_anon:285907 active_file:35149 inactive_anon:95722
[181706.508110]  inactive_file:33632 unevictable:4 dirty:26710 writeback:437 unstable:0
[181706.508110]  free:3269 slab:22972 mapped:15303 pagetables:8144 bounce:0
[181706.508110] Node 0 DMA free:7876kB min:40kB low:48kB high:60kB active_anon:1772kB inactive_anon:2148kB active_file:1188kB inactive_file:984kB unevictable:0kB present:15332kB pages_scanned:0 all_unreclaimable? no
[181706.508110] lowmem_reserve[]: 0 1963 1963 1963
[181706.508110] Node 0 DMA32 free:5200kB min:5648kB low:7060kB high:8472kB active_anon:1141856kB inactive_anon:380740kB active_file:139408kB inactive_file:133544kB unevictable:16kB present:2010592kB pages_scanned:1300 all_unreclaimable? no
[181706.508110] lowmem_reserve[]: 0 0 0 0
[181706.508110] Node 0 DMA: 25*4kB 10*8kB 23*16kB 37*32kB 28*64kB 10*128kB 4*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 7876kB
[181706.508110] Node 0 DMA32: 1034*4kB 9*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 5200kB
[181706.508110] 147806 total pagecache pages
[181706.508110] 27280 pages in swap cache
[181706.508110] Swap cache stats: add 624512, delete 597232, find 4209959/4259844
[181706.508110] Free swap  = 5190824kB
[181706.508110] Total swap = 5919912kB
[181706.508110] 513712 pages RAM
[181706.508110] 9742 pages reserved
[181706.508110] 155236 pages shared
[181706.508110] 397990 pages non-shared
[181706.508110] iwlagn 0000:03:00.0: Can not allocate SKB buffers
bram@owl:~/program/public_html/vserver/perqa/perqa.com/perqa$

And iwlagn will get stuck at a 100% CPU. I'm currently using Ubuntu Karmic with the newest updates. I have not found a way to reproduce it easily, so it may very well be a race condition.