swapper: page allocation failure

[swmike]

When running rtorrent with a lot of open TCP connections, I get page allocation failures. Fully apt-get update:d 8.10.

I include lspci and complete dmesg (in same file).

$ uname -a
Linux ub 2.6.27-7-generic #1 SMP Tue Nov 4 19:33:06 UTC 2008 x86_64 GNU/Linux

This is an Asus P5E-WS-PRO motherboard (Intel X38 based) with E6600 processor, and 2 gig ram.

[gna (nagy-gergely)]

We use Apache and OCFS2 on 3 servers and at least 2 of them are reporting those swapper page allocation failures.

Machines are HP DL380 G5 with 8GB RAM (DIMM and FB-DIMM on different machines) and OCFS2 on a qlogic fiber attached MSA storage.

Example from one machine:
Linux web2 2.6.24-24-server #1 SMP Wed Apr 15 16:36:01 UTC 2009 i686 GNU/Linux

swapper: page allocation failure. order:1, mode:0x4020

Pid: 0, comm: swapper Not tainted 2.6.24-24-server #1

 [<c01785ab>] __alloc_pages+0x2ab/0x380

 [<c02dbb89>] ip_local_deliver_finish+0xf9/0x210

 [<c0196226>] __slab_alloc+0x186/0x4a0

 [<c0197446>] __kmalloc_track_caller+0x106/0x110

 [<c02b4d32>] __netdev_alloc_skb+0x22/0x50

 [<c02b4d32>] __netdev_alloc_skb+0x22/0x50

 [<c02b4125>] __alloc_skb+0x55/0x120

 [<c02b4d32>] __netdev_alloc_skb+0x22/0x50

 [<f89592c1>] bnx2_poll+0x6f1/0xd10 [bnx2]

 [<c012d974>] load_balance_fair+0xc4/0x100

 [<f8958c00>] bnx2_poll+0x30/0xd10 [bnx2]

 [<f8958e15>] bnx2_poll+0x245/0xd10 [bnx2]

 [<c014c024>] getnstimeofday+0x34/0xf0

 [<c02bb7ed>] net_rx_action+0x12d/0x210

 [<c01498f8>] ktime_get+0x18/0x40

 [<c0136e42>] __do_softirq+0x82/0x110

 [<c0136f25>] do_softirq+0x55/0x60

 [<c013720d>] irq_exit+0x6d/0x80

 [<c010a930>] do_IRQ+0x40/0x70

 [<c014989e>] ktime_get_ts+0x1e/0x60

 [<c0108def>] common_interrupt+0x23/0x28

 [<c01062e6>] mwait_idle_with_hints+0x46/0x60

 [<c01066c3>] cpu_idle+0x73/0xd0

 =======================

Mem-info:

DMA per-cpu:

CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0

CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0

Normal per-cpu:

CPU 0: Hot: hi: 186, btch: 31 usd: 176 Cold: hi: 62, btch: 15 usd: 57

CPU 1: Hot: hi: 186, btch: 31 usd: 149 Cold: hi: 62, btch: 15 usd: 50

HighMem per-cpu:

CPU 0: Hot: hi: 186, btch: 31 usd: 53 Cold: hi: 62, btch: 15 usd: 14

CPU 1: Hot: hi: 186, btch: 31 usd: 181 Cold: hi: 62, btch: 15 usd: 11

Active:486821 inactive:222382 dirty:1086 writeback:43 unstable:0 free:1236121 slab:125422 mapped:5242 pagetables:1068 bounce:0

DMA free:3508kB min:68kB low:84kB high:100kB active:4kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 873 8874 8874

Normal free:122668kB min:3744kB low:4680kB high:5616kB active:85852kB inactive:111552kB present:894080kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 64008 64008

HighMem free:4818308kB min:512kB low:9096kB high:17684kB active:1861428kB inactive:777976kB present:8193024kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0

DMA: 137*4kB 48*8kB 32*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3492kB

Normal: 30507*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 122628kB

HighMem: 6664*4kB 14962*8kB 19387*16kB 19951*32kB 10681*64kB 4402*128kB 1783*256kB 819*512kB 323*1024kB 200*2048kB 210*4096kB = 4818304kB

Swap cache: add 1, delete 0, find 0/0, race 0+0

Free swap = 1951884kB

Total swap = 1951888kB

Free swap: 1951884kB

2293759 pages of RAM

2064383 pages of HIGHMEM

216253 reserved pages

341800 pages shared

1 pages sw...

[kernel-janitor (kernel-janitor)]

Hi swmike,

This bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? Can you try with the latest development release of Ubuntu? ISO CD images are available from http://cdimage.ubuntu.com/releases/karmic .

If it remains an issue, could you run the following command from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux 296275

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

[This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

[swmike (ubuntu-s-plass)]

Hi,

I have been running 9.04 for quite a while now (since its release), but I have also upgraded to 8 gigs of ram and limited the TCP window size used.

With these changes I have not seen the problem, so I don't know if they still are there but that these changes made me not hit the problems anymore.

[Leann Ogasawara (leannogasawara)]

Thanks for the update. I'll just go ahead and close this bug out for now.

[bparab (bparab)]

Hi,

The same on a Dell; ftp-server only; max 20 simultaneus conections:

4Gb. RAM - 2 sockets - 2 cores.

Linux soloftp 2.6.28-15-server #52-Ubuntu SMP Wed Sep 9 11:34:09 UTC 2009 x86_64 GNU/Linux

Oct 10 01:07:47 soloftp kernel: [152151.772884] swapper: page allocation failure. order:0, mode:0x4020
Oct 10 01:07:48 soloftp kernel: [152151.772889] Pid: 0, comm: swapper Not tainted 2.6.28-15-server #52-Ubuntu
Oct 10 01:07:48 soloftp kernel: [152151.772892] Call Trace:
Oct 10 01:07:48 soloftp kernel: [152151.772895] <IRQ> [<ffffffff802b6c4e>] __alloc_pages_internal+0x3ee/0x4f0
Oct 10 01:07:48 soloftp kernel: [152151.772920] [<ffffffff802dfc18>] alloc_slab_page+0x28/0x30
Oct 10 01:07:48 soloftp kernel: [152151.772923] [<ffffffff802e0cda>] new_slab+0x5a/0x210
Oct 10 01:07:48 soloftp kernel: [152151.772926] [<ffffffff802e22f8>] __slab_alloc+0x188/0x290
Oct 10 01:07:48 soloftp kernel: [152151.772931] [<ffffffff805a9a7f>] ? __netdev_alloc_skb+0x1f/0x40
Oct 10 01:07:48 soloftp kernel: [152151.772935] [<ffffffff802e3207>] __kmalloc_track_caller+0xd7/0x110
Oct 10 01:07:48 soloftp kernel: [152151.772938] [<ffffffff805a9a7f>] ? __netdev_alloc_skb+0x1f/0x40
Oct 10 01:07:48 soloftp kernel: [152151.772941] [<ffffffff805a971e>] __alloc_skb+0x6e/0x150
Oct 10 01:07:48 soloftp kernel: [152151.772944] [<ffffffff805a9a7f>] __netdev_alloc_skb+0x1f/0x40
Oct 10 01:07:48 soloftp kernel: [152151.772964] [<ffffffffa00bb714>] bnx2_rx_skb+0x74/0x7e0 [bnx2]
Oct 10 01:07:48 soloftp kernel: [152151.772978] [<ffffffff8069b139>] ? _spin_lock+0x9/0x10
Oct 10 01:07:48 soloftp kernel: [152151.772982] [<ffffffff805dab74>] ? ip_rcv+0x264/0x2f0
Oct 10 01:07:48 soloftp kernel: [152151.772987] [<ffffffffa00be94e>] bnx2_rx_int+0x44e/0x7b0 [bnx2]
Oct 10 01:07:48 soloftp kernel: [152151.772993] [<ffffffff80270b09>] ? getnstimeofday+0x59/0xe0
Oct 10 01:07:48 soloftp kernel: [152151.772997] [<ffffffff8022000d>] ? mtrr_ioctl+0x14d/0x6d0
Oct 10 01:07:48 soloftp kernel: [152151.773000] [<ffffffff8027390f>] ? clockevents_program_event+0x4f/0x90
Oct 10 01:07:48 soloftp kernel: [152151.773003] [<ffffffff80274e95>] ? tick_program_event+0x25/0x30
Oct 10 01:07:48 soloftp kernel: [152151.773008] [<ffffffffa00bed20>] bnx2_poll_work+0x70/0x90 [bnx2]
Oct 10 01:07:48 soloftp kernel: [152151.773013] [<ffffffffa00bedb5>] bnx2_poll+0x75/0x1a0 [bnx2]
Oct 10 01:07:48 soloftp kernel: [152151.773016] [<ffffffff80270b09>] ? getnstimeofday+0x59/0xe0
Oct 10 01:07:48 soloftp kernel: [152151.773020] [<ffffffff805b3894>] net_rx_action+0x104/0x240
Oct 10 01:07:48 soloftp kernel: [152151.773024] [<ffffffff80256bdc>] __do_softirq+0x9c/0x170
Oct 10 01:07:48 soloftp kernel: [152151.773027] [<ffffffff80213d8c>] call_softirq+0x1c/0x30
Oct 10 01:07:48 soloftp kernel: [152151.773031] [<ffffffff80214ffd>] do_softirq+0x5d/0xa0
Oct 10 01:07:48 soloftp kernel: [152151.773033] [<ffffffff8025695d>] irq_exit+0x8d/0xa0
Oct 10 01:07:48 soloftp kernel: [152151.773036] [<ffffffff802152c5>] do_IRQ+0xc5/0x110
Oct 10 01:07:48 soloftp kernel: [152151.773039] [<ffffffff80212bf3>] ret_from_intr+0x0/0x29
Oct 10 01:07:48 soloftp kernel: [152151.773040] <EOI> [<ffffffff8021a95a>] ? mwait_idle+0x4a/0x50
Oct 10 01:0...

[swmike (ubuntu-s-plass)]

I just got this on a newer kernel:

[377422.270379] swapper: page allocation failure. order:0, mode:0x4020
[377422.270384] Pid: 0, comm: swapper Not tainted 2.6.28-15-generic #52-Ubuntu
[377422.270387] Call Trace:
[377422.270389] <IRQ> [<ffffffff802b6e1e>] __alloc_pages_internal+0x3ee/0x500
[377422.270401] [<ffffffff802dfe58>] alloc_slab_page+0x28/0x30
[377422.270405] [<ffffffff802e0f1a>] new_slab+0x5a/0x210
[377422.270408] [<ffffffff802e2538>] __slab_alloc+0x188/0x290
[377422.270413] [<ffffffff8024782c>] ? find_busiest_group+0x1dc/0x9a0
[377422.270418] [<ffffffff805aa17f>] ? __netdev_alloc_skb+0x1f/0x40
[377422.270422] [<ffffffff802e3447>] __kmalloc_track_caller+0xd7/0x110
[377422.270425] [<ffffffff805aa17f>] ? __netdev_alloc_skb+0x1f/0x40
[377422.270429] [<ffffffff805a9e1e>] __alloc_skb+0x6e/0x150
[377422.270432] [<ffffffff805aa17f>] __netdev_alloc_skb+0x1f/0x40
[377422.270447] [<ffffffffa00ad780>] sky2_rx_alloc+0x80/0x140 [sky2]
[377422.270454] [<ffffffffa00b09a9>] receive_new+0x29/0x160 [sky2]
[377422.270460] [<ffffffffa00b0c23>] sky2_receive+0x143/0x280 [sky2]
[377422.270466] [<ffffffffa00b220d>] sky2_status_intr+0x17d/0x5a0 [sky2]
[377422.270473] [<ffffffffa00b2697>] sky2_poll+0x67/0x160 [sky2]
[377422.270477] [<ffffffff805b4024>] net_rx_action+0x104/0x240
[377422.270480] [<ffffffff80256c4c>] __do_softirq+0x9c/0x170
[377422.270484] [<ffffffff80213d8c>] call_softirq+0x1c/0x30
[377422.270488] [<ffffffff80214ffd>] do_softirq+0x5d/0xa0
[377422.270491] [<ffffffff802569cd>] irq_exit+0x8d/0xa0
[377422.270494] [<ffffffff802152c5>] do_IRQ+0xc5/0x110
[377422.270497] [<ffffffff80212bf3>] ret_from_intr+0x0/0x29
[377422.270499] <EOI> [<ffffffff8021a95a>] ? mwait_idle+0x4a/0x50
[377422.270507] [<ffffffff80210dd2>] ? enter_idle+0x22/0x30
[377422.270511] [<ffffffff80210e85>] ? cpu_idle+0x65/0xc0
[377422.270515] [<ffffffff806878cc>] ? rest_init+0x5c/0x70
[377422.270517] Mem-Info:
[377422.270519] DMA per-cpu:
[377422.270521] CPU 0: hi: 0, btch: 1 usd: 0
[377422.270523] CPU 1: hi: 0, btch: 1 usd: 0
[377422.270525] DMA32 per-cpu:
[377422.270527] CPU 0: hi: 186, btch: 31 usd: 175
[377422.270530] CPU 1: hi: 186, btch: 31 usd: 181
[377422.270531] Normal per-cpu:
[377422.270533] CPU 0: hi: 186, btch: 31 usd: 170
[377422.270535] CPU 1: hi: 186, btch: 31 usd: 178
[377422.270539] Active_anon:113789 active_file:35082 inactive_anon:20522
[377422.270540] inactive_file:1761059 unevictable:7 dirty:523227 writeback:2505 unstable:0
[377422.270542] free:7779 slab:66831 mapped:45316 pagetables:5195 bounce:0
[377422.270546] DMA free:6700kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:5560kB pages_scanned:0 all_unreclaimable? yes
[377422.270549] lowmem_reserve[]: 0 2999 8049 8049
[377422.270556] DMA32 free:21788kB min:4276kB low:5344kB high:6412kB active_anon:25384kB inactive_anon:8872kB active_file:41816kB inactive_file:2592776kB unevictable:0kB present:3071712kB pages_scanned:0 all_unreclaimable? no
[377422.270559] lowmem_reserve[]: 0 0 5050 5050
[377422.270565] Normal free:2628kB min:7200kB low:9000kB high:10800kB active_anon:42977...

[swmike (ubuntu-s-plass)]

#439761 seems to handle the same issue.

[Brad Figg (brad-figg)]

This bug was filed against a series that is no longer supported and so is being marked as Won't Fix. If this issue still exists in supported series, ple
ase file a new bug.