Page allocation failure on Pandaboard and Beagle XM

[Tobin Davis]

During the course of testing, I have been seeing a lot of page allocation failures on all omap4 images. Not sure what the cause is, but it doesn't appear to be related to any specific application. This also doesn't appear to affect the execution of the application.

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: linux-image-2.6.38-1207-omap4 2.6.38-1207.10
ProcVersionSignature: User Name 2.6.38-1207.10-omap4 2.6.38
Uname: Linux 2.6.38-1207-omap4 armv7l
Architecture: armel
Date: Wed Mar 30 16:59:55 2011
ProcEnviron:
 LANGUAGE=
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: linux-ti-omap4
UpgradeStatus: No upgrade log present (probably fresh install)

[Ricardo Salveti (rsalveti)]

Can you check if this still happens after adding "vm.min_free_kbytes = 8192" to your /etc/sysctl.conf?

From the logs it seems that the usb hub is trying to allocate big chucks of memory. That could be explained by using the rootfs on a usb disk or transferring a lot of data over ethernet.

[Paolo Pisati (p-pisati)]

with this snippet i can reliably reproduce the problem:

-------------8<-------------8<-------------8<-------------8<-------------

PKG="kubuntu-desktop openoffice.org fakeroot build-essential kexec-tools kernel-wedge libncurses5 libncurses5-dev libelf-dev asciidoc binutils-dev gimp chromium texlive-latex-recommended"
while [ 1 ]; do
    sudo rm /var/cache/apt/archives/*
    sudo aptitude install -dy $PKG
done

-------------8<-------------8<-------------8<-------------8<-------------

however adding "vm.min_free_kbytes = 8192" to sysctl.conf seems to fix the issue.

[Paolo Pisati (p-pisati)]

it seems i spoke too early:

natty: http://pastebin.ubuntu.com/591172/
maverick: http://pastebin.ubuntu.com/591194/

both while doing network activity and with vm.min_free_kbytes = 8192

Marcin reported another trace that looks identical: https://bugs.launchpad.net/ubuntu/+source/linux-ti-omap4/+bug/690370/comments/28

He got his while doing a kernel compilation on a usb hdd.

IMO this and lp690370 share the same root problem.

[Dr. David Alan Gilbert (davidgil-uk)]

The work around I've been using is

      Create /etc/modprobe.d/smscnonturbo.conf with the following contents

options smsc95xx turbo_mode=N

then reboot to get that option.

This probably does nasty things to performance but it gets rid of those errors reliably for me.

Dave

[Ricardo Salveti (rsalveti)]

Yes, this is directly related with the turbo_mode and the amount of memory needed when doing a RX transaction.

The two workarounds we have are:
 * Disable turbo_mode at the driver but affecting performance directly
 * Increase the amount of minimum free system memory in the kernel by writing the value at /etc/sysctl.conf

For the second option 8192 seems to fix most of the cases, as I've being building kernel the whole weekend and got only a few allocation errors. Setting it to 12288 seems to make it more than enough, as I didn't get any allocation error while using it.

As Beagle doesn't have a lot of memory already, I'd preferably go with 8192, that can fix most of the cases and still without decreasing ethernet performance.

This workaround could then be set by jasper.

[Ricardo Salveti (rsalveti)]

And remember that this workaround should be set for both OMAP 3 and OMAP 4, as both Beagle XM and Panda uses the same SMSC device/driver.

[Ricardo Salveti (rsalveti)]

Merge proposal for jasper including the sysctl workaround with 8192 kbytes:
 * https://code.launchpad.net/~rsalveti/jasper-initramfs/746137/+merge/57104

[Ricardo Salveti (rsalveti)]

Also, this workaround is currently being used by our Beagle XM builders, and they seem to be running quite well now. Lamont can give more details if needed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package jasper-initramfs - 0.54

---------------
jasper-initramfs (0.54) natty; urgency=low

  * Workaround to increase kernel min_free_kbytes to avoid page allocation
    failures (LP: #746137)
 -- Ricardo Salveti de Araujo <email address hidden> Mon, 11 Apr 2011 00:31:33 -0300

[Bryan Wu (cooloney)]

With the workaround in Jasper, I didn't see any page allocation failure errors on Panda with 1208.12 kernel.

-Bryan

[Matt Zimmerman (mdz)]

With 2.6.38-1209-omap4 and the default min_free_kbytes=8192, I still see page allocation failures when there's a lot of ethernet traffic. I've increased it to 12288 to see if that makes a difference.

[Matt Zimmerman (mdz)]

With 12288, I still see both page allocation failures and "eth0: kevent 2 may have been dropped" errors.

[Oliver Grawert (ogra)]

since we now dropped jasper-initramfs from quantal on we dont have a place to put the sysctl.d hack anymore which causes this bug to show up again in quantal images but along that we got a hint to replace GFP_ATOMIC by GFP_KERNEL in the relevant kmallocs which worked for the rt2x00 wifi driver in the ac100 armhf kernel. re-opening against linux-ti-omap4 for quantal ...

[Oliver Grawert (ogra)]

the same issue is seen on omap3 beagleboard images, so also opening against the linux package.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 746137

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Adam Conrad (adconrad)]

bug #992786 implies that this needs to be higher for sane performance. Testing here with 32768 works well.

I'm not sure having this as an installer hack is sane, though I'm trying to sort out where it's better put. procps owns that directory, but having procps have subarch specific magic is a bit icky.

[Oliver Grawert (ogra)]

well, how about testing the possible kernel fix mentioned above instead of hacking around the issue in userspace ?

[Adam Conrad (adconrad)]

I'm all for kernel fixes, if we can sort out a decent testing strategy to make sure it (a) fixes the bug, and (b) doesn't cause a regression. Would make our buildds pretty happy if we backported it to precise too.

[Oliver Grawert (ogra)]

jasper carries the fix for this since natty

[Leann Ogasawara (leannogasawara)]

Removing the rls-q-incoming since this bug has been nominated and milestoned to stay on the radar.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Paolo Pisati (p-pisati)]

in Q/omap4 we enforced vm.min_free_kbytes=32K at kernel level, while in P/omap4 we had the installer tune that value for us (and put a file in /etc/sysctl.d after the installation): i've been compiling the last two days with no problem on my panda so far, closing here.

[Leann Ogasawara (leannogasawara)]

Closing the linux tasks as this was against linux-ti-omap4.