SRU: Stop kswapd consuming 100% CPU when highest zone is small

Bug #808509 reported by David Nielsen
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Undecided
Unassigned
Natty
Undecided
Tim Gardner
Oneiric
Undecided
Unassigned

Bug Description

SuSEs Mel Gorman fixed yet another kswapd consuming 100% CPU issue. This one is most obvious on Sandybridge hardware but is expected to affect other setups as well. Affected releases should be Oneiric and Natty.

Original posting:
http://lkml.indiana.edu/hypermail/linux/kernel/1106.3/00161.html

Commits upstream in Linus´s tree:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=08951e545918c1594434d000d88a7793e2452a9b
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=d7868dae893c83c50c7824bc2bc75f93d114669f
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=da175d06b437093f93109ba9e5efbe44dfdf9409
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=215ddd6664ced067afca7eebd2d1eb83f064ff5a

While Mel has recommended these for -stable it does not appear that gregkh has picked them up for the 38.9 nor 39.4 releases yet.

Revision history for this message
Tim Gardner (timg-tpi) wrote :

gregkh has declined Mel's patchset for 2.6.38.y as it is no longer supported as a stable release.

affects: linux-meta (Ubuntu) → linux (Ubuntu)
Revision history for this message
Colin Ian King (colin-king) wrote :

@Tim, does this invalid these patches for a SRU then? Or can we still run with them?

Revision history for this message
Tim Gardner (timg-tpi) wrote :

Colin - We could still SRU theses for Natty, but I'd definitely need a reproducer. I assume you have a reliable method for demonstrating the problem?

Revision history for this message
David Nielsen (davidnielsen-deactivatedaccount) wrote :

From the posting: The reproduction case is almost always during
copying large files that kswapd pegs at 100% CPU until the file is
deleted or cache is dropped.

I believe I've seen this behaviour on earlier hardware than sandybridge, I can trigger it with good reliability using abocks image-writer to create a liveusb version of the GNOME3 isos (from gnome.org) however the hardware is question unfortunately died recently. Any large file copy operation should trigger it on sandybridge according to Mel though..

Revision history for this message
Tim Gardner (timg-tpi) wrote :

Colin - In your copious spare time perhaps you could try building this Natty master-next kernel and testing it on an SB:

The following changes since commit 006d932cf137dfcc156647dc08a0dda6e3e0f890:
  Tao Ma (1):
        ext4: init timer earlier to avoid a kernel panic in __save_error_info, CVE-2011-2493

are available in the git repository at:

  git://kernel.ubuntu.com/rtg/ubuntu-natty.git lp808509-kswapd

Mel Gorman (5):
      mm: vmscan: correct check for kswapd sleeping in sleeping_prematurely
      mm: vmscan: kswapd should not free an excessive number of pages when balancing small zones
      mm: vmscan: do not apply pressure to slab if we are not applying pressure to zone
      mm: vmscan: evaluate the watermarks against the correct classzone
      mm: vmscan: only read new_classzone_idx from pgdat when reclaiming successfully

 include/linux/swap.h | 9 ++++++
 mm/vmscan.c | 75 ++++++++++++++++++++++++++++++++------------------
 2 files changed, 57 insertions(+), 27 deletions(-)

Changed in linux (Ubuntu Oneiric):
status: New → Fix Released
Changed in linux (Ubuntu Natty):
assignee: nobody → Tim Gardner (timg-tpi)
status: New → In Progress
Revision history for this message
Colin Ian King (colin-king) wrote :

@Tim, Will do, however I will probably get around to this on Thursday as I'm off on sick leave until then. I have a good set of reproducer scripts + H/W that can faithfully trigger the bug.

Revision history for this message
Colin Ian King (colin-king) wrote :

@Tim, managed to get some overnight testing on this kernel, results are very good:

Did 1000 x test cycles, each copying ~795MB of data (~7 hours of soak testing), no lock-ups on a SNB laptop with 4GB memory. Also re-ran with 500 x test cycles with 2GB memory (as this *really* exercises the bug). My test script (which contains portions of the ubiquity copying script which originally tripped this bug) is attached. Testing - run script to copy data from one ext4 partition to another.

/dev/sda3, ext4, source, containing ~795MB of files from the Natty installer
/dev/sda4, ext4, destination

Summary: Exhaustive testing passed.

Andy Whitcroft (apw)
Changed in linux (Ubuntu Natty):
status: In Progress → Fix Committed
Revision history for this message
Herton R. Krzesinski (herton) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-natty' to 'verification-done-natty'.

If verification is not done by one week from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Also would be good to have a formal SRU justification here.

tags: added: verification-needed-natty
Revision history for this message
Colin Ian King (colin-king) wrote :

I've repeated the test scenario as described in comment #7 with 500 copy operations using the script (see comment #7 too) to give it nearly 4 hours of intensive file copying soak testing. Works fine, no lock-ups/hangs.

Marking it as verified.

tags: added: natty-verification-done
removed: verification-needed-natty
tags: added: verification-done-natty
removed: natty-verification-done
Revision history for this message
Tim Gardner (timg-tpi) wrote :

SRU Justification

Impact: Sandy Bridge platforms with a small normal zone can apparently lockup when copying from a fast I/O source to a slower I/O destination, e.g., disk to USB stick.

Patch Description: A number of upstream changes have made accommodation for how pages are allocated when the normal zone is exhausted.

Patch: See the list of commits in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/808509/comments/5

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (13.4 KiB)

This bug was fixed in the package linux - 2.6.38-11.48

---------------
linux (2.6.38-11.48) natty-proposed; urgency=low

  [Herton R. Krzesinski]

  * Release Tracking Bug
    - LP: #818175

  [ Upstream Kernel Changes ]

  * Revert "HID: magicmouse: ignore 'ivalid report id' while switching
    modes"
    - LP: #814250

linux (2.6.38-11.47) natty-proposed; urgency=low

  [Steve Conklin]

  * Release Tracking Bug
    - LP: #811180

  [ Keng-Yu Lin ]

  * SAUCE: Revert: "dell-laptop: Toggle the unsupported hardware
    killswitch"
    - LP: #775281

  [ Ming Lei ]

  * SAUCE: fix yama_ptracer_del lockdep warning
    - LP: #791019

  [ Stefan Bader ]

  * SAUCE: Re-enable RODATA for i386 virtual
    - LP: #809838

  [ Tim Gardner ]

  * [Config] Add grub-efi as a recommended bootloader for server and
    generic
    - LP: #800910
  * SAUCE: rtl8192se: Force a build for a 2.6/3.0 kernel
    - LP: #805494

  [ Upstream Kernel Changes ]

  * Revert "bridge: Forward reserved group addresses if !STP"
    - LP: #793702
  * Fix up ABI directory
  * bonding: Incorrect TX queue offset, CVE-2011-1581
    - LP: #792312
    - CVE-2011-1581
  * fs/partitions/efi.c: corrupted GUID partition tables can cause kernel
    oops
    - LP: #795418
    - CVE-2011-1577
  * usbnet/cdc_ncm: add missing .reset_resume hook
    - LP: #793892
  * ath5k: Disable fast channel switching by default
    - LP: #767192
  * mm: vmscan: correctly check if reclaimer should schedule during
    shrink_slab
    - LP: #755066
  * mm: vmscan: correct use of pgdat_balanced in sleeping_prematurely
    - LP: #755066
  * ALSA: hda - Use LPIB for ATI/AMD chipsets as default
    - LP: #741825
  * ALSA: hda - Enable snoop bit for AMD controllers
    - LP: #741825
  * ALSA: hda - Enable sync_write workaround for AMD generically
    - LP: #741825
  * cpuidle: menu: fixed wrapping timers at 4.294 seconds
    - LP: #774947
  * drm/i915: Fix gen6 (SNB) missed BLT ring interrupts.
    - LP: #761065
  * USB: ehci: remove structure packing from ehci_def
    - LP: #791552
  * drm/i915: disable PCH ports if needed when disabling a CRTC
    - LP: #791752
  * kmemleak: Do not return a pointer to an object that kmemleak did not
    get
    - LP: #793702
  * kmemleak: Initialise kmemleak after debug_objects_mem_init()
    - LP: #793702
  * Fix _OSC UUID in pcc-cpufreq
    - LP: #793702
  * CPU hotplug, re-create sysfs directory and symlinks
    - LP: #793702
  * Fix memory leak in cpufreq_stat
    - LP: #793702
  * net: recvmmsg: Strip MSG_WAITFORONE when calling recvmsg
    - LP: #793702
  * ftrace: Only update the function code on write to filter files
    - LP: #793702
  * qla2xxx: Fix hang during driver unload when vport is active.
    - LP: #793702
  * qla2xxx: Fix virtual port failing to login after chip reset.
    - LP: #793702
  * qla2xxx: Fix vport delete hang when logins are outstanding.
    - LP: #793702
  * powerpc/kdump64: Don't reference freed memory as pacas
    - LP: #793702
  * powerpc/kexec: Fix memory corruption from unallocated slaves
    - LP: #793702
  * x86, cpufeature: Fix cpuid leaf 7 feature detection
    - LP: #793702
  * ath9k_hw: do noise floor calibration only on required chain...

Changed in linux (Ubuntu Natty):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers