Frequent swapping causes system to hang

Reported by Pete Goodall on 2010-06-16
32
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Linux
Fix Released
High
linux (Ubuntu)
Medium
Unassigned

Bug Description

Periodically I notice my system slows to a near stand still, and the hard drive light is constantly going. This seems to be a massive amount of disk i/o and it lasts for a long time (lets say 30 mins to put a number on it). I installed and ran iotop (`iotop -a`) and it seems to point to jbd2. From what I can see jbd2 is related to ext4 journaling, but I cannot figure out how to kill this operation. It might even be a red herring because I have also stopped the disk activity by kill either chromium or firefox. I need to understand what else I can do to troubleshoot this.

$ lsb_release -rd
Description: Ubuntu maverick (development branch)
Release: 10.10

Up-to-date as of 16th June 2010.
---
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.23.
AplayDevices:
 **** List of PLAYBACK Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: ALC272X Analog [ALC272X Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
Architecture: i386
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: ALC272X Analog [ALC272X Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: pgoodall 1372 F.... pulseaudio
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0x56440000 irq 44'
   Mixer name : 'Realtek ALC272X'
   Components : 'HDA:10ec0272,1025022c,00100001'
   Controls : 14
   Simple ctrls : 8
DistroRelease: Ubuntu 10.10
Frequency: Once a day.
HibernationDevice: RESUME=UUID=145f27a9-859a-4987-8132-ac878c832747
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Alpha i386 (20100602.2)
MachineType: Acer AO531h
Package: linux (not installed)
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.35-6-generic root=UUID=11f96f8b-5e04-4e20-a201-0fa5d0fc07fa ro quiet splash
ProcEnviron:
 PATH=(custom, user)
 LANG=en_GB.utf8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.35-6.9-generic 2.6.35-rc3
Regression: Yes
RelatedPackageVersions: linux-firmware 1.37
Reproducible: No
Tags: maverick ubuntu-une kconfig regression-potential needs-upstream-testing
Uname: Linux 2.6.35-6-generic i686
UserGroups: adm admin cdrom dialout lpadmin plugdev sambashare
dmi.bios.date: 12/22/2009
dmi.bios.vendor: Acer
dmi.bios.version: v0.3304
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.vendor: Acer
dmi.board.version: Base Board Version
dmi.chassis.type: 1
dmi.chassis.vendor: Chassis Manufacturer
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAcer:bvrv0.3304:bd12/22/2009:svnAcer:pnAO531h:pvr1:rvnAcer:rn:rvrBaseBoardVersion:cvnChassisManufacturer:ct1:cvrChassisVersion:
dmi.product.name: AO531h
dmi.product.version: 1
dmi.sys.vendor: Acer

Pete Goodall (pgoodall) wrote :

I think this is related to swap, actually. I have decread swappiness to 10, and will see if I encounter this again. Not sure what the next steps are if that works.

Pete Goodall (pgoodall) wrote :

Apparently reducing the swappiness didn't help. My system is currently hopelessly locked (and I'm writing this on another device on my desk). I'm ssh'ed into my netbook and running `iotop -a`. Here are the top two items:

    1 be/4 root 10.34 M 0.00 B ?unavailable? init
   33 be/4 root 16.00 K 340.00 K ?unavailable? [kswapd0]

The init read numbers keep going up (currently 10.86 MB) and the kswapd write numbers are going up as well.

Pete Goodall (pgoodall) wrote :

To add to the last comment, kswapd has just leaped up to over 7 MB written and rising fast! this is in a span of a couple of minutes.

affects: ubuntu → linux (Ubuntu)
Pete Goodall (pgoodall) wrote :

Ok, from what I can tell the system is swapping way too easily. If I'm running Chromium + XChat I'm fine. As soon as I open OpenOffice or Evolution or Rhythmbox that seems to push the memory over the edge and the system starts swapping. If I can manage to get to a terminal in time and kill the last application I started my system will return to normal. If I don't, I might as well just hard power off. I have set the swappiness to 0 in /etc/sysctl.conf.

Afaict there is no one program that is causing the system to swap. It just seems that Ubuntu is over sensitive. There was no problem w/ this workload with Lucid, so I don't think I'm overstressing the system.

Jeremy Foshee (jeremyfoshee) wrote :

Hi Pete,

Please be sure to confirm this issue exists with the latest development release of Ubuntu. ISO CD images are available from http://cdimage.ubuntu.com/daily/current/ . If the issue remains, please run the following command from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux 595047

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

    [This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

tags: added: needs-kernel-logs
tags: added: needs-upstream-testing
tags: added: kj-triage
Changed in linux (Ubuntu):
status: New → Incomplete
Pete Goodall (pgoodall) wrote :

Per some advice I received on #ubuntu-kernel I'm attaching some debugging output. Here is a list of the files:

vmstat-log-7-July-2010.txt: Output of `vmstat 1 60` once the system started swapping
free.txt: Output of `free -m` a minute later
free-2.txt: Output of `free -m` about 10 seconds after I ran free the first time
top-ouput.txt: Output of `top -b -n 1`

In case it isn't reflected in the top output, I'm running Chromium with four (?) tabs open (no Flash site or anything like that), Xchat connected to two servers and with about seven channels open, Evince viewing a simple pdf document, a Nautilus window, one gnome-terminal window and Banshee playing an mp3. Again, this is a normal load for me that didn't have a problem with previous versions of Ubuntu on the same device.

Pete Goodall (pgoodall) wrote :
Pete Goodall (pgoodall) wrote :
Pete Goodall (pgoodall) wrote :
Pete Goodall (pgoodall) wrote :
Pete Goodall (pgoodall) wrote :

I made the last comment before I read Jeremy's comments (should have refreshed the bug report). Anyway, I'll attach the output he is looking for and be sure to test the upstream kernel as well.

apport information

tags: added: apport-collected
description: updated

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

I have reproduced the bug using the mainline kernel. I installed linux-image-2.6.35-999-generic from http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/current/, and rebooted running that kernel. I was running Chromium, Xchat and Banshee w/ no problem, so I decided to try to stress the system a bit more. I started Tomboy and opened one of my notes with no problem. I opened Nautilus and the system started swapping. I tried to get a vmstat by ssh'ing into the system, but by the time I was able to login the swapping had subsided. It started again and I ran vmstat to collect some stats. Don't know if this is useful, but will attach it.

tags: removed: needs-upstream-testing
Pete Goodall (pgoodall) wrote :

I noticed there is a 'needs kernel logs' tag. Is there something else you need attached? If so, do you need it attached running both the current maverick kernel and the mainline kernel?

Pete Goodall (pgoodall) wrote :

Marking as 'New' since I think I have supplied all the required information.

Changed in linux (Ubuntu):
status: Incomplete → New
Pete Goodall (pgoodall) wrote :

In desperation I tried re-installing to see if I just had something installed with a serious memory leak. Unfortunately, this has not improved the situation. My device is next to useless until this is resolved.

Pete Goodall (pgoodall) on 2010-08-09
summary: - massive i/o renders the system unusable
+ Frequent swapping causes system to hang
Changed in linux (Ubuntu):
assignee: nobody → Jeremy Foshee (jeremyfoshee)
tags: added: kernel-core kernel-needs-review
removed: needs-kernel-logs
Pete Goodall (pgoodall) wrote :

Apparently there are some patches for what appears to be this issue. Linked from the upstream kernel bug report: http://www.phoronix.com/scan.php?page=news_item&px=ODQ3Mw

Changed in linux (Ubuntu):
assignee: Jeremy Foshee (jeremyfoshee) → nobody
Brad Figg (brad-figg) on 2010-12-03
tags: added: acpi-namespace-lookup
tags: added: acpi-parse-exec-fail
Changed in linux:
status: Unknown → Confirmed
Changed in linux:
importance: Unknown → High
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux (Ubuntu):
status: New → Confirmed
Changed in linux:
status: Confirmed → Fix Released

Pete Goodall, thank you for reporting this and helping make Ubuntu better. Maverick reached EOL on April 10, 2012.
Please see this document for currently supported Ubuntu releases:
https://wiki.ubuntu.com/Releases

We were wondering if this is still an issue in a supported release? If so, could you please test for this with the latest development release of Ubuntu? ISO CD images are available from http://cdimage.ubuntu.com/releases/ .

If it remains an issue, could you please run the following command in the development release from a Terminal (Applications->Accessories->Terminal), as it will automatically gather and attach updated debug information to this report:

apport-collect -p linux <replace-with-bug-number>

Also, could you please test the latest upstream kernel available following https://wiki.ubuntu.com/KernelMainlineBuilds ? It will allow additional upstream developers to examine the issue. Please do not test the kernel in the daily folder, but the one all the way at the bottom. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested and remove the tag:
needs-upstream-testing

This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the text:
needs-upstream-testing

If this bug is fixed in the mainline kernel, please add the following tags:
kernel-fixed-upstream
kernel-fixed-upstream-VERSION-NUMBER

where VERSION-NUMBER is the version number of the kernel you tested.

If the mainline kernel does not fix this bug, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-VERSION-NUMBER

where VERSION-NUMBER is the version number of the kernel you tested.

If you are unable to test the mainline kernel, please comment as to why specifically you were unable to test it and add the following tags:
kernel-unable-to-test-upstream
kernel-unable-to-test-upstream-VERSION-NUMBER

where VERSION-NUMBER is the version number of the kernel you tested.

Please let us know your results. Thank you for your understanding.

Helpful Bug Reporting Tips:
https://help.ubuntu.com/community/ReportingBugs

tags: added: maverick
Changed in linux (Ubuntu):
importance: Undecided → Medium
status: Confirmed → Incomplete
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.