[Dell Latitude D420] The IOSCHED default of deadline combined with 4k page size causes poor performance

[Jon Skanes]

WORKAROUND: Set hugepage to use a 2M page size, and utilize kernel boot parameters:
levator=cfq transparent_hugepage=always

I have a Dell Latitude D-420 notebook which has behaved miserably since sometime after I upgraded to Kubuntu 12.10.

I would be the first to admit that it's an older machine, however, the drop in performance was far to great and too rapid to make sense. I went so far as to take the machine apart thinking it was full of dust, as it started running hot enough to run the fan at top speed. It was almost dust free.

It is configured with an Intel dual core U2550, 1.2GHz processor, 2GB of ram, and a 1.8" 4200RPM PATA ZIF HDD. I wondered if the weak link was the hard drive. I borrowed a 128GB SSD and there was almost no improvement.

Activity as minimal as opening Google Chrome with two or three simple tabs and leaving it to sit with Akonadi and Nepomuk, to run in the back ground was enough to make it impossible to watch a full screen MP4 video. This is a far cry from a few months ago when I could have a dozen Chrome windows open, Kontact running, watching a full screen MP4 with no loss in quality, all while downloading at a speed which saturated my DSL connection.

What was even stranger was it was only moderately swapping out.

From there, I decided to investigate. The load average was regularly around 3. User load was only responsible for roughly 20% of an almost constant 100% CPU load. Combining this with the fact that swap used wasn't all that high, I started to dig a little deeper.

Interestingly, both regular memory usage AND buffer allocation were constantly high, with buffer allocation around 300MB. Disk cache hovered around a paltey 100MB. The cache didn't change by increasing vm.swappiness to 100, either. On top of it all, the Intel iwl3945 wireless adapter was loosing packets and constantly having to renegotiate WPA.

I ran vmstat for a while. Context switching was occurring roughly 1500 per second along with about 3000 software interrupts per second. From here, I looked into IOSCHED. As it turns out, linux-image-3.5.0-26-generic is configured to use the deadline scheduler, by default, as shown in /boot/config-3.5.0-26-generic:

CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
CONFIG_DEFAULT_DEADLINE=y
# CONFIG_DEFAULT_CFQ is not set
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="deadline"

Switching to CFQ improved the situation immensely, however, I was still getting far more kernel time using up the CPU, and the context switching was far too high.

I happened to come across documentation on the transparent hugepage system while I was reading up on IOSCHED and it hit me. For some reason the kernel was probably bogged down in memory fragmentation and huge page tables due to the 4k page side and the pressures of deadline scheduling with a rather slow hard drive. The hugepage system seems to do a wonderful job. I have it set to use a 2M page size. The system, itself, uses almost know resources, with khugepaged clocking in an insignificant 5:00m of CPU time in roughly six hours.

My poor old computer is now running as fast or faster then before this issue occurred. Switching between intensive applications such as having tens of Javascript heavy tabs like Facebook open in Chrome, bittorrent maxed out at about 500kB/s while using only encrypted connections, and watching a full screen MP4 flawlessly, rarely saturates CPU usage. The CPU is no longer mostly all kernel and wait time, either. Context switching has settled to a reasonable 500 to 700 switches per second. Buffer allocations are rarely higher than 25MB. As a bonus, the memory allocated for page tables has dropped from around 80MB to 20MB. Best of all, my fan is now back on low, and I can use the notebook without the risk of burns!

I realize that an aggressive scheduling approach is probably the way to go with new and resource rich hardware. however, with lightly provisioned systems it seems to create the perfect storm. After all, this notebook is still quite a bit more powerful than a recent tablet. If there is a wish to see Ubuntu running on hardware such as that, something will have to be done.

It could be as easy as a small script to assess resources, perhaps with each kernel update. and configure a more suitable default for these lighter systems.

Cheers,
Jonathan Skanes
---
ApportVersion: 2.6.1-0ubuntu10
Architecture: i386
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: jon 2267 F.... pulseaudio
DistroRelease: Ubuntu 12.10
EcryptfsInUse: Yes
HibernationDevice: RESUME=UUID=d1980c70-8ee4-4f44-a4d3-fa1e12b2b123
InstallationDate: Installed on 2009-12-28 (1181 days ago)
InstallationMedia: Kubuntu 9.10 "Karmic Koala" - Release i386 (20091028.5)
MachineType: Dell Inc. Latitude D420
MarkForUpload: True
Package: linux (not installed)
PccardctlIdent:
 Socket 0:
   no product info available
PccardctlStatus:
 Socket 0:
   no card
ProcEnviron:
 LANGUAGE=en_CA
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_CA.UTF-8
 SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.5.0-26-generic root=UUID=60cdd746-3be0-4d48-a03f-45ba7381db4f ro elevator=cfq transparent_hugepage=always quiet
ProcVersionSignature: Ubuntu 3.5.0-26.42-generic 3.5.7.6
RelatedPackageVersions:
 linux-restricted-modules-3.5.0-26-generic N/A
 linux-backports-modules-3.5.0-26-generic N/A
 linux-firmware 1.95
Tags: quantal
Uname: Linux 3.5.0-26-generic i686
UpgradeStatus: Upgraded to quantal on 2012-10-29 (145 days ago)
UserGroups: adm admin cdrom dialout kismet lp lpadmin plugdev pulse-rt sage sambashare tilp vboxusers wireshark
dmi.bios.date: 02/02/2008
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A06
dmi.board.vendor: Dell Inc.
dmi.chassis.type: 8
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvrA06:bd02/02/2008:svnDellInc.:pnLatitudeD420:pvr:rvnDellInc.:rn:rvr:cvnDellInc.:ct8:cvr:
dmi.product.name: Latitude D420
dmi.sys.vendor: Dell Inc.

[Jon Skanes (jon-skanes)]

Oh, I almost forgot. I'm running linux-image-3.5.0-26-generic.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1159258

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Jon Skanes (jon-skanes)]

The idea that led me to try transparent_hugepage came from another piece of evidence I forgot to mention. I noticed that half of my RAM, 1GB, was consumed by anonymous page allocations. Transparent_hugepage, through khugepaged, defragments AND merges the 4k anonymous pages into hugepages. I figured that it would be far better to have a few hundred 2MB pages to reduce the overhead of context switches. It seems the savings of 60MB in reduced page tables, and its seemingly miraculous return to the living, is all the evidence needed. My system is now chugging along with roughly 400MB of hugepages, at any given time.

[Jon Skanes (jon-skanes)]

apport information

[Jon Skanes (jon-skanes)]

apport information

[Jon Skanes (jon-skanes)]

apport information

[Jon Skanes (jon-skanes)]

apport information

[Jon Skanes (jon-skanes)]

apport information

[Jon Skanes (jon-skanes)]

apport information

[Jon Skanes (jon-skanes)]

If you notice the kernel command line in my apport report, there really isn't much to do if something like this were to be by a script, as needed, through the deb packages.

[Jon Skanes (jon-skanes)]

One thing to keep in mind, though, is I noticed in the transparent_hugepage docs that the configuration should be provided at boot. There are some issues with khugepaged cleaning up tasks which are started before it's switched on.

[Jon Skanes (jon-skanes)]

I have done a little more poking around. I'm becoming even more convinced it's a kernel issue.

I installed 3.5.7-991-generic #201304060456 from the Kernel Team's mainline PPA. Along with this, I have continued to use anonymous_hugepage and the CFQ scheduler.

I nearly dropped the machine when I rebooted. I didn't accurately time the difference, but there was easily a 30%-40% speed-up in boot time and login to KDE 4.10.2. Combine this with my earlier use of anonymous_hugepages and the CFQ scheduler, and my performance has increased by at least 50%-60%.

The only other change which may be a consideration, since I installed the vanilla kernel, could be that I'm no longer using the backported wireless stack.

When I have a little time in the next couple of days, I'm going to try the mainline version of 3.5.0-26-generic from the PPA. Perhaps this may narrow things down.

Can anyone tell me if it might be worthwhile to go so far as to try a 13.04 kernel build? I'm not going to face a totally broken system, if I do, will I?

[Jon Skanes (jon-skanes)]

My reason to give the 13.04 builds a chance is I really don't want this to follow me past the upgrade. I'm more than happy to spend a little time testing.

[Jon Skanes (jon-skanes)]

Here's a discussion I was part of, on the Kubuntu Forums, with some ideas: http://www.kubuntuforums.net/showthread.php?61615-High-CPU-load-during-startup-and-sometimes-after