IO limit 1.2 MB/s for 32bit ubuntu (PAE, 32 Gb RAM)

[Ilya Murav'jov]

I recently installed a new motherboard Asus M5A99FX PRO R2.0 and found out that IO speed is very low; i.e. the dd command output:
dd if=/dev/zero of=/tmp/output bs=8k count=1000k
^C13885+0 records in
13885+0 records out
113745920 bytes (114 MB) copied, 99.6639 s, 1.1 MB/s

This issue is reproducing with my new SSD (with ext4), my USB drives and my hard disk (with ext4) from Ubuntu i386, 13.04 and 13.10. Such a speed is not ok, awfully slow. When I run two or more the same dd commands simultaneously, I have 1,2 Mb/s for every process.

WORKAROUNDs:
+ turn off ACPI services, see more at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1264707/comments/18
+ Use the amd64 version
+ Use CONFIG_HIGHMEM4G=y
+ CONFIG_VMSPLIT_1G=y

ProblemType: Bug
DistroRelease: Ubuntu 13.10
Package: linux-image-3.11.0-12-generic 3.11.0-12.19
ProcVersionSignature: Ubuntu 3.11.0-12.19-generic 3.11.3
Uname: Linux 3.11.0-12-generic i686
ApportVersion: 2.12.5-0ubuntu2
Architecture: i386
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC2: ubuntu 4507 F.... pulseaudio
 /dev/snd/controlC1: ubuntu 4507 F.... pulseaudio
 /dev/snd/controlC0: ubuntu 4507 F.... pulseaudio
CasperVersion: 1.336ubuntu1
Date: Sat Dec 28 15:56:33 2013
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.
LiveMediaBuild: Ubuntu 13.10 "Saucy Salamander" - Release i386 (20131016.1)
MachineType: To be filled by O.E.M. To be filled by O.E.M.
MarkForUpload: True
ProcEnviron:
 LANGUAGE=en_US
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 radeondrmfb
ProcKernelCmdLine: noprompt cdrom-detect/try-usb=true persistent file=/cdrom/preseed/hostname.seed boot=casper initrd=/casper/initrd.lz quiet splash -- maybe-ubiquity
PulseList:
 Error: command ['pacmd', 'list'] failed with exit code 1: Home directory not accessible: Permission denied
 No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-3.11.0-12-generic N/A
 linux-backports-modules-3.11.0-12-generic N/A
 linux-firmware 1.116
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/10/2013
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1708
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: M5A99FX PRO R2.0
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1708:bd04/10/2013:svnTobefilledbyO.E.M.:pnTobefilledbyO.E.M.:pvrTobefilledbyO.E.M.:rvnASUSTeKCOMPUTERINC.:rnM5A99FXPROR2.0:rvrRev1.xx:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: To be filled by O.E.M.
dmi.product.version: To be filled by O.E.M.
dmi.sys.vendor: To be filled by O.E.M.

[Brad Figg (brad-figg)]

This change was made by a bot.

[penalvch (penalvch)]

Ilya Murav'jov, thank you for updating your BIOS. Could you please confirm this issue exists with the latest development release of Ubuntu? ISO images are available from http://cdimage.ubuntu.com/daily-live/current/ . If the issue remains, please just make a comment to this.

If reproducible, could you also please test the latest upstream kernel available (not the daily folder) following https://wiki.ubuntu.com/KernelMainlineBuilds ? It will allow additional upstream developers to examine the issue. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this bug is fixed in the mainline kernel, please add the following tags:
kernel-fixed-upstream
kernel-fixed-upstream-VERSION-NUMBER

where VERSION-NUMBER is the version number of the kernel you tested. For example:
kernel-fixed-upstream-v3.13-rc7

This can be done by clicking on the yellow circle with a black pencil icon next to the word Tags located at the bottom of the bug description. As well, please remove the tag:
needs-upstream-testing

If the mainline kernel does not fix this bug, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-VERSION-NUMBER

As well, please remove the tag:
needs-upstream-testing

Once testing of the upstream kernel is complete, please mark this bug's Status as Confirmed. Please let us know your results. Thank you for your understanding.

[Ilya Murav'jov (muravjov-il)]

Christopher,
Yes, the issue exists with the latest development release of Ubuntu (Ubuntu 14.04 as of 12.01.14).

[Ilya Murav'jov (muravjov-il)]

uname -a:

Linux ubuntu 3.13.0-1-generic #16-Ubuntu SMP Tue Jan 7 19:47:28 UTC 2014 i686 athlon i686 GNU/Linux

[Ilya Murav'jov (muravjov-il)]

I've run the latest upstream kernel, http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc7-trusty/ , on my Ubuntu 13.04 and the issue still exists.

uname -a:
Linux ilhome 3.13.0-031300rc7-generic #201401041835 SMP Sat Jan 4 23:46:54 UTC 2014 i686 athlon i686 GNU/Linux

[penalvch (penalvch)]

Ilya Murav'jov, thank you for performing the requested tests. Just to confirm, this problem is reproducible with the i386 iso, and upstream kernel, but not the amd64 iso and upstream kernel correct?

[Ilya Murav'jov (muravjov-il)]

Christopher,
Yes, this problem is reproducible with all i386 variants: upstream kernel, ubuntu kernels (13.04, 13.10, daily-live=); and I tried amd64 iso (13.10) and it is ok.

[penalvch (penalvch)]

Ilya Murav'jov, anything stopping you from just using the amd64 variant going forward?

[Ilya Murav'jov (muravjov-il)]

Christopher,
Yes, it is a "reason" to go for 64bit; I am trying to go to future 14.04.
But in general my linux experience shows it is better to fix a bug rather than to upgrade. :)

[Ilya Murav'jov (muravjov-il)]

I've tested Arch live iso and there is no problem with IO, both i386 and amd64, uname -a:

Linux archiso 3.12.6-1-ARCH #1 SMP PREEMPT Fri Dec 20 19:54:53 CET 2013 i686 GNU/Linux
Linux archiso 3.12.6-1-ARCH #1 SMP PREEMPT Fri Dec 20 19:39:00 CET 2013 x86_64 GNU/Linux

Thus this issue is a combination of kernel + Ubuntu userland settings for my system. :/
Does Ubuntu use cgroups for something like http://unix.stackexchange.com/questions/48138/how-to-throttle-per-process-i-o-to-a-max-limit ?

[Ilya Murav'jov (muravjov-il)]

I've located the bug more closely: after installing ubuntu-minimal for 13.04, i386 system from scratch and booting into it I found no issue with IO speed. (hurray!)
Then I "apt-get install ubuntu-desktop" and the issue appeared again. After purge'ing and install'ing packages from 'ubuntu-desktop' - 'ubuntu-minimal' set I found the guilty: lightdm package.

So:
- if I boot without starting lightgm service (GUI) then everything is ok (see the process list `ps aux` = 'good_ps.lst' file)
- then I start lightdm (initctl start lightdm) and that dd command is that slow (see the process list 'bad_ps_lightdm.lst')
- then I stop lightdm (initctl stop lightdm) and that dd command is that slow still (see the process list 'bad_ps_after_lightdm.lst'); killing userspace processes doesn't help either (only reboot helps).

Any ideas?

[Ilya Murav'jov (muravjov-il)]

I've investigated a lot and found out that it is an ACPI issue, IMHO.

After more installing and purging packages I found out that two packages "breaks" the system: acpi-support and upower.
If I uninstall them then I have no IO issues.

The new workaround without uninstalling packages:
- disable /etc/init.d/acpi-support script: just remove executable flag,
     sudo chmod a-x /etc/init.d/acpi-support script
- disable UPower daemon:
     cd /usr/share/dbus-1/system-services
     sudo mv org.freedesktop.UPower.no_service org.freedesktop.UPower.no_service
- then reboot (because if you have the IO issue already then reboot helps only)

After that you don't use suspend to RAM or you have the IO issue after wakeup.

Just now I have 135 Mb/s for SATA SSD and 120 Mb/s for SATA HDD, in 13.04, i386. It is not perfect (under some amd64 Ubuntu I have 370 Mb/s for the same SSD while his max speed is 340 Mb/s) but much better.

Any ideas? I'm ready to test and hack linux. :)

[Ilya Murav'jov (muravjov-il)]

One more (maybe political) question: is amd64 support better than i386 for Ubuntu?
Because:
- I see some littel issues when using 64bit Linux'es for the desktop (the sound hissing in Skype etc)
- Skype, Steam and other commercial products are shipped for i386 only; so I still need to use i386 packages anyway; so why to duplicate stuff.

[Ilya Murav'jov (muravjov-il)]

I also tested with the following kernel parameters:
- acpi=off gives me 18Mb/s; it is better than 1 Mb/s but still bad
- pci=noacpi has no effect, 1 Mb/s.

[Ilya Murav'jov (muravjov-il)]

Similar bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/119730

[Ilya Murav'jov (muravjov-il)]

Hi again!

I've tested latest Fedora distro and it has no issue with IO. :(

Distro image: Fedora-Live-Desktop-i686-20-1.iso
$ uname -a
Linux localhost 3.11.10-301.fc20.i686 #1 SMP Thu Dec 5 14:21:31 UTC 2013 i686 i686 i386 GNU/Linux

I also checked that UPower daemon worked in Fedora as usual (and didn't cause this ACPI problem as in Ubuntu).

I'm going to find out what ACPI actions UPower daemon does; so I will be able to reproduce the problem explicitly.

[Phillip Susi (psusi)]

That is one strange problem.

[Ilya Murav'jov (muravjov-il)]

After investigating UPower sources I've triaged the problem to this:

"/usr/sbin/pm-powersave false" => "/usr/lib/pm-utils/power.d/laptop-mode false" => "echo 10 > /proc/sys/vm/dirty_ratio"

If I change /proc/sys/vm/dirty_ratio to any different value (more or less than previous value!) then I got my issue.

That is, at startup I have dirty_ratio equal to 20 and I should leave it unchanged (neither 19 nor 21 may be used) if I don't want to get 1Mb/s speed. :)

I've read a few articles about dirty_ratio' impact to IO; nevertheless I believe it is a bug. Any ideas?

Also I'll try another scenario tonight: I have 32Gb RAM now so I try to increase swap size from 8Gb to 32Gb, and then run "sysctl -w vm.swappiness=100" (maybe it helps to offload IO activity).

[Phillip Susi (psusi)]

So all you have to do is change /proc/sys/vm/dirty_ratio, even only slightly, up, or down, and it triggers this behavior? This just keeps getting weirder and weirder. If you put it back to 20, does it go away?

[Ilya Murav'jov (muravjov-il)]

Phillip,
Yes, exactly.
No, it doesn't go away; I don't know a way to get back normal IO speed without a reboot. :(

[Ilya Murav'jov (muravjov-il)]

Hi all,

After compiling kernels for a while I found out why the Fedora's kernel works for me: it is built without PAE support.
I've bisected Ubuntu' and Fedora's .config files and it turnes out that CONFIG_HIGHMEM4G=y "fixes" my issue ;) .

After that I rebooted my machine with default Ubuntu kernel (PAE) and 16Gb RAM, not 32GB, and it also "fixed" my issue ;)

After that I built a kernel with PAE, 32GB but with CONFIG_VMSPLIT_1G=y , and again, it really fixed my issue (but it is incompatible with binary drivers, see documentation; and my fglrx didn't work).

To wit: Linux kernel doesn't work well with 32bit, PAE and >=32 Gb RAM. (I even saw such statemets somewhere on the Net.)
Am I right? If so, I think Ubuntu documentation should state it explicitly.

[Ilya Murav'jov (muravjov-il)]

I've reproduced the issue with Thinkpad W520, 32GB RAM (Ubuntu Saucy, 480kb/s).

[hATrayflood (h-rayflood)]

I've reproduced the issue when 24GB RAM after ubuntu 12.10 and 12.04.2.

[hATrayflood (h-rayflood)]

avoid to this probrem, set kernel parameter "mem=16384M".

[Jah (thejar)]

I believe that this issue is a lot more common, but goes unnoticed.

Can reproduce in 14.04.1 LTS on multiple laptops/pc motherboards.

[Jah (thejar)]

setting kernel parameter "mem=16384M" did not help
 (was set via grub GRUB_CMDLINE_LINUX="mem=16384mb")

killing lightgm resolves the problem and speed jumps to 12mb/s . But once back in GUI, problem returns.

Tested using Netcat and iperf, LAN speeds peek out at 1.2MB/s

Effected:
PC1
Desotop, Ram Page 3 Extreme motherboard:
Description: Ubuntu 14.04.1 LTS
Linux ubuntu 3.13.0-34-generic #60-Ubuntu SMP Wed Aug 13 15:45:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
network
       description: Ethernet interface
       product: 82567V-2 Gigabit Network Connection
       vendor: Intel Corporation
       physical id: 19
       bus info: pci@0000:00:19.0
       logical name: eth1
       version: 00
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=2.3.2-k duplex=full firmware=1.8-5 ip=10.10.10.121 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:75 memory:f7de0000-f7dfffff memory:f7ddf000-f7ddffff ioport:9c00(size=32)

LAPTOP 1
Description: Ubuntu 12.04.4 LTS
Linux user 3.11.0-26-generic #45~precise1-Ubuntu SMP Tue Jul 15 04:04:35 UTC 2014 i686 i686 i386 GNU/Linux

network
       description: Ethernet interface
       product: RTL8101E/RTL8102E PCI Express Fast Ethernet controller
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:09:00.0
       logical name: eth0
       version: 05
       serial: 88:ae:1d:49:8e:f3
       size: 100Mbit/s
       capacity: 100Mbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=2.3LK-NAPI duplex=full firmware=rtl_nic/rtl8168e-2.fw ip=192.168.9.19 latency=0 link=yes multicast=yes port=MII speed=100Mbit/s
       resources: irq:43 ioport:2000(size=256) memory:f050c000-f050cfff memory:f0508000-f050bfff

+ several other laptops/computers.

Older laptops are not effected.

Ilya your post where my guiding stars! Thank you.

Also, I think issues like “Remote Desktop VNC very slow”
http://ubuntuforums.org/showthread.php?t=1752116&page=2
and many others could have been results of this bug and users not realising that LAN speed is the issue..

Issue remains unresolved for me.

[Jah (thejar)]

Intel(R) PRO/1000 Network Driver - 3.1.0.2-NAPI

Ethernet controller [0200]: Intel Corporation 82567V-2 Gigabit Network Connection [8086:10ce]
 Subsystem: ASUSTeK Computer Inc. Device [1043:82d5]
 Kernel driver in use: e1000e

Updating driver does not help.

[Ilya Murav'jov (muravjov-il)]

Jah,

You're welcome.

Your PC1 is under amd64 Linux so I think this issue is not your case.

[Jah (thejar)]

Ilya that exact 1.2mb/s trottle speed reading and lightgm connection,
maybe not the same game, but could be the same field

if you got a minute, look this over pls.
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1275161

[Ilya Murav'jov (muravjov-il)]

Jah,

You mean lightdm, don't you?
If your IO speed with local disk is slow too then I think slow network speed is just a consequence of a bug like this one.

[Jah (thejar)]

yeah, sorry, I ment lightdm.

killing gui/unity fixes the problem.

Local disk IO is not slow like in your cases, but I still think issues are related.

Will get to the buttom of this!

[Valentin Hilbig (muctino)]

Perhaps try: sysctl vm.highmem_is_dirtyable=1

Before: 1-3 MB/s write speed, one CPU (25%) wasted on waiting.
After: more than 200 MB/s write speed, 1-2% waiting.
Kernel: 32 Bit 4.4.0-45-generic Ubuntu, 32 GB RAM, SSD