32-bit kernel HDD slow write speed

[Valdemar Lemche]

Summary

After upgrading my Ubuntu 32-bit from 12.04 to 14.04 the write performance is painfully slow (less than 4.6MB/s) of both my SSD drives. The read speed is still fine. I've tested with the latest version of System Rescue CD (4.2.0 at this time) to see if was a hardware problem. However the performance of my drives was even better better than using Ubuntu 12.04.

I've also tried to make a fresh installation of Ubuntu 14.04 on a spare SSD drive of mine to see if its because I upgraded from 12.04. But after waiting 4 hours for it to install, I finally lost patience, and stopped the installation.

I'm using the deadline scheduler, but it doesn't matter which one I use. I'm also using the lowlatency kernel, but using the generic kernel doesn't have any effect.

I've tested with mechanical hard drives, and the problem also occurs there.

I've also tried to update to the proposed updates, and updated to kernel 3.13.0-30.54, but problem still exists there.

The only think that helps is telling the kernel only to address 8GB RAM using the "mem=8GB" kernel option, but that is obviously not a solution when I need to utilize all the 32GB RAM that I have .

The reason that I use a 32-bit OS is because most of the applications I use on my laptop are propriety software that only exists in 32-bit versions.

Details

Kernel version
root@ubuntu:~# uname -a
Linux fk20563 3.13.0-29-lowlatency #53-Ubuntu SMP PREEMPT Wed Jun 4 21:20:42 UTC 2014 i686 i686 i686 GNU/Linux

List hardware
lshw > lshw.txt

Timings of devices (which is also 4x slower than normal)
root@ubuntu:~# hdparm -t /dev/sda && hdparm -t /dev/sda && hdparm -t /dev/sda
/dev/sda:
 Timing buffered disk reads: 470 MB in 3.01 seconds = 156.33 MB/sec

/dev/sda:
 Timing buffered disk reads: 530 MB in 3.03 seconds = 174.86 MB/sec

/dev/sda:
 Timing buffered disk reads: 500 MB in 3.01 seconds = 166.36 MB/sec

root@ubuntu:~# hdparm -t /dev/sdb && hdparm -t /dev/sdb && hdparm -t /dev/sdb
/dev/sdb:
 Timing buffered disk reads: 500 MB in 3.32 seconds = 150.61 MB/sec

/dev/sdb:
 Timing buffered disk reads: 508 MB in 3.03 seconds = 167.49 MB/sec

/dev/sdb:
 Timing buffered disk reads: 500 MB in 3.21 seconds = 155.81 MB/sec

SSD partition alignment check
root@ubuntu:~# fdisk -l

Disk /dev/sda: 256.1 GB, 256060514304 bytes
255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd162fc6d

Device Boot Start End Blocks Id System
/dev/sda1 2048 483332095 241665024 83 Linux
/dev/sda2 483332096 500117503 8392704 5 Extended
/dev/sda5 483334144 500117503 8391680 82 Linux swap / Solaris

Disk /dev/sdb: 480.1 GB, 480103981056 bytes
255 heads, 63 sectors/track, 58369 cylinders, total 937703088 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sdb1 2048 937703087 468850520 83 Linux
root@ubuntu:~# parted
GNU Parted 2.3
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) select /dev/sda
Using /dev/sda
(parted) align-check opt 1
1 aligned
(parted) select /dev/sdb
Using /dev/sdb
(parted) align-check opt 1
1 aligned
(parted) quit

Scheduler:
root@ubuntu:~# cat /sys/block/sd{a,b}/queue/scheduler
noop [deadline] cfq
noop [deadline] cfq

Mount options:
root@fk20563:~# mount | grep sda
/dev/sda1 on / type ext4 (rw,noatime,nodiratime,errors=remount-ro,discard)
root@fk20563:~# mount | grep sdb
/dev/sdb1 on /media/vmware type xfs (rw,noatime,nodiratime,logbsize=256k,discard)

FS read/write check of /dev/sda1 on Ubuntu
root@fk20563:~# dd if=/dev/zero of=/test.raw bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 229.091 s, 4.6 MB/s
root@fk20563:~# dd of=/dev/null if=/test.raw
2048000+0 records in
2048000+0 records out
1048576000 bytes (1.0 GB) copied, 2.58461 s, 406 MB/s

FS read/write check of /dev/sdb1 on Ubuntu
root@fk20563:~# dd if=/dev/zero of=/media/vmware/test.raw bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 229.395 s, 4.6 MB/s
root@fk20563:~# dd of=/dev/null if=/media/vmware/test.raw
2048000+0 records in
2048000+0 records out
1048576000 bytes (1.0 GB) copied, 2.75578 s, 381 MB/s

Read/write speed using SysResCd:
root@sysresccd /root % dd if=/dev/zero of=/mnt/backup/test.raw bs=1M
count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 1.8919 s, 554 MB/s
root@sysresccd /root % dd of=/dev/null if=/mnt/backup/test.raw
2048000+0 records in
2048000+0 records out
1048576000 bytes (1.0 GB) copied, 2.14372 s, 489 MB/s
root@sysresccd /root % dd if=/dev/zero of=/mnt/custom/test.raw bs=1M
count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 3.93656 s, 266 MB/s
root@sysresccd /root % dd of=/dev/null if=/mnt/custom/test.raw
2048000+0 records in
2048000+0 records out
1048576000 bytes (1.0 GB) copied, 1.91224 s, 548 MB/s
root@sysresccd /root %

[Valdemar Lemche (atterdag)]

Sorry this is linked to the wrong package - it should be linked to the kernel

[Valdemar Lemche (atterdag)]

I've tried with different mem= values. I can go all the up to mem=18GB, and keep a write speed of ~256MB/s using the command "dd if=/dev/zero of=/test.raw bs=1M count=1000", but 19GB the write speeds starts to drop:
 2GB = ~256MB/s
[...]
18GB = ~256MB/s
19GB = ~232MB/s
20GB = ~170MB/s
21GB = ~4.6MB/s
[...]
32GB = ~4.6MB/s

[Phil Evans (pbil)]

I have had a similar problem recently, although not with Ubuntu. When the speed starts to go very slow, try doing:

echo 1 > /proc/sys/vm/drop_caches

I think there may be a kernel bug related to buffers which causes this slowdown when the buffers are full.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[oiiio (oiiio)]

for a work around see post by Volker Siegel Aug 3 '14 at 22:40 in
http://unix.stackexchange.com/questions/17936/setting-proc-sys-vm-drop-caches-to-clear-cache :

$ sync
$ echo 3 | sudo tee /proc/sys/vm/drop_caches

brings HDD write speed up again. I'm running 16.04 LTS; 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:15 UTC 2016 i686 i686 i686 GNU/Linux. (I observed the same write speed slow down in 14.04 LTS kernel.)

[David Flater (f-dave-7)]

I ran into the same problem under Slackware. What fixed it for me was changing the user/kernel memory split from 3G/1G to 2G/2G. Details here: http://flaterco.com/kb/PAE_slowdown.html

[Nick (q-nick)]

This bug still exists in even 4.4.0-xx x86 kernels, but the slow HDD writes disappears after I changed to amd64 kernel of the same versions even the userland is still 32-bit.

I did like as below.

sudo dpkg --add-architecture amd64
sudo apt-get update
sudo apt-get install linux-image-generic:amd64

The virtualbox and nvidia modules compiled by DKMS become unusable though.

[Alkis Georgopoulos (alkisg)]

I originally reported this issue in https://bugs.launchpad.net/ubuntu/+source/linux-hwe/+bug/1698118.
Just now I found this bug report so I'm marking mine as duplicate.

In the meantime, I had reported it upstream in the kernel in:
https://bugzilla.kernel.org/show_bug.cgi?id=196157

Thanks to the author of http://flaterco.com/kb/PAE_slowdown.html for the very nice overview of the issue and of the possible workarounds.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Norbert (nrbrtx)]

Got this bug on Ubuntu 16.04 LTS server with 32-bit linux-image-4.4.0-92-generic.
My system has 32GB of RAM.

With USB 3.0 SanDisk Extreme Flash I got:
$ dd if=/dev/zero of=1G bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1,1 GB, 1,0 GiB) copied, 171,214 s, 6,3 MB/s

It's terrible slow!

Note: `sudo -w vm.highmem_is_dirtyable=1` helps on 32 bit system.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.13 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.13-rc6

[insaner (insaner)]

I can confirm the same problem for me (however the system in question is actually Fedora 27, but no other online source provided a solution except for this bug report):

kernel: 4.10.10 SMP PREEMPT i686 (32bit)
compiled with gcc_7.3.1

started with write speeds at around 180 MB/s and within minutes would slow down and stabilize at 3 MB/s

# echo 3 > /proc/sys/vm/drop_caches

did not work, but

# sysctl -w vm.highmem_is_dirtyable=1

worked like a charm!

I will try to see if I can test some of the other kernels as mentioned above (64bit, versions above 4.13) and see if those manifest the same issues.

Thanks Norbert (nrbrtx)!