extremely bad dm-crypt latency

[Michael Zugelder]

The latency induced by dm-crypt seems to have increased at least five-fold with the Maverick 2.6.35 kernel, compared to the Lucid 2.6.32 kernel.
This makes a very fast SSD system feel like slower than with an HDD.
I did some benchmarks from my current Lucid install with stock kernel and Maverick kernel and was really surprised how slow it really got.

My test system is Ubuntu 10.04 Lucid Lynx x64 on an Intel Core i5 M520 (2.40GHz) Laptop with an Intel X25-V SSD, booting from encrypted root, created with 'cryptsetup luksFormat -c aes-xts-plain -s 256'. The 'aesni_intel' module is loaded and seems to work, since I get well over 250MB/s read/write with an encrypted ramdisk, using various cipher modes (e.g. aes-ecb-plain, aes-cbc-essiv:sha256, aes-xts-plain).
The only difference between the following benchmarks is the booted kernel.

2.6.32:
Linux vpcz1 2.6.32-25-generic #44-Ubuntu SMP Fri Sep 17 20:05:27 UTC 2010 x86_64 GNU/Linux

2.6.35:
Linux vpcz1 2.6.35-22-generic #33-Ubuntu SMP Mon Sep 20 08:11:39 UTC 2010 x86_64

I also just installed the Maverick RC, mounted my encrypted Lucid root and did some benchmarking, with exactly the same results as using Lucid with the Maverick 2.6.35 kernel.

[Michael Zugelder (michaelzugelder)]

Ignore the performance spikes, these are caused by empty space, which the firmware of the drive handles faster.
Comparing the encrypted disk (2.6.32) with the raw disk, there seems to be virtually no performance penalty.

[Michael Zugelder (michaelzugelder)]

Note the huge difference in the amount of context-switches.

[Michael Zugelder (michaelzugelder)]

Note that listing all files with find (exact commandline see below) is 34 times faster on 2.6.32 (367s vs 10.7s).

# echo 1 > /proc/sys/vm/drop_caches
# time find / > /dev/null

[Veronika Loitzenbauer (vero)]

I can confirm this with 2.6.32-27-generic (find / 10.5 s) and 2.6.33-02063301-generic (176.3 s).

[René Scheibe (rene-scheibe)]

I can confirm it too for 2.6.35-24-generic.

Is there any progress?

[Michael Zugelder (michaelzugelder)]

There is a bit progress, you can read the details at http://www.linux-archive.org/device-mapper-development/474248-problem-ssd-access-time-dm-crypt-way-too-high.html. There are currently two patches, which completely resolve the issue for me, but have their own problems and presumably won't go into mainline.

There is also a bugreport about similar issues on the kernel bugzilla (https://bugzilla.kernel.org/show_bug.cgi?id=17892), but since the issues persist with 2.6.32, it could be another problem.

[René Scheibe (rene-scheibe)]

Thanks Michael. I knew the mentioned posts.

I just tried the current Natty kernel 2.6.38-020638rc2-generic. Looks like it's solved there.

Seeker v2.0, 2007-01-15, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sda [152627MB], wait 30 seconds..............................
Results: 4543 seeks/second, 0.22 ms random access time

[Michael Zugelder (michaelzugelder)]

Seems like I should have been testing 2.6.38-rc2 better. I checked a few days ago, benchmarked and tought I had forgotten to revert the patch. Went back to 2.6.37, because standby on my machine is broken with in 2.6.38. Just compiled a pristine mainline 2.6.38-rc2 to be sure and it really seems fixed (4500 seeks/second -> 3700), the same values I get with the patched 2.6.37 or 2.6.32.