writing to ecryptfs partition on SSD drive is many times slower than writing to unencrypted partition on the same drive

[Daniel Kończyk]

Binary package hint: ecryptfs-utils

I've been using ecryptfs from the very introduction in Ubuntu. Recently I installed Maverick RC on a brand new SSD drive (120GB - OCZSSD2-2VTXE120G) with home encrypted with ecryptfs - as usual.

I felt something was wrong with the overall speed and I did some testing and it seems to me something is wrong with the performance of ecryptfs, here are some tests - the speed is quite similar to real life writing (I moved a lot of data to this partition).

The drive has been secure erased, partitions properly aligned, trim enabled in fstab

------------------------
test in encrypted ~

time dd if=/dev/zero of=filename bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 51.601 s, 19.8 MB/s

real 0m51.605s
user 0m0.280s
sys 0m49.400s
--------------------------
test in unencrypted /tmp (same hdd, different partition)

1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 3.6165 s, 283 MB/s

real 0m3.686s
user 0m0.190s
sys 0m2.910s
----------------------------

Clearly there is a 14 times speed difference. Please note, that I copied real-life data to this partition and speed never exceeded 33MB/s with both code repositories (lots of tiny files) and virtual machines (>2GB files)

For comparison, here is the same test done in Lucid on a 3-year old 5400RPM drive, same machine, 32-bit install.

---------------------
test in encrypted ~

time dd if=/dev/zero of=filename bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 89.7334 s, 11.4 MB/s

real 1m29.739s
user 0m0.248s
sys 0m58.100s
--------------------
test in unencrypted /tmp (same hdd, different partition)

time dd if=/dev/zero of=filename bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 47.7698 s, 21.4 MB/s

real 0m48.069s
user 0m0.240s
sys 0m4.444s
---------------------

So, my new SSD drive is 13 times faster on unencrypted partition and not even 2 times faster on encryptfs one.
I report it as bug, because it surely looks like one.

If you want me to do any other tests, I'll be happy to help

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: ecryptfs-utils 83-0ubuntu3
ProcVersionSignature: Ubuntu 2.6.35-22.33-generic 2.6.35.4
Uname: Linux 2.6.35-22-generic x86_64
Architecture: amd64
Date: Mon Oct 4 20:33:05 2010
EcryptfsInUse: Yes
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Release Candidate amd64 (20100928)
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: ecryptfs-utils

[Dustin Kirkland  (kirkland)]

Hey Tyler,

Can you take a look and comment on this?

[Heiko Wegeler (heiko-wegeler)]

the ssd has a sandforce controller with buildin compression. Sequential speeds for random data are ~200MB/s read, ~130MB/s write. 4k random data write with one thread is ~55MB/s, ~115MB/s with 64 threads. It looks like the encryption transforms the 0-data stream in a more or less random data stream.

[Daniel Kończyk (drmartens)]

That may be the case, hopefully Tyler will comment on that.

More speed tests of this drive can be found here:
http://www.storagereview.com/ocz_vertex_2_review_120gb

[Tyler Hicks (tyhicks)]

I've taken a cursory look at this and believe you're just hitting the performance limitations of eCryptfs. It is just much more apparent when using an SSD as compared to spinning media.

However, I've spotted something that looks a little fishy in ecryptfs_write_end(). We're calling ecryptfs_encrypt_page(), which encrypts the page and writes it to the lower filesystem. But we're calling ecryptfs_encrypt_page() again in ecryptfs_writepage(). I need to do some more work to see if we can remove the call to ecryptfs_encrypt_page() in ecryptfs_write_end() and just let ecryptfs_writepage() handle it. If so, the early testing of removing this call improves performance greatly, but I'm not going to post numbers until I do a little more digging.

[Daniel Kończyk (drmartens)]

Strange, I've always thought I'm limited by the disk performance...Althought my tests seemed to be curbed at a certain level, I hoped it was just a bug, not a limitation.
Is there any documentation for eCryptfs regarding its read/write performance limits? Or it's just typical for encrypted filesystems?

Thanks for looking into this issue.

[Tyler Hicks (tyhicks)]

> Strange, I've always thought I'm limited by the disk performance...
> Althought my tests seemed to be curbed at a certain level

Try eCryptfs on tmpfs to test the theory. I see very similar numbers to
eCryptfs on ext4 on an SSD.

> I hoped it was just a bug, not a limitation.

That's what I was getting at in comment #5, I think we're doing an
extra page encrypt and writing to the lower file system on every
buffered write. It ensures some cache consistency between the eCryptfs
page cache and the lower page cache, but I'm not sure if that is
actually helpful in the case of eCryptfs mounted on a local filesystem.

[Daniel Kończyk (drmartens)]

You're right, tmpfs with ecryptfs is as slow as my SSD when it comes to writing...

Hopefully it can be improved

[chifamba (rchifamba)]

Hi
I ran the same tests on a Lenovo T500

===========================================================
   encryptfs
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 58.1711 s, 17.6 MB/s

real 0m58.242s
user 0m0.150s
sys 0m57.850s
===========================================================

  non encryptfs
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 16.4165 s, 62.4 MB/s

real 0m16.463s
user 0m0.170s
sys 0m5.740s
===========================================================

for comparison.

[SabreWolfy (sabrewolfy)]

I know this bug is about SSD performance. I'm adding my comments about performance on magnetic disks here rather than filing another bug, as the issue seems to be the same.

I have encrypted my /home folder on a Maverick Meerkat Ubuntu (and later Xubuntu installation), fully updated.

Root (/) and /home share the same hard drive partition.

Disk performance is sometimes so slow that it interferes with my ability to use the machine.

Disk performance on the /home folder is dramatically different from /tmp:

$ time dd if=/dev/zero of=~/dummy bs=512 count=10240

2.4 MB.s

real 0m2.217s
user 0m0.028s
sys 0m2.176s

$ time dd if=/dev/zero of=/tmp/dummy bs=512 count=10240

42.6 MB.s

real 0m0.152s
user 0m0.012s
sys 0m0.136s

[Dustin Kirkland  (kirkland)]

Encryption ain't free.

Here are three really rough tests on my side:

To encrypted home:

time dd if=/dev/zero of=~/dummy bs=512 count=10240
10240+0 records in
10240+0 records out
5242880 bytes (5.2 MB) copied, 0.456936 s, 11.5 MB/s

real 0m0.459s
user 0m0.010s
sys 0m0.440s

To my SSD, not encrypted:

time dd if=/dev/zero of=/local/dummy bs=512 count=10240
10240+0 records in
10240+0 records out
5242880 bytes (5.2 MB) copied, 0.0222484 s, 236 MB/s

real 0m0.024s
user 0m0.000s
sys 0m0.020s

To shared memory:

time dd if=/dev/zero of=/tmp/dummy bs=512 count=10240
10240+0 records in
10240+0 records out
5242880 bytes (5.2 MB) copied, 0.0177176 s, 296 MB/s

real 0m0.020s
user 0m0.000s
sys 0m0.010s

There is some encryption overhead, but also, the ecryptfs data is
written synchronously to the disk, whereas these other tests are
cached (as I understand it).

[Tyler Hicks (tyhicks)]

On Fri Jan 28, 2011 at 03:18:38AM -0000, Dustin Kirkland <email address hidden> wrote:
> time dd if=/dev/zero of=~/dummy bs=512 count=10240

There are a couple issues here:

1. eCryptfs currently implements a write-through cache. This means that
   reads are cached, but writes are not. There's currently someone
   investigating a switch to a proper write-back cache on files opened
   without the O_SYNC flag. This would result in writes being cached
   and, more importantly, write() returning before the page is
   encrypted. Some rough, initial tests show a very nice improvement.

2. bs=512 results in a 4 writes to a single page, meaning that single
   page is encrypted 4 times in a row and dd has to wait on the page
   encryption and writing 4096 bytes to the lower filesystem every
   time. This is why doing what I talked about in #1 has so much
   potential to increase performance.

Encryption isn't free, but we should be a little smarter about when we
do it.

[brx75 (brx75x)]

I've tested it on my new Dell E6410 and the difference is about 1/10 of performances between ecryptfs ext4 and regular ext4

[Dustin Kirkland  (kirkland)]

Marking confirmed/medium in Ubuntu to match upstream.

Note that this bug will be fixed in the kernel, rather than in the ecryptfs-utils userspace toolset, but we're tracking discussions here. I'll mark this fix-released when an Ubuntu kernel contains a fix.

[Tyler Hicks (tyhicks)]

Marking this as "fixed released" after the 2.6.39-rc5 kernel. It included a patch which included commit 57db4e8d73ef2b5e94a3f412108dff2576670a8a to convert eCryptfs to a writeback caching model.

The performance boost isn't as big as I'd hoped because we still have to flush the dirty data upon closing the file. This involves encrypting the data and writing it to the lower filesystem.

Due to the current eCryptfs architecture, there's simply not many more options to significantly increase the performance. Software encryption is expensive.

[Nicolay Doytchev (lightrush)]

Using full-disk encryption with encrypted LVM suffers from no such speed issues. However I am not sure how an SSD will fare without TRIM being sent to it. I am determined to try that soon. :D

[Michael Schuerig (michael-schuerig)]

I'm using eCryptfs on top of ext4 on a Crucial C300 256GB SDD. For the first couple of months, I had the ext4 filesystem mounted with option discard and I was very disappointed by the performance. Specifically, in KMail, jumping through unread messages, stored in my eCryptfs HOME, was accompanied by a very disruptive lag. Listing a directory on the command line (ls) the first time had a similar lag.

SSDs are supposed to be fast, aren't they? I was resigned to accept this as the price I had to pay for encryption. A few weeks ago, triggered by another discussion, I mounted the ext4 fs without discard. The effect is extremely noticeable: there are no lags anymore, everything happens instantly. I gladly exchange this newfound speed for running fstrim every now and then.

I haven't undertaken any measurements as I didn't need any more convincing. In particular, I can't say whether eCryptfs was at all implicated in the lags I observed. The bit of advice I have to offer is to try and mount the SSD without discard and see if the performance improves.