Ecryptfs is very slow at listing directories with many files

[Pedro Côrte-Real]

Listing an ecryptfs directory with lots of files is very slow. Here is how to reproduce this:

• Mount an ecryptfs volume (using encrypted home directories or ~/Private/ for example)
• Create a directory with 10.000 files in it (10.000 copies of the same text file for example)
• Unmount and remount the directory (or logout/login or reboot)
• Run ls on the directory

Running find with no arguments on the same directory is immediate so it seems it is the stat()'ing of each file done by ls that makes this very slow. This is probably because stat() has to decrypt every single file. Keeping the information for stat() in the directory file would be a solution.

Here is a user account of how this problem makes encrypted home directories infeasible:

http://www.satansgarden.org/2010/03/05/removing-encryption-from-home-directories-in-ubuntu-9-10/

Should I also submit an upstream bug report as this probably isn't ubuntu specific?

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: linux-image-2.6.32-21-generic 2.6.32-21.32
Regression: No
Reproducible: Yes
ProcVersionSignature: Ubuntu 2.6.32-21.32-generic 2.6.32.11+drm33.2
Uname: Linux 2.6.32-21-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.21.
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: pedrocr 1481 F.... pulseaudio
 /dev/snd/controlC0: pedrocr 1481 F.... pulseaudio
CRDA:
 country 98:
  (2402 - 2472 @ 40), (3, 27)
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xf8220000 irq 17'
   Mixer name : 'Analog Devices AD1984'
   Components : 'HDA:11d41984,17aa20d6,00100400'
   Controls : 29
   Simple ctrls : 18
Card1.Amixer.info:
 Card hw:1 'U0x46d0x9c1'/'USB Device 0x46d:0x9c1 at usb-0000:00:1d.7-1, high speed'
   Mixer name : 'USB Mixer'
   Components : 'USB046d:09c1'
   Controls : 2
   Simple ctrls : 1
Card1.Amixer.values:
 Simple mixer control 'Mic',0
   Capabilities: cvolume cvolume-joined cswitch cswitch-joined penum
   Capture channels: Mono
   Limits: Capture 0 - 3072
   Mono: Capture 3072 [100%] [30.00dB] [on]
Card29.Amixer.info:
 Card hw:29 'ThinkPadEC'/'ThinkPad Console Audio Control at EC reg 0x30, fw 7MHT25WW-1.03'
   Mixer name : 'ThinkPad EC 7MHT25WW-1.03'
   Components : ''
   Controls : 1
   Simple ctrls : 1
Card29.Amixer.values:
 Simple mixer control 'Console',0
   Capabilities: pswitch pswitch-joined penum
   Playback channels: Mono
   Mono: Playback [on]
Date: Sat May 29 21:32:16 2010
EcryptfsInUse: Yes
HibernationDevice: RESUME=UUID=13a2cbed-32ea-49ec-8630-6cad479c043d
MachineType: LENOVO 7673CR8
PccardctlIdent:
 Socket 0:
   no product info available
PccardctlStatus:
 Socket 0:
   no card
ProcCmdLine: root=UUID=7e1b8cc5-9055-41c8-bd30-ddde4717884c ro quiet splash
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.utf8
 SHELL=/bin/bash
RelatedPackageVersions: linux-firmware 1.34
SourcePackage: linux
WpaSupplicantLog:

dmi.bios.date: 03/12/2009
dmi.bios.vendor: LENOVO
dmi.bios.version: 7NETC0WW (2.20 )
dmi.board.name: 7673CR8
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvr7NETC0WW(2.20):bd03/12/2009:svnLENOVO:pn7673CR8:pvrThinkPadX61:rvnLENOVO:rn7673CR8:rvrNotAvailable:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 7673CR8
dmi.product.version: ThinkPad X61
dmi.sys.vendor: LENOVO

[Jeremy Foshee (jeremyfoshee)]

Hi Pedro,

If you could also please test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

    [This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

[Pedro Côrte-Real (pedrocr)]

I've just tested this with 2.6.34 and as expected the bug is still there. It would be as this is probably a result of the design of ecryptfs. Should I submit an upstream bug report?

[Pedro Côrte-Real (pedrocr)]

Submitted upstream here:

https://bugzilla.kernel.org/show_bug.cgi?id=16157

[msznapka (bigmartin)]

my measurements:

--------------------------------------------------------------------------------
system restart
--------------------------------------------------------------------------------
maco@maco:~/Private$ time ls -lR > /home/maco/Desktop/temp/1st_e.txt

real 10m25.155s
user 0m2.390s
sys 0m14.960s
--------------------------------------------------------------------------------
system restart
--------------------------------------------------------------------------------
maco@maco:~/.Private$ time ls -lR > /home/maco/Desktop/temp/1st.txt

real 0m8.680s
user 0m0.990s
sys 0m1.520s
--------------------------------------------------------------------------------

Results:
ecryptfs listing => 625sec
non-ecryptfs listing => 9sec

70 times slower listing with ecryptfs on 1TB folder

[msznapka (bigmartin)]

I compared EncFS with EcryptFS.

I created 10.000 files:
for((X=0;X<10000;X+=1)); do dd if=/dev/zero of=file$X.log bs=1 count=1; done

Than I measured directory listing time:

EcryptFS:
maco@maco-desktop:~/Private$ time ls -lR > /dev/null
real 1m4.619s
user 0m0.210s
sys 0m1.460s

EncFS:
maco@maco-desktop:~/visible$ time ls -lR > /dev/null
real 0m0.921s
user 0m0.080s
sys 0m0.280s

no encryption:
maco@maco-desktop:~/.encrypted$ time ls -lR > /dev/null
real 0m0.670s
user 0m0.160s
sys 0m0.250s

Result is, that EncFS do not have this bug, so it can be easy solution for those who are using EcryptFS and have big troubles with this bug - simply move files from EcryptFS directory to EncFS directory.

EncFS:
https://help.ubuntu.com/community/FolderEncryption

[Pedro Côrte-Real (pedrocr)]

The upstream bug has been marked as WILL_NOT_FIX. The developer seems to not care about the use case claiming

"It isn't fair to say that eCryptfs won't work to encrypt user directories because very few people have 100,000 files in a single directory. This is the first time I've heard of this performance complaint."

The fact is any stat() call is slow so you just have to have a bunch of files being stat()'ed, they don't need to be in the same directory. As is obvious in the tests done by msznapka 10k files shows the slowdown nicely and that's not too hard to have in a home directory. My Photos/ has 40k files and a kernel tree has 37k files so this is hardly academic.

Switching to encfs may very well be a good solution.

[Alecz20 (alexguzu)]

I have two tests folders both wit many files and sub folders, one is on a full disk encryption with TrueCrypt, another one is eCryptfs on a ZFS RAID-5 array (much fast disk unecrypted).

I have a python script that stats files in a folder and its directories.

Under the eCryptfs, the script stats about 50k files, and it takes about 12 minutes
Under the TrueCrypt volume, it stats about 800k files (yes, 16 times more), yet it takes only 1:15 minutes

So the TrueCrypt version is about 150 times faster? (at stating)

Therefore the argument that the slowness is normal due to statting a large number of files is invalid.

A better explanation might be here:
https://bugs.launchpad.net/ubuntu/+source/ecryptfs-utils/+bug/654764
So it might be because of the design of eCryptfs.

Might give a try to EncFS and post the results

[Alecz20 (alexguzu)]

Update:
I transferred the same data from the eCryptfs folder to the EncFS folder.

The results of statting about 50k files (twice in a row) are the following:

eCryptfs : 16 minutes !
EncFS : 20 seconds !

Clearly there must be some fundamental differences between the two designs.

eCryptfs has some features that EncFS does not have, but performance-wise, the differences can be astonishing

[Gioele Barabucci (gioele)]

This problem is still present in Ubuntu 14.04 LTS.

[Vaclav Cermak (disnel)]

Tried an encrypted home with a new installation of 17.10. And probably this bug causes Syncthing on that machine useless.

Syncthing does initial directory scan on each start and, probably due to this bug, it takes about 45 min in my case a renders computer useless during the scan.