file name too long when creating new file (ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry)

[Jens]

===
IMPORTANT: eCryptfs can only store filenames of up to 143 characters when filename encryption is enabled. The remaining 112 characters are used for storing metadata such as the encrypted filename prefix, the signature of the filename encryption key, and an identifier for the cipher used, as well as random bytes to make /foo/file and /bar/file encrypt to different ciphertext and padding bytes to achieve proper cipher block size alignment.

This bug is considered 'fixed' by the upstream maintainers. The eCryptfs kernel error message has been reduced to a debug level log message and eCryptfs now correctly reports its maximum supported filename length through the statfs() syscall. This is all that can be done without implementing a completely new encrypted filename design. A design that allows 255 character filenames without introducing other design limitations has not been identified and no one is currently working to come up with such a design.

Please do not add comments or create new bugs saying that mv reports 'File name too long' or that you can't create a long filename in your eCryptfs mounts. It is an unfortunate design limitation that cannot be avoided at this time.

Please do create new bugs when an application generates filenames that are too long to be stored in an eCryptfs mount. The application may be able to use the statfs() syscall to check the filename length limits of eCryptfs. Note that this does not include something like a torrent or ftp client trying to download a file with a long filename. The application is not generating the filename in those cases, it is just downloading the file that the user told it to download.
===

When trying to create a new file with a relatively long filename (f. ex. dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt)
I get an error: file name to long, when in fact the file name is not to long, but the encrypted name created for this file is to long, so, the file was not created.

this is no problem when I try to create a file, but when I'm copying a lot of files to my home folder I get some: filename to long error's and it's hard to fix (first locate the file, create shorter name, move again)

so, maybe you could create a check for to long filenames?

I'm using ext4 here...

mv dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt /home/jens/
mv: cannot stat `/home/jens/dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt': File name too long

libecryptfs0:
  Installed: 71-0ubuntu2
  Candidate: 71-0ubuntu2
  Version table:
 *** 71-0ubuntu2 0
        500 http://be.archive.ubuntu.com jaunty/main Packages
        100 /var/lib/dpkg/status

[Jens (jens.timmerman)]

To clarify:
When I enter perfectly valid names (for my filesystem) some "become" to invalid after encryption, and thus I get an error...

[Tyler Hicks (tyhicks)]

Jens wrote:
> Public bug reported:
>
> When trying to create a new file with a relatively long filename (f. ex. dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt)
> I get an error: file name to long, when in fact the file name is not to long, but the encrypted name created for this file is to long, so, the file was not created.

Unfortunately, this is a limitation of filename encryption. The
original plaintext name must be encrypted, encoded (to be printable,
POSIX compliant filenames after encryption), and prepended with some
eCryptfs specific data. This significantly increases the filename
length when written to the lower filesystem (ext4 in your case).

>
> this is no problem when I try to create a file, but when I'm copying a
> lot of files to my home folder I get some: filename to long error's and
> it's hard to fix (first locate the file, create shorter name, move
> again)
>
> so, maybe you could create a check for to long filenames?

We do have a check and you're hitting it. :)

I'll we can do it pass along the filename to the lower filesystem, if it
says the name is too long, we return an error to userspace. I'm open to
other ideas if you have them.

>
> I'm using ext4 here...
>
> mv dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt /home/jens/
> mv: cannot stat `/home/jens/dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt': File name too long

This is probably the best we can do for this case.

Tyler Hicks

[Dustin Kirkland  (kirkland)]

Thanks for the report, Jens.

Unfortunately, this is a known limitation of filename encryption.

I'll leave the bug open for now, in case Tyler or someone can think of
some better way to address this...

:-Dustin

[Jens (jens.timmerman)]

Ok, I get it,

I was under (the wrong) impression that ecryptfs would have the same properties as the lower file system...
But it now occurs to me that there isn't an obvious solution.

Thank you anyway, this bug may be closed (in my opinion)
I'll just have to learn not to use very long filenames (I only had a couple of files with such long filenames, so, ok :) )

[Dustin Kirkland  (kirkland)]

As an alternative, you could disable filename encryption. This would
affect all files, however, and not just the ones that are too long
....

:-Dustin

[Tyler Hicks (tyhicks)]

I shouldn't have made it sound like there's nothing we can do. For
short term, I'm not convinced that the lower filename
"ECRYPTFS_FNEK_ENCRYPTED." prefix is all that necessary. But that
really isn't going to help a whole lot.

Long term, we may be able to leave the encrypted filename out of the
actual filename in the lower directory and place it in the header of the
file. The filename in the lower directory could be something unique
(possibly random) with an indication to extract the encrypted filename
from the file header. I believe that design was considered by the guy
who implemented the current filename encryption feature, but I can't
currently remember all of the problems with that. If more people run
into this problem, I'll revisit this idea.

Thanks for using eCryptfs and please continue to open bugs and/or
suggest features. :)

Tyler Hicks

[Dustin Kirkland  (kirkland)]

On Wed, Mar 18, 2009 at 2:07 PM, Tyler Hicks <email address hidden> wrote:
> I'm not convinced that the lower filename
> "ECRYPTFS_FNEK_ENCRYPTED." prefix is all that necessary.

Tyler-

I tend to agree... I feel like this is too verbose. Something
shorter/simpler should suffice, if anything at all.

I like your idea about using the header. What's the limitation on
length in that header? What about performance?

:-Dustin

[Benni (benedikt-mas)]

Hi Dustin,

how do I disable filename encryption for an existing encrypted home directory?

Unfortunately, I couldn't find any pointers on the internet

Thank you!

Benni

[oss_test_launchpad (oss-test-launchpad)]

Why is this bug considered "Wishlist" only? Obviously, plain ext3/ext4 are in this respect incompatible with ecryptfs encrypted file systems, so IMHO this is not "Wishlist" but "Importance: high" (unless you think that having no file system incompatibilities is some sort of luxury).

[Mikko Rantalainen (mira)]

Clear documentation about the limits is all that's required. There're already differences between different filesystems and ecryptfs is a different filesystem from every point of view (compared to lower filesystem). Filesystems do have different limits for the length of directory entries (filenames), the maximum number of files in a single directory, the maximum number of subdirectories in a single directory, the maximum file size of a single file, the maximum size of the whole filesystem. This bug is about a difference between the limit in ecryptfs compared to ext4.

Increasing the maximum length of a directory entry in a single filesystem is definitely a Wishlist item.

A POSIX compatible filesystem must support up to 14 letters in range A-Z, a-z, 0-9 and letters "_.-" in the filenames. Anything above that is some sort of luxury.

Documentation should explain at least following items:
- limits depend on lower filesystems (with formulas to compute actual limits from lower system's limits)
- ecryptfs is not a feature of lower filesystem (e.g. native support for "encrypted directory") but an another filesystem that just happens to use lower filesystem as storage
- documentation should probably explain what mounting a filesystem means -- I would guess that most users are not familiar with the concept.

[Alex Dean (alex-dean)]

Just wanted to add my name as someone who's been affected by this bug. The calendarserver package creates long filenames, and trying to store these on an eCryptfs filesystem fails with a "File name too long" error. I am going to look into storing the filenames unencrypted, but if someone were to store the real filename in the header (as suggested above), I would certainly see that as a superior solution.

[Jesse (sbjesse)]

+1
Thank Dustin and Tyler and everybody that contributed!
I mean before ecryptfs I used encfs and the file name encryption there works practically way better! No offense but the current bloat ratio of FNEK in ecryptfs is nearly unbearable!
So I have this file which would be quite common (spaces escaped)
1-01\ Serenade\ No.\ 13\ in\ G\ major\,\ K.\ 525\,\ _Eine\ kleine\ Nachtmusik__\ I.\ Allegro_\ Serenade\ No.\ 13\ in\ G\ major\,\ K.\ 525\,\ Eine\ kleine\ Nachtmusik_\ I.\ Allegro.mp3
length of original name: 154
after encfs file name encryption (all default settings): 217
in ecryptfs: too long ...

a table of comparison
        file/folder name original length encfs len ecryptfs len
Various Artists 16 25 85
The Very Best of Mozart 24 47 105

Really hope this can somehow be fixed

[Andrei Pozolotin (andrei-pozolotin)]

Dustin, Tyler:

my 2 cents:

can we do zlib compression before you mangle it into the POSIX lower name?

most long file names are human-readable, which mostly means redundant;

I did a zlib compression test with some 1K of my files with name size > 100 chars;
result: about 4.0 compression ratio : 4 bytes into 1; chances are everything will fit!

and thank you for ecryptfs!

cheers,

Andrei

[Andrei Pozolotin (andrei-pozolotin)]

this is my current work around: truncate before copy into ecryptfs folder:

#!/bin/bash
#
# Bash Script to truncate file names
#

ROOT="/path/to/your/files/with/long/names"
SIZE=110

find "$ROOT" -type f | \
while read filepath1; do
  path=$(dirname "$filepath1")
  file1=$(basename "$filepath1")
  name1=${file1%.*}
  extn=${file1##*.}
  if [ ${#name1} -gt $SIZE ] ; then
    name2=$(echo $name1 | cut -c1-$SIZE)
    file2="${name2}.${extn}"
    filepath2="$path/$file2"
    echo "1 :: $filepath1"
    echo "2 :: $filepath2"
    mv "$filepath1" "$filepath2"
  else
    # echo $name1
    :
  fi
done

[AleksanderAdamowski (aadamowski)]

Andrei, did you compress each file name separately or their list as one block of data?

I 've tested on a single long file name (250 characters) and gzip compressed that to 191 bytes, bzip2 to 222 bytes.

I suppose that plain simple huffman encoding could potentially be much more effective...

[AleksanderAdamowski (aadamowski)]

An idea: use BCL (Basic Compression Library from http://bcl.comli.eu/) to compress file names using some basic encoding like Huffman.

"The Basic Compression Library is released under the zlib license, which means that it is free for anyone to use, modify and redistribute, either in source code or binary form. The complete (yet short) license text can be found in the library distribution."

[Reik Red (reikred)]

I ran into the filename length limitation, too. Would be great to see the Huffman idea or other appropriate soluton implemented.

[Reik Red (reikred)]

I discovered a new twist on this problem: It appears to me that I am not running into the ext3 limitation of 254 chars per se, but rather something more akin to a PATH LENGTH limitation somewhere else in the code.

In fact, when you look at the error messages (with dmesg), you see a pathname which is much longer than 254 chars, for example (my test case)

ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry =
[ECRYPTFS_FNEK_ENCRYPTED.FfYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDaEqXtCmeQTYUmp0L7EuVHMros54PfYi9lghUYG4pmMilVowMecKV2rPFuIDn9O4u.LJI5CqZRYX68tbVD-MR-ep4InDbzYArt.WRudsdq-YTD2epjmTNjgjpFDqWvTZvPC9gJgh9BN4njWSXubQnz2W-.1Eus44cOgPqXiHHdmvYSLigmrMcRaMtzCuJVNvWLhHzu1i2rWiTaH0pUnk0T---]

There is more than 1400 chars here (maybe more, I do not know whether the --- at the end means more chars follow).

I may be blowing smoke here, so please correct me as needed. I could not find the code that generates the error message by searching (find . -type f -print | xargs grep "File name too long") in the ecryptfs-utils src code, so the message must be coming from the kernel module? I have to download my kernel src and take a look.

[Reik Red (reikred)]

Oops, sorry, that string is ~276 chars long, not ~1400, I was squinting at the wrong number in emacs.

[starslights (starslights)]

Hello,

I have a fresh install of kubuntu Lucid 10.04 LTS on x86 64 and this bug still.

2010-05-01 11:37:30 xxxxx kernel [ 5241.275899] ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry =
2010-05-01 11:37:55 xxxxx kernel \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
2010-05-01 11:37:55 xxxxx kernel [ 5265.922996] ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry =

Linux xxxxxx 2.6.32-21-generic #32-Ubuntu SMP Fri Apr 16 08:09:38 UTC 2010 x86_64 GNU/Linux

[Reik Red (reikred)]

Another observation: Why are there so many common characters in all the encrypted file and directory names? Here is an example where ALL names start with the common string

CRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mD

Here is a full listing from the directory:

ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mD2Lz4-FJNiMfvL6Mj2Kf7rU--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mD5AdbaVG5O4wqyugrD46p8---/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDGkJ5KGK32GDepJ6DQ87HyU--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDJ-fMFoaIqRqlQ0OAf2CXo---/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDKBlPjnMOBOX9bckG2zn8YE--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDObUlIA4sm.pXaLQxRod1Tk--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDR5JfvofxlwgkgZEpT2GkP---
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDTH8BI9gnM8SLxwmNyAuH-E--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDU0yxSLP2eg7Ol7Hy8HvR1k--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDWFBnB5x9HpyE4dRkkHEg0---/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDYhpXlIrMIOgDDv-NVA3sX---
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDYyRxgrT7.KwBPosEE4BZh---/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDbTrYtWXX-yoZuNDudHkp.---/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDgdic5ulNRL-mjrIv.TcaoU--@
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDk75x8aJeiY7ix6pe.8tnY---/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDmvspLzxntktoo-i-dWpV0---
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDwWb1mJvdMbE630M.CVr6AE--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDwsXPm1-Y87g3jWcuRre2hU--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDxbUmhLxBnVrapd6k0tUZOU--/
ECRYPTFS_FNEK_ENCRYPTED.FXYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDQes.rPTBp6XZYLa.RM979.Cb5EKPMJ8q.pLmnS-DZVc-@
ECRYPTFS_FNEK_ENCRYPTED.FXYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDiw-dvFM34ohEFQ9eibhGauRH6Jc0-0tZdyHFb0CD3Lg-/
ECRYPTFS_FNEK_ENCRYPTED.FXYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDx0B9EEfuj-bembZstKerw1SDWAYfwQeWoOED2ww.AyQ-/

[Robbie Morrison (robbie-actrix)]

Another path (as opposed to filename) length constraint surely arises when archiving.

I was planning to back up '/home/$USER/.Private' directly to DDS tape.

But 'afio' is currently path limited to 1023 chars and 'star' to 4095 chars. Older versions and other utilities usually fare worse.

There is some merit in using unencrypted filenames for archival purposes. Indeed I could live with that work around, but other user may not want too.

Robbie Morrison, Berlin, Germany.

[Dustin Kirkland  (kirkland)]

Personally, I backup my unencrypted content to trusted, local storage
(ie, my local backup server in my house). And I backup my encrypted
content to untrusted, remote storage (such as Ubuntu One or Dropbox).

[AlexGenaud (alexgenaud)]

The long-filename bug affects me quite often. Though the work around, to just reduce the length of the plaintext filename, usually works ok - albeit annoying.

However, I just experienced a case where a long ecryptfs file was copied from one Ext4 device (from within an encrypted Ubuntu 09.10 Private directory) to another Ext4 device (within a TrueCrypt partition), but could not be copied again to a third Ext4 device (within a large encrypted Ubuntu 10.04 home directory). How did the file get written in the first place?

I've deleted the encrypted file, do not know the original plain text file name, and could not share details of the file anyway. :(

[bitinerant (bitinerant)]

When I set up eCryptfs on 10.04 and tried copying my huge home directory (formerly ext3) to the new system, I was hit by 27,278 errors and it cost me days of work. (Most of these paths were too long because of a bug in other software, but many were not.)

I realize we're not voting here, but I am very much in favor of at least compressing the filename data prior to POSIX encoding (see comment #13).

[Keith Grizzell (grizzell)]

I see these entries in my kern.log but I'm not aware of any particularly long filenames or any files not copying. So, is this *silently* corrupting some collection of files somewhere in the deep recesses of my home directory by not allowing an occasional file to be written? (I.e. I get no warning popup with a siren going off, so should I assume it's being handled gracefully and everything is okay? Or, is the kern.log entry the only warning of an actual problem that the simple end user can expect to get?)

Also, what does it mean when there's a bunch of backslashes instead of a garbled filename in log message?

[Jamie Strandboge (jdstrand)]

Just to chine in with a little 'me too' here. I hit this with launchpadlib quite frequently, so many scripts that use launchpadlib fail. This makes using filename crypto more or less unusable for (at least some) Ubuntu developers. :(

[Jamie Strandboge (jdstrand)]

I wonder if ecryptfs_enable_filename_crypto should be turned off by default since the option of using encrypted HOME is so prominent in the installer. There is currently no clean way to opt out of filename encryption without having to copy the files out of the ecryptfs mount to a non-filename-encrypted location.

[Jamie Strandboge (jdstrand)]

I've also had a really hard time figuring out how to disable filename encryption with the ecryptfs_enable_filename_crypto mount option using pam (ie with encrypted HOME). Setting it in $HOME/.ecryptfsrc and /home/.ecryptfs/$USER/.ecryptfsrc doesn't seem to work.

[Jamie Strandboge (jdstrand)]

For anyone else hitting this, a workaround I did was to move ~/.launchpadlib to somewhere outside of $HOME, then symlink from there to ~/.launchpadlib. Ugly, but it works until I can figure out how to ecryptfs_enable_filename_crypto=n while use pam_ecryptfs.so.

[Stefan Sauer (ensonic)]

So nif I get errors like this:
[ 2932.820163] ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry = [\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
[ 3119.644712] ecryptfs_decrypt_page: Error attempting to read lower page; rc = [-4]
[ 3119.644720] ecryptfs_write_begin: Error decrypting page at index [1]; rc = [-4]

How can I figure the original filename that is causing it? Wouldn't that be something to include in the eror message?

[Marc Deslauriers (mdeslaur)]

I am hitting this issue regularly with evolution when downloading images in HTML mail. This has a serious impact on ordinary users who choose to encrypt their home directory by default.

This either needs to get fixed, or encrypted filenames needs to get turned off by default.

[Dustin Kirkland  (kirkland)]

Tyler,

Can you give us any update here?

Is this something that upstream will *ever* fix? This is the biggest
barrier to wider adoption of eCryptfs that I know of ....

:-Dustin

[Tyler Hicks (tyhicks)]

Dustin - I was just looking into this last week. I was hoping to come up with
a way to salvage the current design of storing all metadata and ciphertext
in the lower file name for ease of implementation and performance
reasons. My intention was to improve the format. I am pretty confident
that I can squeeze an extra 20 characters out, but I don't think that
will solve the problem for those affected by this bug.

Several folks have requested that we do compression on the filename. I
don't believe that will get us anywhere. There are only a few
non-random bytes after encryption and before encoding.

The biggest issues are from block alignment and random data to generate
unique ciphertext (between 17 and 32 bytes) and the expansion from
encoding to POSIX compliant characters (between 17 and 57 bytes or about
a 33% increase in length), not to mention the static 24 character prefix.

I see a couple options going forward:
1. Provide an easier way for users to disable file name encryption on
   newly created files, but still be able to decrypt existing file
   names.
   + Easy to implement
   + Easy to turn on, as it would simply be an extra mount option
   - Doesn't solve the problem of encrypting long file names

2. Fail over to storing the ciphertext in the file metadata (header or
   xattr) when filenames are too long for the lower file system. The
   lower file name will contain some packet information (similar the
   tag 70 packet in the current design) that indicates the ciphertext
   is actually stored in the file metadata.
   - Considerably more difficult to implement - I haven't yet worked
     out how lookup() will happen
   + Allows for 255 char file names and solves all problems related to
     this bug
   - Performance of lookup() will be affected when long file names are
     in use

Maybe it would be best to implement #1 soon and then begin to
approach #2 afterwards.

[Tim Gardner (timg-tpi)]

With regard to option 1, how about a mount option that would allow FNEK to fail gracefully, i.e., if the encrypted file name is too long then just use the unencrypted name on the lower file system ? This would make the user complicit in the decision to leak unencrypted file names. This would also at least allow programs to run (such as Evolution) that generate long meaningless file names.

[Tyler Hicks (tyhicks)]

On 07/28/2010 09:44 AM, Tim Gardner wrote:
> ... how about... if the encrypted file name is too long then
> just use the unencrypted name on the lower file system ?

Dustin has suggested this before and while it would make our lives as
developers easier, I don't like it from a security standpoint. You
either want a security feature or you don't. If a user turns this on to
make some application work in their encrypted home, now they have to
make sure they don't create a meaningful file name that is 144 chars or
longer.

I much rather prefer a mount option to load a file name key encryption
key to decrypt old file names, but not encrypt any new file names. The
decision to encrypt or not is much more predictable.

[Ralf Ebert (info-ralfebert-deactivatedaccount)]

I also encountered this while building software packages.
As the text-based installer offered this, I assumed this feature is rock-solid - how about a warning message that it isn't?
It's hard to tell from the comments - can data get lost because of this? I only saw the warnings in the log while everything working as expected (which makes me even more suspicious). If so, is there a safe workaround to disable this feature? Is there a way to see which files are affected?

Aug 5 23:06:54 lucy kernel: [12407.756965] ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry = [\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\

grep lookup_one_len syslog | wc
    334 4676 342684

[Soo-Hyun Choi (s.choi)]

I am also facing with this problem, too. I'm using Ubuntu 10.04.1 LTS (64-bit server edition) together with encrypted home directory settings. Is this not fixed yet? I would love to see this problem going away.

[patcito (patcito)]

Affected too here on 10.04.1, you could at least put a warning in the installer of 10.04.2 about this bug or disable the feature completely until fixed.

[theghost (theghost)]

This bug affects me as well, e.g. when I try to sync files with long filenames via UbuntuOne. The result was, that all files, that were too long, are lost.

Please fix this serious bug or offer encryption with luks (like fedora does) at the ubuntu installation in the future.

[Bruno Medeiros (brunojcm)]

I'm still getting this error here, why do you say it's fixed?

brunojcm@brunojcm-desktop:~$ uname -a
Linux brunojcm-desktop 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
brunojcm@brunojcm-desktop:~$ touch dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt
touch: cannot touch `dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt': File name too long

[Bruno Medeiros (brunojcm)]

Well, just to complete my example, trying to create the same file on a non-encrypted FS works:

brunojcm@brunojcm-desktop:~$ cd /tmp/
brunojcm@brunojcm-desktop:/tmp$ touch dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt
brunojcm@brunojcm-desktop:/tmp$

[Tyler Hicks (tyhicks)]

On 2012-06-13 18:04:46, Bruno Medeiros wrote:
> I'm still getting this error here, why do you say it's fixed?

Because this is a design limitation and it is working as expected.
Attempting to use 255 character filenames, while using filename
encryption, will never work with the current design.

The best we can do with the current design is to let applications know
what the maximum filename length is (by calling statfs() and checking
statfs.f_namelen).

Due to how eCryptfs is implemented, a filename encryption design that
would allow 255 character filenames would be difficult to implement and
it would have its own set of design limitations.

[Tyler Hicks (tyhicks)]

On 2012-06-13 18:08:00, Bruno Medeiros wrote:
> Well, just to complete my example, trying to create the same file on a
> non-encrypted FS works:

You're ignoring the fact that the non-encrypted filesystem doesn't
actually have to encrypt the filename. There is overhead involved in
that.

This overhead is no different in the overhead involved in encrypting the
file contents. Your lower filesystem may have 512 bytes available to
store data (I'm ignoring inode storage requirements here), but you can't
create a 1 byte eCryptfs file because the design of eCryptfs pads out
the file to multiples of 4096 bytes before encrypting it and storing it
to the lower filesystem. It also requires 8192 bytes per file for
cryptographic metadata storage.

There is typically some trade-off between security and usability and
this, unfortunately, is no different.

[Scott Emmons (lscotte)]

For other people struggling with this issue specifically related to scala class filenames, there is an option to the scala compiler, -Xmax-classfile-name, which can be used to work around this issue. Any class names longer than the specified length will use a hash. Filenames are still prefixed with the Class name. As an example, filenames will look something like:

ThisIsTheNameOfMyClass$$anonfun$1$$$$451f27b826de0e7dafedb447696907b$$$$5$$anonfun$apply$mcV$sp$16$$anon$3.class

So they still have enough human grokable information in them.

And of course this doesn't help if you need to unjar/extract anything which was compiled with long filenames, but at least you can fix your own build systems with this option.

The limit that seems to work is 144: -Xmax-classfile-name 144

[Wuestenschiff (wuestenschiff)]

I still hit this Bug on 12.04 (long torrent names in Tranmission). The description says this bug is fixed since alpha 2. Do I have to reinstall the system to get it? (Because I did an upgrade from 11.10). If so this should be mentioned somewhere.

[papukaija (papukaija)]

@Desertship: You don't need to reinstall but only to install all updates. Please open a new bug if you are affected by this bug after installing all updates and rebooting. Thanks.

[papukaija (papukaija)]

Please also tell us the name of the application that you were using. Please see more details in comment 83.

[Wuestenschiff (wuestenschiff)]

@papukaija Did report new bug:
https://bugs.launchpad.net/ecryptfs/+bug/1018050

sry for the bad formatting I'm nor really used to launchpad yet. Problem affects tranmission (bittorrent cliecnt (there is already at least one bug)) but also low level tools such as mv.

[Tyler Hicks (tyhicks)]

eCryptfs cannot store filenames longer than 143 characters. It is simply a design limitation. You can read the history of this bug report and see that we've tried to come up with a better way of doing filename encryption but there is no known solution to store 255 character filenames without creating new design limitations.

Please only create new bugs if an application generates filenames that are too long to store in eCryptfs. The statfs() syscall provides a way for applications to check maximum filename length. eCryptfs now correctly reports its maximum filename length and applications should obey it.

When mv'ing files with long filenames into an eCryptfs mount, you may hit the filename limit. This is unfortunate, but it is a necessary limit when filename encryption is enabled.

[Mark Szentes-Wanner (markiboy)]

Dear Tyler,

can you please repeat it for some of us what these possible new design limitations would be?

While from the strict technical point of view I agree with you that the code works as per specification, I am very disappointed that this issue could not be solved in 3 years. For current end user systems, a file system with 143 characters maximum file name length is not fit for everyday use.

Thanks for your support

Mark

[Tyler Hicks (tyhicks)]

On 2012-06-26 22:21:29, Mark Szentes-Wanner wrote:
> can you please repeat it for some of us what these possible new design
> limitations would be?

Hi Mark - I added some high level details to top of the bug description.

> While from the strict technical point of view I agree with you that the
> code works as per specification, I am very disappointed that this issue
> could not be solved in 3 years. For current end user systems, a file
> system with 143 characters maximum file name length is not fit for
> everyday use.

I understand that it is a frustrating technical limitation, especially
for an end user.

The filename length restriction affects each user differently. I know
many people that have never hit the limit in everyday use but I do
appreciate the large number of people that are affected by the limit.

I hope that someone comes up with a great idea and submits a patch that
I can commit upstream. Until then, eCryptfs will have this filename
length limitation.

[Mark Szentes-Wanner (markiboy)]

Thanks Tyler for explaining.

Mit freundlichen Grüßen

Mark Szentes-Wanner

[Thomas (user-thomas)]

Hello,

what about to encoded long filenames in filenames of extra files? So that if a filename is too long an extra file is created which sole purpose it is to provide additional filename data in its own filename. The size of the extra file could be 0 bytes or filled with random data up to the minimum possible filesize (4096 bytes if I'm not mistaken).

So it could "look" somewhat like this:

ECRYPTFS_FNEK_ENCRYPTED.AbC.LONG_FILENAME_INDICATOR_ID_1234_1_2_This_is_a_ve--
ECRYPTFS_FNEK_ENCRYPTED.AbC.LONG_FILENAME_INDICATOR_ID_1234_2_2_ry_long_name--

ECRYPTFS_FNEK_ENCRYPTED.AbC.LONG_FILENAME_INDICATOR_ID_5782_1_3_Another_very--
ECRYPTFS_FNEK_ENCRYPTED.AbC.LONG_FILENAME_INDICATOR_ID_5782_2_3_speaking_and--
ECRYPTFS_FNEK_ENCRYPTED.AbC.LONG_FILENAME_INDICATOR_ID_5782_3_3_longish_name--

Pros:
+ No file content must be read.
+ No problems with hardlinks.
+ No xattr related problems.

Cons:
- Uses more disk space.
- Possible notable performance loss when creating many small files.
- Possible arising of new problems when creating a massive massive amount of files. (Do to number of files limitation. But I believe that that is a theoretical problem and one will run out of space before that could happen.)

(I'm effected by this limitation as a Consumer-NAS-User. Non-ASCII-Filenames must be even shorter than 143. I first hit the limitation with a 63 character long filename.)
(English isn't my first language nor am I a FS expert. So I'm very sorry if I couldn't make my self clear or the proposal doesn't make any sense.)

[Tyler Hicks (tyhicks)]

On 2012-10-15 14:03:46, Thomas wrote:
> what about to encoded long filenames in filenames of extra files? So
> that if a filename is too long an extra file is created which sole
> purpose it is to provide additional filename data in its own filename.

Storing the extra part of the filename somewhere else (whether it is in
an xattr, in the file's header metadata, or in a special per-directory
file) is a possibility but there are still downsides to that type of
solution. Added code complexity is also a big issue from a maintenance
point-of-view.

If quality patches were proposed, I'd strongly consider merging them but
I do not currently have plans to implement this type of functionality.

[Gabriel (misc-evotex)]

Do you really have to have that prefix in there? If you REALLY have to have some way to know it is encrypted/obfuscated can't you just store a flag in the xattr?

[Tyler Hicks (tyhicks)]

On 2012-11-05 09:37:27, Gabriel wrote:
> Do you really have to have that prefix in there?

No, we don't have to, but that's how it was in the original design.

Doing away with the prefix doesn't really gain us much in the bigger
picture of trying to support 255 characters.

[Felix Moreno (info-justdust)]

Hi, i respect the work of the open source community, and they take decissions because of they create the code and they will have to take care of it, but ....

Reduce the number of chars of the filename, just because the directory is crypted is not at all a good decision.

Is like not use doors in a car because of is more safe to have only one door at the top of the car, or becasue of the designer of the car decided to put the door in the top....

Is very conmon to use the number of chars ext4 (for example) offers, and if you crypt a ext4 directory you should respect the max file chars of ext4... if not, just create your own crypted file system with your own limits.

Is a step in the wrong direction reduce the possibilites of a file system just because of is crypt.

Bests.

[Yanpas (yanpaso)]

Faced this on trusrty when was executin "ecryptfs-migrate..."

[Yanpas (yanpaso)]

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
IMPORTANT

By the way the limitation for non latin users are stricter. For example cyrillic symbols in UTF8 consume 2bits, not 1. So filename may be maximux 71 symbols. AFAIR chinese symbols consume 3 bytes...

IMPORTANT
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

PS Change the description please

[joshua42 (lummony)]

I would recommend in this case to try program Long Path Tool

[Jason M. (jason-marx)]

My comment is not directly related to this issue, but still it affected me:

To all git users out there: Using git on encrypted file system, that's where your filename too long error might come from!
Solution: Move your git directory to a non encrypted file system.

[Ilya (mirraz1)]

Interesting: encfs can deal with long filenames, but ecryptfs can't (I know encfs is based on fuse and ecryptfs isn't). And it's very sad that you are not thinking about non-ANSI users (mentioned by Yanpas) . Now I constantly facing with this limitation and I constantly regretting about migrating from encfs.

[Jarno Suni (jarnos)]

Are the above two messages spam about some Windows program?

[baltic (bugsgenerator)]

Guys you should at least have warned users about this shit at the install screen, where you offer to encrypt the home folder. Those limits are not really that hard to hit, when you use non latin alphabet and are a serious disadvantage.

[Gary Brainin (brainin)]

@baltic (bugsgenerator) You're right, but the powers that be have for years refused to see the distinction between changing the way filenames are handled (which they have, I think, adequately explained the technical reasons for not doing) and providing adequate documentation and error handling (which they have conveniently ignored).

[Nicola Bernardelli (jazztp)]

@baltic +1 At least a warning **should** be issued when offering to encrypt the home folder, clarifying what the length limits are with and without that encryption. The limit is not **at all** hard to hit, I hope a solution will be implemented.

[mhalcrow (mhalcrow)]

Hi, original designer/author of eCryptfs chiming in. Sorry for the mess with file name lengths, but because of technical constraints in the kernel file system stack and security concerns, I went with the design I did.

Good news is, I fixed this issue for file-based encryption by designing it (sort of) right in file system native encryption. That's currently available in ext4 for kernels 4.1 and later. We're working to switch from eCryptfs to ext4 encryption for Ubuntu 17.10.

https://github.com/google/fscrypt/issues/24

We've released a really great user space utility to help you do pretty much everything you can do with eCryptfs today:

https://github.com/google/fscrypt

https://launchpad.net/~ubuntu-security/+archive/ubuntu/ubuntu-security-staging/+build/13255257

Prior to Ubuntu 17.10, you'll be in "early adopter" territory with ext4 encryption on Ubuntu.

Note that I did make a security tradeoff with the current version of file name encryption in ext4 for the sake of giving you (1) the full 255 characters and (2) decent performance. If the plaintext prefixes for two file names in the same directory collide, then the ciphertext prefixes will also collide. I asked Alex Cope and Eric Biggers to try to fix that by implementing a new encryption mode that they proposed upstream, but getting new encryption modes upstream isn't necessarily a trivial prospect. No real crypto review/comments so far.

https://www.spinics.net/lists/linux-crypto/msg22416.html

Note that Android is carrying the HEH patches and is using that mode for file name encryption. Hopefully we'll be able to get that upstream at some point.

[iGadget (igadget)]

Thanks mhalcrow for your elaborate reply. While appreciated, IMHO you don't need to apologize. It's not your fault we as end-users weren't notified of this limitation during installation.

For a lot of us, the situation looks pretty sad. There's eCryptfs which has the file name limitation and there's full disk encryption which is (AFAIK) only available on the alternate installer. And last time I heard, full disk encryption poses problems when Windows is installed on the same disk, leaving eCryptfs as the only option to many of us. So kudos for trying to fix this!

About your proposed solution - as an Ubuntu 16.04.2 user with (currently) 4.10 kernel, does this mean I could use ext4 encryption without too much risks or would I still be in "early adopter" territory?

[mhalcrow (mhalcrow)]

If you want to use ext4 encryption with a subdirectory of your home directory and don't mind using the e4crypt or fscrypt tools to set it up, you shouldn't run into any real problems. If you want feature parity with how eCryptfs works in Ubuntu (i.e., automatically unlocking your home directory when logging in), I'd wait until 17.10 comes out with full support for that. fscrypt has all the functionality needed to implement unlock-on-login, but until 17.10 you'll have to go through the trouble of setting up PAM and what not yourself.