file name too long when creating new file (ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry)

[Jens]

===
IMPORTANT: eCryptfs can only store filenames of up to 143 characters when filename encryption is enabled. The remaining 112 characters are used for storing metadata such as the encrypted filename prefix, the signature of the filename encryption key, and an identifier for the cipher used, as well as random bytes to make /foo/file and /bar/file encrypt to different ciphertext and padding bytes to achieve proper cipher block size alignment.

This bug is considered 'fixed' by the upstream maintainers. The eCryptfs kernel error message has been reduced to a debug level log message and eCryptfs now correctly reports its maximum supported filename length through the statfs() syscall. This is all that can be done without implementing a completely new encrypted filename design. A design that allows 255 character filenames without introducing other design limitations has not been identified and no one is currently working to come up with such a design.

Please do not add comments or create new bugs saying that mv reports 'File name too long' or that you can't create a long filename in your eCryptfs mounts. It is an unfortunate design limitation that cannot be avoided at this time.

Please do create new bugs when an application generates filenames that are too long to be stored in an eCryptfs mount. The application may be able to use the statfs() syscall to check the filename length limits of eCryptfs. Note that this does not include something like a torrent or ftp client trying to download a file with a long filename. The application is not generating the filename in those cases, it is just downloading the file that the user told it to download.
===

When trying to create a new file with a relatively long filename (f. ex. dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt)
I get an error: file name to long, when in fact the file name is not to long, but the encrypted name created for this file is to long, so, the file was not created.

this is no problem when I try to create a file, but when I'm copying a lot of files to my home folder I get some: filename to long error's and it's hard to fix (first locate the file, create shorter name, move again)

so, maybe you could create a check for to long filenames?

I'm using ext4 here...

mv dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt /home/jens/
mv: cannot stat `/home/jens/dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt': File name too long

libecryptfs0:
  Installed: 71-0ubuntu2
  Candidate: 71-0ubuntu2
  Version table:
 *** 71-0ubuntu2 0
        500 http://be.archive.ubuntu.com jaunty/main Packages
        100 /var/lib/dpkg/status

[Jens (jens.timmerman)]

To clarify:
When I enter perfectly valid names (for my filesystem) some "become" to invalid after encryption, and thus I get an error...

[Tyler Hicks (tyhicks)]

Jens wrote:
> Public bug reported:
>
> When trying to create a new file with a relatively long filename (f. ex. dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt)
> I get an error: file name to long, when in fact the file name is not to long, but the encrypted name created for this file is to long, so, the file was not created.

Unfortunately, this is a limitation of filename encryption. The
original plaintext name must be encrypted, encoded (to be printable,
POSIX compliant filenames after encryption), and prepended with some
eCryptfs specific data. This significantly increases the filename
length when written to the lower filesystem (ext4 in your case).

>
> this is no problem when I try to create a file, but when I'm copying a
> lot of files to my home folder I get some: filename to long error's and
> it's hard to fix (first locate the file, create shorter name, move
> again)
>
> so, maybe you could create a check for to long filenames?

We do have a check and you're hitting it. :)

I'll we can do it pass along the filename to the lower filesystem, if it
says the name is too long, we return an error to userspace. I'm open to
other ideas if you have them.

>
> I'm using ext4 here...
>
> mv dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt /home/jens/
> mv: cannot stat `/home/jens/dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt': File name too long

This is probably the best we can do for this case.

Tyler Hicks

[Dustin Kirkland  (kirkland)]

Thanks for the report, Jens.

Unfortunately, this is a known limitation of filename encryption.

I'll leave the bug open for now, in case Tyler or someone can think of
some better way to address this...

:-Dustin

[Jens (jens.timmerman)]

Ok, I get it,

I was under (the wrong) impression that ecryptfs would have the same properties as the lower file system...
But it now occurs to me that there isn't an obvious solution.

Thank you anyway, this bug may be closed (in my opinion)
I'll just have to learn not to use very long filenames (I only had a couple of files with such long filenames, so, ok :) )

[Dustin Kirkland  (kirkland)]

As an alternative, you could disable filename encryption. This would
affect all files, however, and not just the ones that are too long
....

:-Dustin

[Tyler Hicks (tyhicks)]

I shouldn't have made it sound like there's nothing we can do. For
short term, I'm not convinced that the lower filename
"ECRYPTFS_FNEK_ENCRYPTED." prefix is all that necessary. But that
really isn't going to help a whole lot.

Long term, we may be able to leave the encrypted filename out of the
actual filename in the lower directory and place it in the header of the
file. The filename in the lower directory could be something unique
(possibly random) with an indication to extract the encrypted filename
from the file header. I believe that design was considered by the guy
who implemented the current filename encryption feature, but I can't
currently remember all of the problems with that. If more people run
into this problem, I'll revisit this idea.

Thanks for using eCryptfs and please continue to open bugs and/or
suggest features. :)

Tyler Hicks

[Dustin Kirkland  (kirkland)]

On Wed, Mar 18, 2009 at 2:07 PM, Tyler Hicks <email address hidden> wrote:
> I'm not convinced that the lower filename
> "ECRYPTFS_FNEK_ENCRYPTED." prefix is all that necessary.

Tyler-

I tend to agree... I feel like this is too verbose. Something
shorter/simpler should suffice, if anything at all.

I like your idea about using the header. What's the limitation on
length in that header? What about performance?

:-Dustin

[Benni (benedikt-mas)]

Hi Dustin,

how do I disable filename encryption for an existing encrypted home directory?

Unfortunately, I couldn't find any pointers on the internet

Thank you!

Benni

[oss_test_launchpad (oss-test-launchpad)]

Why is this bug considered "Wishlist" only? Obviously, plain ext3/ext4 are in this respect incompatible with ecryptfs encrypted file systems, so IMHO this is not "Wishlist" but "Importance: high" (unless you think that having no file system incompatibilities is some sort of luxury).

[Mikko Rantalainen (mira)]

Clear documentation about the limits is all that's required. There're already differences between different filesystems and ecryptfs is a different filesystem from every point of view (compared to lower filesystem). Filesystems do have different limits for the length of directory entries (filenames), the maximum number of files in a single directory, the maximum number of subdirectories in a single directory, the maximum file size of a single file, the maximum size of the whole filesystem. This bug is about a difference between the limit in ecryptfs compared to ext4.

Increasing the maximum length of a directory entry in a single filesystem is definitely a Wishlist item.

A POSIX compatible filesystem must support up to 14 letters in range A-Z, a-z, 0-9 and letters "_.-" in the filenames. Anything above that is some sort of luxury.

Documentation should explain at least following items:
- limits depend on lower filesystems (with formulas to compute actual limits from lower system's limits)
- ecryptfs is not a feature of lower filesystem (e.g. native support for "encrypted directory") but an another filesystem that just happens to use lower filesystem as storage
- documentation should probably explain what mounting a filesystem means -- I would guess that most users are not familiar with the concept.

[Alex Dean (alex-dean)]

Just wanted to add my name as someone who's been affected by this bug. The calendarserver package creates long filenames, and trying to store these on an eCryptfs filesystem fails with a "File name too long" error. I am going to look into storing the filenames unencrypted, but if someone were to store the real filename in the header (as suggested above), I would certainly see that as a superior solution.

[Jesse (sbjesse)]

+1
Thank Dustin and Tyler and everybody that contributed!
I mean before ecryptfs I used encfs and the file name encryption there works practically way better! No offense but the current bloat ratio of FNEK in ecryptfs is nearly unbearable!
So I have this file which would be quite common (spaces escaped)
1-01\ Serenade\ No.\ 13\ in\ G\ major\,\ K.\ 525\,\ _Eine\ kleine\ Nachtmusik__\ I.\ Allegro_\ Serenade\ No.\ 13\ in\ G\ major\,\ K.\ 525\,\ Eine\ kleine\ Nachtmusik_\ I.\ Allegro.mp3
length of original name: 154
after encfs file name encryption (all default settings): 217
in ecryptfs: too long ...

a table of comparison
        file/folder name original length encfs len ecryptfs len
Various Artists 16 25 85
The Very Best of Mozart 24 47 105

Really hope this can somehow be fixed

[Andrei Pozolotin (andrei-pozolotin)]

Dustin, Tyler:

my 2 cents:

can we do zlib compression before you mangle it into the POSIX lower name?

most long file names are human-readable, which mostly means redundant;

I did a zlib compression test with some 1K of my files with name size > 100 chars;
result: about 4.0 compression ratio : 4 bytes into 1; chances are everything will fit!

and thank you for ecryptfs!

cheers,

Andrei

[Andrei Pozolotin (andrei-pozolotin)]

this is my current work around: truncate before copy into ecryptfs folder:

#!/bin/bash
#
# Bash Script to truncate file names
#

ROOT="/path/to/your/files/with/long/names"
SIZE=110

find "$ROOT" -type f | \
while read filepath1; do
  path=$(dirname "$filepath1")
  file1=$(basename "$filepath1")
  name1=${file1%.*}
  extn=${file1##*.}
  if [ ${#name1} -gt $SIZE ] ; then
    name2=$(echo $name1 | cut -c1-$SIZE)
    file2="${name2}.${extn}"
    filepath2="$path/$file2"
    echo "1 :: $filepath1"
    echo "2 :: $filepath2"
    mv "$filepath1" "$filepath2"
  else
    # echo $name1
    :
  fi
done

[AleksanderAdamowski (aadamowski)]

Andrei, did you compress each file name separately or their list as one block of data?

I 've tested on a single long file name (250 characters) and gzip compressed that to 191 bytes, bzip2 to 222 bytes.

I suppose that plain simple huffman encoding could potentially be much more effective...

[AleksanderAdamowski (aadamowski)]

An idea: use BCL (Basic Compression Library from http://bcl.comli.eu/) to compress file names using some basic encoding like Huffman.

"The Basic Compression Library is released under the zlib license, which means that it is free for anyone to use, modify and redistribute, either in source code or binary form. The complete (yet short) license text can be found in the library distribution."

[Reik Red (reikred)]

I ran into the filename length limitation, too. Would be great to see the Huffman idea or other appropriate soluton implemented.

[Reik Red (reikred)]

I discovered a new twist on this problem: It appears to me that I am not running into the ext3 limitation of 254 chars per se, but rather something more akin to a PATH LENGTH limitation somewhere else in the code.

In fact, when you look at the error messages (with dmesg), you see a pathname which is much longer than 254 chars, for example (my test case)

ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry =
[ECRYPTFS_FNEK_ENCRYPTED.FfYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDaEqXtCmeQTYUmp0L7EuVHMros54PfYi9lghUYG4pmMilVowMecKV2rPFuIDn9O4u.LJI5CqZRYX68tbVD-MR-ep4InDbzYArt.WRudsdq-YTD2epjmTNjgjpFDqWvTZvPC9gJgh9BN4njWSXubQnz2W-.1Eus44cOgPqXiHHdmvYSLigmrMcRaMtzCuJVNvWLhHzu1i2rWiTaH0pUnk0T---]

There is more than 1400 chars here (maybe more, I do not know whether the --- at the end means more chars follow).

I may be blowing smoke here, so please correct me as needed. I could not find the code that generates the error message by searching (find . -type f -print | xargs grep "File name too long") in the ecryptfs-utils src code, so the message must be coming from the kernel module? I have to download my kernel src and take a look.

[Reik Red (reikred)]

Oops, sorry, that string is ~276 chars long, not ~1400, I was squinting at the wrong number in emacs.

[starslights (starslights)]

Hello,

I have a fresh install of kubuntu Lucid 10.04 LTS on x86 64 and this bug still.

2010-05-01 11:37:30 xxxxx kernel [ 5241.275899] ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry =
2010-05-01 11:37:55 xxxxx kernel \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
2010-05-01 11:37:55 xxxxx kernel [ 5265.922996] ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry =

Linux xxxxxx 2.6.32-21-generic #32-Ubuntu SMP Fri Apr 16 08:09:38 UTC 2010 x86_64 GNU/Linux

[Reik Red (reikred)]

Another observation: Why are there so many common characters in all the encrypted file and directory names? Here is an example where ALL names start with the common string

CRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mD

Here is a full listing from the directory:

ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mD2Lz4-FJNiMfvL6Mj2Kf7rU--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mD5AdbaVG5O4wqyugrD46p8---/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDGkJ5KGK32GDepJ6DQ87HyU--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDJ-fMFoaIqRqlQ0OAf2CXo---/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDKBlPjnMOBOX9bckG2zn8YE--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDObUlIA4sm.pXaLQxRod1Tk--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDR5JfvofxlwgkgZEpT2GkP---
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDTH8BI9gnM8SLxwmNyAuH-E--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDU0yxSLP2eg7Ol7Hy8HvR1k--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDWFBnB5x9HpyE4dRkkHEg0---/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDYhpXlIrMIOgDDv-NVA3sX---
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDYyRxgrT7.KwBPosEE4BZh---/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDbTrYtWXX-yoZuNDudHkp.---/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDgdic5ulNRL-mjrIv.TcaoU--@
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDk75x8aJeiY7ix6pe.8tnY---/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDmvspLzxntktoo-i-dWpV0---
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDwWb1mJvdMbE630M.CVr6AE--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDwsXPm1-Y87g3jWcuRre2hU--/
ECRYPTFS_FNEK_ENCRYPTED.FWYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDxbUmhLxBnVrapd6k0tUZOU--/
ECRYPTFS_FNEK_ENCRYPTED.FXYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDQes.rPTBp6XZYLa.RM979.Cb5EKPMJ8q.pLmnS-DZVc-@
ECRYPTFS_FNEK_ENCRYPTED.FXYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDiw-dvFM34ohEFQ9eibhGauRH6Jc0-0tZdyHFb0CD3Lg-/
ECRYPTFS_FNEK_ENCRYPTED.FXYZTa2tGpIYbEZbarH8eVFOQu-N7jr7t2mDx0B9EEfuj-bembZstKerw1SDWAYfwQeWoOED2ww.AyQ-/

[Robbie Morrison (robbie-actrix)]

Another path (as opposed to filename) length constraint surely arises when archiving.

I was planning to back up '/home/$USER/.Private' directly to DDS tape.

But 'afio' is currently path limited to 1023 chars and 'star' to 4095 chars. Older versions and other utilities usually fare worse.

There is some merit in using unencrypted filenames for archival purposes. Indeed I could live with that work around, but other user may not want too.

Robbie Morrison, Berlin, Germany.

[Dustin Kirkland  (kirkland)]

Personally, I backup my unencrypted content to trusted, local storage
(ie, my local backup server in my house). And I backup my encrypted
content to untrusted, remote storage (such as Ubuntu One or Dropbox).

[AlexGenaud (alexgenaud)]

The long-filename bug affects me quite often. Though the work around, to just reduce the length of the plaintext filename, usually works ok - albeit annoying.

However, I just experienced a case where a long ecryptfs file was copied from one Ext4 device (from within an encrypted Ubuntu 09.10 Private directory) to another Ext4 device (within a TrueCrypt partition), but could not be copied again to a third Ext4 device (within a large encrypted Ubuntu 10.04 home directory). How did the file get written in the first place?

I've deleted the encrypted file, do not know the original plain text file name, and could not share details of the file anyway. :(

[bitinerant (bitinerant)]

When I set up eCryptfs on 10.04 and tried copying my huge home directory (formerly ext3) to the new system, I was hit by 27,278 errors and it cost me days of work. (Most of these paths were too long because of a bug in other software, but many were not.)

I realize we're not voting here, but I am very much in favor of at least compressing the filename data prior to POSIX encoding (see comment #13).

[Keith Grizzell (grizzell)]

I see these entries in my kern.log but I'm not aware of any particularly long filenames or any files not copying. So, is this *silently* corrupting some collection of files somewhere in the deep recesses of my home directory by not allowing an occasional file to be written? (I.e. I get no warning popup with a siren going off, so should I assume it's being handled gracefully and everything is okay? Or, is the kern.log entry the only warning of an actual problem that the simple end user can expect to get?)

Also, what does it mean when there's a bunch of backslashes instead of a garbled filename in log message?

[Jamie Strandboge (jdstrand)]

Just to chine in with a little 'me too' here. I hit this with launchpadlib quite frequently, so many scripts that use launchpadlib fail. This makes using filename crypto more or less unusable for (at least some) Ubuntu developers. :(

[Jamie Strandboge (jdstrand)]

I wonder if ecryptfs_enable_filename_crypto should be turned off by default since the option of using encrypted HOME is so prominent in the installer. There is currently no clean way to opt out of filename encryption without having to copy the files out of the ecryptfs mount to a non-filename-encrypted location.

[Jamie Strandboge (jdstrand)]

I've also had a really hard time figuring out how to disable filename encryption with the ecryptfs_enable_filename_crypto mount option using pam (ie with encrypted HOME). Setting it in $HOME/.ecryptfsrc and /home/.ecryptfs/$USER/.ecryptfsrc doesn't seem to work.

[Jamie Strandboge (jdstrand)]

For anyone else hitting this, a workaround I did was to move ~/.launchpadlib to somewhere outside of $HOME, then symlink from there to ~/.launchpadlib. Ugly, but it works until I can figure out how to ecryptfs_enable_filename_crypto=n while use pam_ecryptfs.so.

[Stefan Sauer (ensonic)]

So nif I get errors like this:
[ 2932.820163] ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry = [\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
[ 3119.644712] ecryptfs_decrypt_page: Error attempting to read lower page; rc = [-4]
[ 3119.644720] ecryptfs_write_begin: Error decrypting page at index [1]; rc = [-4]

How can I figure the original filename that is causing it? Wouldn't that be something to include in the eror message?

[Marc Deslauriers (mdeslaur)]

I am hitting this issue regularly with evolution when downloading images in HTML mail. This has a serious impact on ordinary users who choose to encrypt their home directory by default.

This either needs to get fixed, or encrypted filenames needs to get turned off by default.

[Dustin Kirkland  (kirkland)]

Tyler,

Can you give us any update here?

Is this something that upstream will *ever* fix? This is the biggest
barrier to wider adoption of eCryptfs that I know of ....

:-Dustin

[Tyler Hicks (tyhicks)]

Dustin - I was just looking into this last week. I was hoping to come up with
a way to salvage the current design of storing all metadata and ciphertext
in the lower file name for ease of implementation and performance
reasons. My intention was to improve the format. I am pretty confident
that I can squeeze an extra 20 characters out, but I don't think that
will solve the problem for those affected by this bug.

Several folks have requested that we do compression on the filename. I
don't believe that will get us anywhere. There are only a few
non-random bytes after encryption and before encoding.

The biggest issues are from block alignment and random data to generate
unique ciphertext (between 17 and 32 bytes) and the expansion from
encoding to POSIX compliant characters (between 17 and 57 bytes or about
a 33% increase in length), not to mention the static 24 character prefix.

I see a couple options going forward:
1. Provide an easier way for users to disable file name encryption on
   newly created files, but still be able to decrypt existing file
   names.
   + Easy to implement
   + Easy to turn on, as it would simply be an extra mount option
   - Doesn't solve the problem of encrypting long file names

2. Fail over to storing the ciphertext in the file metadata (header or
   xattr) when filenames are too long for the lower file system. The
   lower file name will contain some packet information (similar the
   tag 70 packet in the current design) that indicates the ciphertext
   is actually stored in the file metadata.
   - Considerably more difficult to implement - I haven't yet worked
     out how lookup() will happen
   + Allows for 255 char file names and solves all problems related to
     this bug
   - Performance of lookup() will be affected when long file names are
     in use

Maybe it would be best to implement #1 soon and then begin to
approach #2 afterwards.

[Tim Gardner (timg-tpi)]

With regard to option 1, how about a mount option that would allow FNEK to fail gracefully, i.e., if the encrypted file name is too long then just use the unencrypted name on the lower file system ? This would make the user complicit in the decision to leak unencrypted file names. This would also at least allow programs to run (such as Evolution) that generate long meaningless file names.

[Tyler Hicks (tyhicks)]

On 07/28/2010 09:44 AM, Tim Gardner wrote:
> ... how about... if the encrypted file name is too long then
> just use the unencrypted name on the lower file system ?

Dustin has suggested this before and while it would make our lives as
developers easier, I don't like it from a security standpoint. You
either want a security feature or you don't. If a user turns this on to
make some application work in their encrypted home, now they have to
make sure they don't create a meaningful file name that is 144 chars or
longer.

I much rather prefer a mount option to load a file name key encryption
key to decrypt old file names, but not encrypt any new file names. The
decision to encrypt or not is much more predictable.

[Ralf Ebert (info-ralfebert-deactivatedaccount)]

I also encountered this while building software packages.
As the text-based installer offered this, I assumed this feature is rock-solid - how about a warning message that it isn't?
It's hard to tell from the comments - can data get lost because of this? I only saw the warnings in the log while everything working as expected (which makes me even more suspicious). If so, is there a safe workaround to disable this feature? Is there a way to see which files are affected?

Aug 5 23:06:54 lucy kernel: [12407.756965] ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry = [\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\

grep lookup_one_len syslog | wc
    334 4676 342684

[Soo-Hyun Choi (s.choi)]

I am also facing with this problem, too. I'm using Ubuntu 10.04.1 LTS (64-bit server edition) together with encrypted home directory settings. Is this not fixed yet? I would love to see this problem going away.

[patcito (patcito)]

Affected too here on 10.04.1, you could at least put a warning in the installer of 10.04.2 about this bug or disable the feature completely until fixed.

[theghost (theghost)]

This bug affects me as well, e.g. when I try to sync files with long filenames via UbuntuOne. The result was, that all files, that were too long, are lost.

Please fix this serious bug or offer encryption with luks (like fedora does) at the ubuntu installation in the future.

[Alex Muntada (alex.muntada)]

@TheGhost luks is already available in the alternate ISO installation, and most probably on the server installation too (IIRC, it's provided by partman).

[Ahmad Gharbeia أحمد غربية (gharbeia)]

Hello Tyler and every body,

I guess those who suggested compressing the filename meant for it to take place before encrypting the filename, when there's still reasonable redundancy in it, not after encryption and before encoding.

I like both ideas of the storage in file header and in xattr. I guess the hit on performance should be the determinant of which is more feasible, and I'm slightly favourable of xattr.

If such choice is made, then I think making the process uniformal is sounder on the long term, even though it will be a hassle for current users of eCryptfs - like me - who will have to recreate their encrypted storage.

By uniformal I mean there shouldn't be a check of whether the name is short enough and as such is to be found in the underlying filesystem, or longer than the threshold and thus is stored elsewhere. This will make the next version of eCryptfs incompatible with the previous. Yeah, radical but better now than bloat, in my opinion.

Lastly, regarding the name to leave in the underlying filesystem, I guess the encoded cryptographic hash of the encrypted filename should be far within the length limit, while almost globally unique.

I've hit this the first day I upgraded to 10.04 using eCryptfs over ext4, almost 6 months ago.

Thank you for your efforts, Tyler.
I'm interested to know the direction the solution is heading.

Regards,

[Jakob Unterwurzacher (jakobunt)]

FYI, filename length limits on Karmic, the ecryptfs limit just should be documented somewhere -> Relase Notes?
ecryptfs: 143
encfs: 174
ext4: 255

@Tyler: Squeezing out those 20 chars would bring ecryptfs on par with encfs - might be worth it.

[David Pollak (dpp+launchpad)]

Many pieces of software (e.g., the Scala compiler) are written with EXT4 filesystem limits in mind. Having a < 255 character limit in filenames is not simply a "documentation issue", but is a total blocker for using encrypted filesystems for me.

[Serge Hallyn (serge-hallyn)]

Quoting David Pollak (<email address hidden>):
> Many pieces of software (e.g., the Scala compiler) are written with EXT4
> filesystem limits in mind. Having a < 255 character limit in filenames
> is not simply a "documentation issue", but is a total blocker for using
> encrypted filesystems for me.

Note again that you can use ecryptfs without encrypted filenames,
in which case you do not have this issue.

[Jakob Unterwurzacher (jakobunt)]

@Compression: This may work for english, but will then fail for arabic, chinese etc. filenames. The strings are just too short I'm afraid.
Disabling the base64-style encoding and removing the prefix would give us an average limit of > 250 chars (not 255 because we'd need to encode / and NULL). Worst-case (all / or NULL) would be 127. I understand that the encoding is for portability, but couldn't it be made optional? Standard Linux filesystems don't care as long as the file name does not contain / and NULL.

[John Johansen (jjohansen)]

I have begun working on trying to resolve this bug. So far it has been poking and breaking code to get familiar with it, and working at understanding the problem in better detail (eg hardlinks have implications for storing the encrypted name in the header).

As I work on this I am going to be pushing kernel dev code to the ecryptfs branch of git://kernel.ubuntu.com/jj/ubuntu-natty.git

[John Johansen (jjohansen)]

So just an update on my probing and messing with this bug and documenting some of what I have encountered/reasons for making some of the choices I have.

The Prototype is in progress, but isn't up yet. It should be working in a day or two. I started messing with putting the long names in the header but switched to xattrs due to the factors detailed below.

Storing long names in meta data is problematic for
• hardlinks - this can be handled somewhat by either allowing a single longname or by allowing multiple longnames as space allows

Fail over to storing long names in header
• long directory names - problematic. There is no header for directories
• long symlink names - problematic.
• require update of header (reencryption) on rename

Fail over to storing long names in xattrs
• problematic for filesystems that don't support xattrs
• leaks fact that long name is present unless all files are given longname xattr
• requires update of xattr (reencryption) on rename

Combining current FNEK names and long names requires encoding information to distinguish that a given name was long and stored differently, this leaks some information about the filename. This leak provides more information than the xattr on the file, in that it provides information on which dentry is long when a file has multiple names.

Prototype
• filenames stored using current FNEK method
• detect when encrypted and encoded filename is too long and fallback to longname support
• longname support
  ∘ create unique shortname that indicates a longname exists and on which file it is stored.
     ‣ Need to be able handle name collisions, deterministic way to modify shortname until a suitable name is found. Needs to be reversed by lookup to verify that lookedup file/dir is correct one
• Prototype doesn't handle collisions
  ∘ store longname in xattr using current FNEK encryption
     ‣ currently support only a single longname
  ∘ longname needed when?
     ‣ listing directory contents
     ‣ lookup of shortname, to validate against name collisions
        • ie first shortname may not be unique, only way to verify that shortname lookup is valid is to compare name to stored longname

Once the prototype is up, I'll update here, and post it out to the ecryptfs mailing list with more details, especially of what needs to be done to move from prototype to a full functional patch.

[Dustin Kirkland  (kirkland)]

Hi John,

Wow, thanks for stepping up and taking a crack at this!

Just a few words of feedback on your concerns about leaking that long
filenames exist...

I agree that meta leakage is suboptimal, and should be avoided when possible.

On the other hand, this sort of thing happens elsewhere with eCryptfs.
 File permissions, owners, and timestamps all must be stored in the
clear, as well as file structures. That's sometimes enough to
identify which file is which.

I've always preferred calling eCryptfs' filename "encryption" ->
"obfuscation", which is a more accurate description, in my opinion.
File contents are most certainly encrypted, in a rather strong manner.
 File names and file metadata are merely obfuscated. ie, the filename
is not immediately available, but enough metadata about the file
(perms, owners, timestamps, rough sizes) to make reasonable guesses as
to real filenames in a regularly laid out filesystem.

Dustin

[BlueT - Matthew Lien - 練喆明 (bluet)]

This hits my develop server when building perl, too...
Reproduce:
bluet@Ubuntu-Rocks:~$ LC_ALL=C sudo touch really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-long-directory-name-1
touch: cannot touch `really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-really-long-directory-name-1': File name too long

bluet@Ubuntu-Rocks:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 10.04 LTS
Release: 10.04
Codename: lucid
bluet@Ubuntu-Rocks:~$ uname -a
Linux Ubuntu-Rocks 2.6.32-27-generic-pae #49-Ubuntu SMP Thu Dec 2 00:07:52 UTC 2010 i686 GNU/Linux

[crypt (cryptogranny1)]

 I'm glad that this bug is not forgotten. I'm affected as well by using sarg (perl log analyzer).

SARG: (report) Cannot open file: /home/crypt/Private/squid-reports//2011Jan11-2011Jan12/12.12.12.1/tt12.12.12.1-ppa_launchpad_net_ubuntu_mozilla_daily_ppa_ubuntu_pool_main_f_firefox_firefox_branding_3_6_14%7Ehg20110111r34860+nobinonly_0ubuntu1%7Eumd1%7Elucid_amd64_deb.html

Turning off file name encryption is definitely meaningless here.

[crypt (cryptogranny1)]

My mistake sarg isn't written in perl, just wrapper. Anyway, file names may be too informative.

[jangad butta (christmasgifts31)]

Instead of using obsolete solution from www.longpathtool.com, you'd better visit the website of original manufacturer - www.pathtoolong.com - to download the latest version of Path Too Long.

[Guido Maria Serra (zeph1ro)]

Also using TinyCA2, creating certificates for servers with long hostnames (35 characters in my case), you encounter this bug. TinyCA2 tries to create a filename out of a Base64 hash of the CN (when it generates the CSR and the private key for the host).

DEBUG return: /usr/bin/openssl req -new -keyform PEM -outform PEM -passin env:SSLPASS -config /home/me/.TinyCA/CA/openssl.cnf -out /home/me/.TinyCA/CA/req/Z2VvMDEubWFwcy5kZXZibG4uZXVyb3BlLm5va2lhLmNvbTpudW56aW8udmlzY2lhbm9Abm9raWEuY29tOkxvY2F0aW9uIFN5c3RlbSBFbmdpbmVlcnM6Tm9raWEgU2VydmljZXM6QmVybGluOkJlcmxpbjpERQ==.pem -key /home/me/.TinyCA/CA/keys/Z2VvMDEubWFwcy5kZXZibG4uZXVyb3BlLm5va2lhLmNvbTpudW56aW8udmlzY2lhbm9Abm9raWEuY29tOkxvY2F0aW9uIFN5c3RlbSBFbmdpbmVlcnM6Tm9raWEgU2VydmljZXM6QmVybGluOkJlcmxpbjpERQ==.pem -sha1

Error opening Private Key /home/me/.TinyCA/CA/keys/Z2VvMDEubWFwcy5kZXZibG4uZXVyb3BlLm5va2lhLmNvbTpudW56aW8udmlzY2lhbm9Abm9raWEuY29tOkxvY2F0aW9uIFN5c3RlbSBFbmdpbmVlcnM6Tm9raWEgU2VydmljZXM6QmVybGluOkJlcmxpbjpERQ==.pem
6343:error:02001024:system library:fopen:File name too long:bss_file.c:356:fopen('/home/me/.TinyCA/CA/keys/Z2VvMDEubWFwcy5kZXZibG4uZXVyb3BlLm5va2lhLmNvbTpudW56aW8udmlzY2lhbm9Abm9raWEuY29tOkxvY2F0aW9uIFN5c3RlbSBFbmdpbmVlcnM6Tm9raWEgU2VydmljZXM6QmVybGluOkJlcmxpbjpERQ==.pem','r')
6343:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:358:
unable to load Private Key

[John Johansen (jjohansen)]

Okay, I have been playing with this for a while and it seem stable (famous last words), the patch in its current state is in the git tree.
  git://kernel.ubuntu.com/jj/ubuntu-natty.git

And a current build against the natty kernel is available from
  kernel.ubuntu.com/~jj/linux-headers-2.6.38-3-generic_2.6.38-3.30~ecryptfslongnameV2_amd64.deb
  kernel.ubuntu.com/~jj/linux-image-2.6.38-3-generic_2.6.38-3.30~ecryptfslongnameV2_amd64.deb

if someone requests I can build i386 too.

Now a few notes about this:
- as this isn't the final version it is possible that future changes will cause breakage, you have been warned
- it may contain bugs that will eat your data
- Longname support is enabled by default and can be disabled with the mount flag ecryptfs_no_longname_xattr
  - eg. -o ecryptfs_no_longname_xattr
- the patch substitutes a shortname (hash) into the lower filesystem that is encrypted like any other filename
  so you will not be able to tell which filenames are short just by looking at them.
- longnames are stored as an xattr on the file. The name of the xattr is derived from the shortname, and the
  value is the encrypted longname
- the longname's are stored in the trusted xattr namespace so you will need to be root to even see that they
  are there.
- you can look at the xattr using the xattr command from the python-xattr package (or any similar tool) also
  if you are using -h is much more helpful than the man page
- if for some reason the longname becomes out of sync with the shortname with be shown in its place
- the encrypted files are compatible with older versions of ecryptfs, so you can dual boot between kernels
  etc without loosing data. What you do risk loosing is the longname information stored in the xattr, however
  the file should still be accessible even if this happens
- have fun breaking it and please attach the errors you get

[Reik Red (reikred)]

Hey, great to see that there is progress on this enhancement.

Is there any hope of getting a package that will work with fedora 14 x86_64? From the above comment I have (perhaps errantly) deduced that I need both a new ecryptfs-utils AND a new kernel source to compile.

But because it appears to be bundled into one Ubuntu git object I'm not quite sure how to proceed -- any advice?

[John Johansen (jjohansen)]

The way the patch is currently setup you don't need a new ecryptfs-utils, but I think that will likely change before the final iteration. If you want to test this in a fedora kernel you will need to pull the patch from the git repo and apply it to the fedora kernel and build.

pulling the patch out of the git tree is relatively simple
git clone git://kernel.ubuntu.com/jj/ubuntu-natty.git
cd ubuntu-natty.git
git format-patch HEAD^

this will drop a patch file in the directory 0001-EcryptFS-Add-support-for-file-names-that-are-too-lon.patch (or something similar) and then you can apply this to the fedora kernel

[John Johansen (jjohansen)]

Update on the current status
* the on disk format is NOT stable. Changes will likely be made during the upstreaming review cycle.
** If you have data stored using the test kernel, the migration process will be to copy your data out of ecryptfs, update your kernel and then copy your data back in.* the current patch use the xattr storage method
* development on storing long name in file headers isn't stable yet and is eating data (it is not included in the above test kernel)
* the on disk format is not stable. Changes will likely be made during the upstreaming review cycle.
** If you have data stored using the test kernel, the migration process will be to copy your data out of ecryptfs, update your kernel and then copy your data back in.
* the userspace tools have not been updated to make configuring longname support easy has not been done yet
** the test kernels have it enabled by default
** to change it to off requires doing a manual mount (remount won't work) passing the flag -o ecryptfs_no_longname_xattr

A ppa a kernel containing the patch is being set up at
https://launchpad.net/~jjohansen/+archive/ecryptfs-kernel-dev

[crypt (cryptogranny1)]

 Thanks for your work. Please don't forget LTS 10.04 users after this fix will be considered stable. Also thought that may be this bug is somehow connected https://bugs.launchpad.net/ubuntu/+source/ecryptfs-utils/+bug/372014
 If ecryptfs fails to create file or create an empty one, logs then overflooded...
---
grep eCrypt /var/log/kern.log.1 | wc -l
16280

[Dustin Kirkland  (kirkland)]

I'm sorry to disappoint, but this fix will involve a major feature
added to the eCryptfs kernel, which won't make it into the 10.04 LTS.
We might be able to provide special kernels in a PPA, but I don't see
any way that we can get this new feature into the last LTS. Sorry.

[crypt (cryptogranny1)]

I understand that backporting fix isn't that easy, but could you clarify things a bit more. I though Long Term Support means that all critical bugs in main repository (eCryptfs is in main, isn't it?) would be fixed by Canonical according to their policy. Corruption of user's data (which caused by inability to write files with long filenames) is really a serious bug. How can this be?

[Jakob Unterwurzacher (jakobunt)]

@crypt: If you can't write files because the disk is full it's not data corruption, nor is it here.

@John Johansen:
That is, for every readdir(), for each file the xattr will be read, right? Sounds like a lot of seeking, are you sure about enabling it by default? ( If you could build an i386 kernel I will happily do some performance testing )

[crypt (cryptogranny1)]

Jakob, it's always bad when application can write some portion of data, and can't another because of using long names. And I also saw when several new local folders appeared in Evolution with profile on eCryptfs. Their names were like InboxkwER, InboxRbko, and so on. Maybe I faced a new bug, but who can distinguish them, since there are "known bugs", that wouldn't be fixed.

[Dustin Kirkland  (kirkland)]

Talking to John, unfortunately this fix won't make it into Natty's kernel.

Re-targeting it for the Obnoxious Ossifrage Ubuntu release (or whatever).

[Andrew M. (ender-neo)]

This may be a naive question, but will the proposed fix break the ability to backup an encrypted directory to a fs that doesn't support xattrs?

(I'm talking specifically about the use-case where I copy my encrypted home directory directly to like cifs or something.)

[John Johansen (jjohansen)]

Jakob,

it can be a lot of extra seeks, but it is only done for entries that are long names, which are hopefully the exception. Unfortunately this is the case for any method we use to store long name information that doesn't fit in the underlying file system dentry (special dentry file, file header, xattr).

I have some more updates/alternatives coming for this patch, and will post some test kernels once the new versions are ready.

[John Johansen (jjohansen)]

Andrew M,

As to whether this breaks backups that will depend on how you do them.

If you backup to a file system that doesn't support extended attributes and you don't stuff the files in an archive (tar etc) with xattr preserving options set then, yes the backed up files will lose there long names. The files will will still be accessible, but you will need to use the short place holder name that is used when the long name can not be found. This can break some applications as they will only know to look for files using the long file name, however if the name is known the file can be restored by renaming it.

I have been experimenting with an alternative scheme where the long name information is stored in the encrypted file header instead of a xattr. This scheme will work for filesystems that don't support xattrs, but it won't allow for long names on directories, sysmlinks, etc.

[Jakob Unterwurzacher (jakobunt)]

Am 14.03.2011 19:26, schrieb John Johansen:
> Jakob,
>
> it can be a lot of extra seeks, but it is only done for entries that are
> long names, which are hopefully the exception.

Excellent, this resolves my doubts about enabling it by default. Thanks
for the response BTW!

[Mark Szentes-Wanner (markiboy)]

I accidentally hit this bug today.

I think the filename length limitation in current file systems is ridiculously low. Look at
http://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits

You can create single files of multiple Terabytes and Exabytes but their names will be limited to 255 bytes, which means less than 255 characters if you need to use Unicode. As to comment #12, a song's or book's filename could be a lot longer than that.

The only practical Linux file system that seems to support significantly more is ReiserFS/Reiser4. Btrfs is stuck at insane 255 bytes.

As there is no quick fix on the horizon, I will switch to Reiser4 (and do more frequent backups and send a kind letter to Hans...)

John, please think about the -o i-have-been-warned-about-missing-xattr mount option when the lower filesystem lacks xattr support. Still, I somehow see trouble coming for unsophisticated users who do not know xattr. Also, not all user space software treats xattr well.

Thank you for encryptfs, it is one of my favourite features when combined with pam.

[Mark Szentes-Wanner (markiboy)]

255 bytes is a kernel limit but libc and other user space tools also impose it.

[crypt (cryptogranny1)]

I've reported my bug with evolution (and data corruption) here:
https://bugs.launchpad.net/ubuntu/+source/ecryptfs-utils/+bug/745408
and together with this bug
https://bugs.launchpad.net/ubuntu/+source/ecryptfs-utils/+bug/372014
eCryptfs doesn't look very good in current Ubuntu 10.04.2 LTS.

Does current eCryptfs implementation in Lucid benefit from enabling xattr on fs?

[Andreas Raster (rakete)]

Google send me here because of a mysterious line from my syslog:
Apr 13 22:22:08 capcom kernel: [35953.661716] ecryptfs_lookup: lookup_one_len() returned [-2] on lower_dentry = [2]

Which appears in my syslog several times.

I am _very_ curious about the reasons for that, because I just mysteriously lost ~3GB of data. It was only my .local/share/Trash directory, so it is not too drastic, but I am kind of worried that it might happen again with some other directory.

When I lost that directory, I tried to recreate it. But mkdir insited that it couldn't create the directory because it was already there. Neither KDE or my shell were able to find the Trash directory, let alone list its contents.

Before all that, I (luckily?) just made a backup of my home. I use a little script that creates an lvm snapshot of my home partition first, then makes the backup and afterwards removes the snapshot. Maybe this has something to do with it? Maybe ecryptfs somehow does not like that? The snapshot itself is really large, larger than my home partition itself, so I don't think that I ran out of space there (which could maybe explain my mysteriously disappering directory).

[Andreas Raster (rakete)]

I should probably have filed a new bug, instead of commenting here. I even disabled that ecryptfs file name encryption thing when I installed Lucid because I got problems with too long filenames in my home.

So, there is something else going on.

[Tyler Hicks (tyhicks)]

On Wed Apr 13, 2011 at 09:26:40PM -0000, Andreas Raster <email address hidden> wrote:
> I should probably have filed a new bug, instead of commenting here.

Yes, please file a new bug since it sounds unrelated to this bug.

[crypt (cryptogranny1)]

After all my trouble with ecryptfs, I removed problematic zero-size files and remounted home with xattr. Now it is working without problems.

[mogliii (mogliii)]

Also Zotero sometimes generates too long filenames for ecryptfs. I am affected by that.
http://forums.zotero.org/discussion/13320/

[Martin Spacek (mspacek)]

So how does one go about disabling file name encryption/obfuscation to avoid this bug? This has been asked here several times, with no answer AFAICT. I know there's an argument you can pass during mounting, but I can't find an explanation of where to change that mount option. How does one go about permanently disabling file name encryption during system install, or at the very least, immediately after setting up an encrypted home?

Also, as mentioned previously, given the severity of this bug, file name encryption should really be disabled by default, or at least made easier to disable. As Dustin's explained, it's more obfuscation than encryption anyway.

This is a real blocker for me. I have hundreds, if not thousands of files with long names that I've accumulated over the years, and these aren't even auto-generated. The upshot of this bug is that it's made me investigate alternatives, and I'm starting to lean towards whole partition encryption with dm-crypt/cryptsetup instead. Too bad that isn't an option on the ubuntu desktop CDs, just on the alternate install CDs.

[Dustin Kirkland  (kirkland)]

Martin,

Unfortunately, there is no option for disabling it at install, or after the fact.

Currently, the only supported method is on new setups from the command line, using 'ecryptfs-setup-private -n'. See the http://manpg.es/ecryptfs-setup-private manpage for details. Note that this can be used in conjunction with -a to encrypt home.

I know this response is going to be unpopular. Please keep the subsequent flaming kind.

We're still trying to get the long-filename problem solved for 12.04, at the highest ecryptfs priority.

[Jamie Strandboge (jdstrand)]

I took the liberty of having the security team handle this bug now that we have Tyler. We plan to either have this bug dealt with in one way or another for 'P' (eg fixed (hopefully) or filename encryption turned off in the installer as a fallback). I expect Tyler and John will collaborate on it, but it seems to make sense to assign it to Tyler for now.

[Dustin Kirkland  (kirkland)]

Excellent, thanks for the commitment, Jamie!

[Brad Figg (brad-figg)]

Launchpad Lib's cache file names an be quite large and are in $HOME/.cache. I have run into the filename too long issue several times using this library.

[Tyler Hicks (tyhicks)]

An explanation is in order for why this was marked fixed released in the Precise kernel.

This has been a long standing bug that is due to the design of encrypted filenames in eCryptfs. It is very difficult, if not impossible, to come up with a new design for encrypted filenames that doesn't have other side effects.

It was pointed out that there is an interface for userspace applications to query the max filename length of a mounted filesystem. The length that eCryptfs was providing was incorrect and if an application tried to create a filename of that length, then the error reported here was seen. In other words, applications had no idea what length of filenames they could use on eCryptfs filesystems.

A patch now exists in the Precise kernel that reports the correct max filename length of eCryptfs. If still try to create too long of filenames, that likely means that they aren't querying for the max filename length or that their filenames cannot be shortened.

So, if you still see these errors in the Precise kernel, please leave a comment in this bug about the application that triggered it. We'll see if the application can create shorter filenames. If the majority of affected applications cannot create shorter filenames, then we'll have to revisit fixing this in eCryptfs.

Thanks!

[Dustin Kirkland  (kirkland)]

Thanks, Tyler!

Reiterating, please mark 'affects distribution' and the package or program
that's still causing -36 errnos.

[DanielFird (danielfird2012)]

I have been suffering from accessing, managing and even renaming files that have more than 255 characters over a long time. I have tried various ways but failed. Then I have searched this problem in internet. Then I have found a solution. This software is very easy to use. Named Long path Tool. To use the program all you need to do is to download this program online and save all the settings to your computer. This program is compatible with Windows NT, 2000, XP, Vista and Windows 7. you can find it from longpathtool(dot)com

[Andrew Cowie (afcowie)]

This is still affecting Precise. Fresh install on both sides, and rsync can't cope with ecryptfs filenames in .Private

AfC

[papukaija (papukaija)]

This bug is fixed. Please open a new bug for your issue. Thanks in advance.

[Roque (roque-ing)]

sorry for insist but Is there some patch to it for ubuntu 11.x?

[Bruno Medeiros (brunojcm)]

I'm still getting this error here, why do you say it's fixed?

brunojcm@brunojcm-desktop:~$ uname -a
Linux brunojcm-desktop 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
brunojcm@brunojcm-desktop:~$ touch dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt
touch: cannot touch `dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt': File name too long

[Bruno Medeiros (brunojcm)]

Well, just to complete my example, trying to create the same file on a non-encrypted FS works:

brunojcm@brunojcm-desktop:~$ cd /tmp/
brunojcm@brunojcm-desktop:/tmp$ touch dfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqbdfkmqsdgjfmqsldjfmlsqkjlqkdmfsmgjlqlskdqshgpoizehmlqkbgmlqbgmqb.txt
brunojcm@brunojcm-desktop:/tmp$

[Tyler Hicks (tyhicks)]

On 2012-06-13 18:04:46, Bruno Medeiros wrote:
> I'm still getting this error here, why do you say it's fixed?

Because this is a design limitation and it is working as expected.
Attempting to use 255 character filenames, while using filename
encryption, will never work with the current design.

The best we can do with the current design is to let applications know
what the maximum filename length is (by calling statfs() and checking
statfs.f_namelen).

Due to how eCryptfs is implemented, a filename encryption design that
would allow 255 character filenames would be difficult to implement and
it would have its own set of design limitations.

[Tyler Hicks (tyhicks)]

On 2012-06-13 18:08:00, Bruno Medeiros wrote:
> Well, just to complete my example, trying to create the same file on a
> non-encrypted FS works:

You're ignoring the fact that the non-encrypted filesystem doesn't
actually have to encrypt the filename. There is overhead involved in
that.

This overhead is no different in the overhead involved in encrypting the
file contents. Your lower filesystem may have 512 bytes available to
store data (I'm ignoring inode storage requirements here), but you can't
create a 1 byte eCryptfs file because the design of eCryptfs pads out
the file to multiples of 4096 bytes before encrypting it and storing it
to the lower filesystem. It also requires 8192 bytes per file for
cryptographic metadata storage.

There is typically some trade-off between security and usability and
this, unfortunately, is no different.

[Scott Emmons (lscotte)]

For other people struggling with this issue specifically related to scala class filenames, there is an option to the scala compiler, -Xmax-classfile-name, which can be used to work around this issue. Any class names longer than the specified length will use a hash. Filenames are still prefixed with the Class name. As an example, filenames will look something like:

ThisIsTheNameOfMyClass$$anonfun$1$$$$451f27b826de0e7dafedb447696907b$$$$5$$anonfun$apply$mcV$sp$16$$anon$3.class

So they still have enough human grokable information in them.

And of course this doesn't help if you need to unjar/extract anything which was compiled with long filenames, but at least you can fix your own build systems with this option.

The limit that seems to work is 144: -Xmax-classfile-name 144

[Wuestenschiff (wuestenschiff)]

I still hit this Bug on 12.04 (long torrent names in Tranmission). The description says this bug is fixed since alpha 2. Do I have to reinstall the system to get it? (Because I did an upgrade from 11.10). If so this should be mentioned somewhere.

[papukaija (papukaija)]

@Desertship: You don't need to reinstall but only to install all updates. Please open a new bug if you are affected by this bug after installing all updates and rebooting. Thanks.

[papukaija (papukaija)]

Please also tell us the name of the application that you were using. Please see more details in comment 83.

[Wuestenschiff (wuestenschiff)]

@papukaija Did report new bug:
https://bugs.launchpad.net/ecryptfs/+bug/1018050

sry for the bad formatting I'm nor really used to launchpad yet. Problem affects tranmission (bittorrent cliecnt (there is already at least one bug)) but also low level tools such as mv.

[Tyler Hicks (tyhicks)]

eCryptfs cannot store filenames longer than 143 characters. It is simply a design limitation. You can read the history of this bug report and see that we've tried to come up with a better way of doing filename encryption but there is no known solution to store 255 character filenames without creating new design limitations.

Please only create new bugs if an application generates filenames that are too long to store in eCryptfs. The statfs() syscall provides a way for applications to check maximum filename length. eCryptfs now correctly reports its maximum filename length and applications should obey it.

When mv'ing files with long filenames into an eCryptfs mount, you may hit the filename limit. This is unfortunate, but it is a necessary limit when filename encryption is enabled.

[Mark Szentes-Wanner (markiboy)]

Dear Tyler,

can you please repeat it for some of us what these possible new design limitations would be?

While from the strict technical point of view I agree with you that the code works as per specification, I am very disappointed that this issue could not be solved in 3 years. For current end user systems, a file system with 143 characters maximum file name length is not fit for everyday use.

Thanks for your support

Mark

[Tyler Hicks (tyhicks)]

On 2012-06-26 22:21:29, Mark Szentes-Wanner wrote:
> can you please repeat it for some of us what these possible new design
> limitations would be?

Hi Mark - I added some high level details to top of the bug description.

> While from the strict technical point of view I agree with you that the
> code works as per specification, I am very disappointed that this issue
> could not be solved in 3 years. For current end user systems, a file
> system with 143 characters maximum file name length is not fit for
> everyday use.

I understand that it is a frustrating technical limitation, especially
for an end user.

The filename length restriction affects each user differently. I know
many people that have never hit the limit in everyday use but I do
appreciate the large number of people that are affected by the limit.

I hope that someone comes up with a great idea and submits a patch that
I can commit upstream. Until then, eCryptfs will have this filename
length limitation.

[Mark Szentes-Wanner (markiboy)]

Thanks Tyler for explaining.

Mit freundlichen Grüßen

Mark Szentes-Wanner

[Thomas (user-thomas)]

Hello,

what about to encoded long filenames in filenames of extra files? So that if a filename is too long an extra file is created which sole purpose it is to provide additional filename data in its own filename. The size of the extra file could be 0 bytes or filled with random data up to the minimum possible filesize (4096 bytes if I'm not mistaken).

So it could "look" somewhat like this:

ECRYPTFS_FNEK_ENCRYPTED.AbC.LONG_FILENAME_INDICATOR_ID_1234_1_2_This_is_a_ve--
ECRYPTFS_FNEK_ENCRYPTED.AbC.LONG_FILENAME_INDICATOR_ID_1234_2_2_ry_long_name--

ECRYPTFS_FNEK_ENCRYPTED.AbC.LONG_FILENAME_INDICATOR_ID_5782_1_3_Another_very--
ECRYPTFS_FNEK_ENCRYPTED.AbC.LONG_FILENAME_INDICATOR_ID_5782_2_3_speaking_and--
ECRYPTFS_FNEK_ENCRYPTED.AbC.LONG_FILENAME_INDICATOR_ID_5782_3_3_longish_name--

Pros:
+ No file content must be read.
+ No problems with hardlinks.
+ No xattr related problems.

Cons:
- Uses more disk space.
- Possible notable performance loss when creating many small files.
- Possible arising of new problems when creating a massive massive amount of files. (Do to number of files limitation. But I believe that that is a theoretical problem and one will run out of space before that could happen.)

(I'm effected by this limitation as a Consumer-NAS-User. Non-ASCII-Filenames must be even shorter than 143. I first hit the limitation with a 63 character long filename.)
(English isn't my first language nor am I a FS expert. So I'm very sorry if I couldn't make my self clear or the proposal doesn't make any sense.)

[Tyler Hicks (tyhicks)]

On 2012-10-15 14:03:46, Thomas wrote:
> what about to encoded long filenames in filenames of extra files? So
> that if a filename is too long an extra file is created which sole
> purpose it is to provide additional filename data in its own filename.

Storing the extra part of the filename somewhere else (whether it is in
an xattr, in the file's header metadata, or in a special per-directory
file) is a possibility but there are still downsides to that type of
solution. Added code complexity is also a big issue from a maintenance
point-of-view.

If quality patches were proposed, I'd strongly consider merging them but
I do not currently have plans to implement this type of functionality.

[Gabriel (misc-evotex)]

Do you really have to have that prefix in there? If you REALLY have to have some way to know it is encrypted/obfuscated can't you just store a flag in the xattr?

[Tyler Hicks (tyhicks)]

On 2012-11-05 09:37:27, Gabriel wrote:
> Do you really have to have that prefix in there?

No, we don't have to, but that's how it was in the original design.

Doing away with the prefix doesn't really gain us much in the bigger
picture of trying to support 255 characters.

[Felix Moreno (info-justdust)]

Hi, i respect the work of the open source community, and they take decissions because of they create the code and they will have to take care of it, but ....

Reduce the number of chars of the filename, just because the directory is crypted is not at all a good decision.

Is like not use doors in a car because of is more safe to have only one door at the top of the car, or becasue of the designer of the car decided to put the door in the top....

Is very conmon to use the number of chars ext4 (for example) offers, and if you crypt a ext4 directory you should respect the max file chars of ext4... if not, just create your own crypted file system with your own limits.

Is a step in the wrong direction reduce the possibilites of a file system just because of is crypt.

Bests.

[Yanpas (yanpaso)]

Faced this on trusrty when was executin "ecryptfs-migrate..."

[Yanpas (yanpaso)]

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
IMPORTANT

By the way the limitation for non latin users are stricter. For example cyrillic symbols in UTF8 consume 2bits, not 1. So filename may be maximux 71 symbols. AFAIR chinese symbols consume 3 bytes...

IMPORTANT
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

PS Change the description please

[joshua42 (lummony)]

I would recommend in this case to try program Long Path Tool

[Jason M. (jason-marx)]

My comment is not directly related to this issue, but still it affected me:

To all git users out there: Using git on encrypted file system, that's where your filename too long error might come from!
Solution: Move your git directory to a non encrypted file system.

[Ilya (mirraz1)]

Interesting: encfs can deal with long filenames, but ecryptfs can't (I know encfs is based on fuse and ecryptfs isn't). And it's very sad that you are not thinking about non-ANSI users (mentioned by Yanpas) . Now I constantly facing with this limitation and I constantly regretting about migrating from encfs.

[Jarno Suni (jarnos)]

Are the above two messages spam about some Windows program?

[baltic (bugsgenerator)]

Guys you should at least have warned users about this shit at the install screen, where you offer to encrypt the home folder. Those limits are not really that hard to hit, when you use non latin alphabet and are a serious disadvantage.

[Gary Brainin (brainin)]

@baltic (bugsgenerator) You're right, but the powers that be have for years refused to see the distinction between changing the way filenames are handled (which they have, I think, adequately explained the technical reasons for not doing) and providing adequate documentation and error handling (which they have conveniently ignored).

[Nicola Bernardelli (jazztp)]

@baltic +1 At least a warning **should** be issued when offering to encrypt the home folder, clarifying what the length limits are with and without that encryption. The limit is not **at all** hard to hit, I hope a solution will be implemented.

[mhalcrow (mhalcrow)]

Hi, original designer/author of eCryptfs chiming in. Sorry for the mess with file name lengths, but because of technical constraints in the kernel file system stack and security concerns, I went with the design I did.

Good news is, I fixed this issue for file-based encryption by designing it (sort of) right in file system native encryption. That's currently available in ext4 for kernels 4.1 and later. We're working to switch from eCryptfs to ext4 encryption for Ubuntu 17.10.

https://github.com/google/fscrypt/issues/24

We've released a really great user space utility to help you do pretty much everything you can do with eCryptfs today:

https://github.com/google/fscrypt

https://launchpad.net/~ubuntu-security/+archive/ubuntu/ubuntu-security-staging/+build/13255257

Prior to Ubuntu 17.10, you'll be in "early adopter" territory with ext4 encryption on Ubuntu.

Note that I did make a security tradeoff with the current version of file name encryption in ext4 for the sake of giving you (1) the full 255 characters and (2) decent performance. If the plaintext prefixes for two file names in the same directory collide, then the ciphertext prefixes will also collide. I asked Alex Cope and Eric Biggers to try to fix that by implementing a new encryption mode that they proposed upstream, but getting new encryption modes upstream isn't necessarily a trivial prospect. No real crypto review/comments so far.

https://www.spinics.net/lists/linux-crypto/msg22416.html

Note that Android is carrying the HEH patches and is using that mode for file name encryption. Hopefully we'll be able to get that upstream at some point.

[iGadget (igadget)]

Thanks mhalcrow for your elaborate reply. While appreciated, IMHO you don't need to apologize. It's not your fault we as end-users weren't notified of this limitation during installation.

For a lot of us, the situation looks pretty sad. There's eCryptfs which has the file name limitation and there's full disk encryption which is (AFAIK) only available on the alternate installer. And last time I heard, full disk encryption poses problems when Windows is installed on the same disk, leaving eCryptfs as the only option to many of us. So kudos for trying to fix this!

About your proposed solution - as an Ubuntu 16.04.2 user with (currently) 4.10 kernel, does this mean I could use ext4 encryption without too much risks or would I still be in "early adopter" territory?

[mhalcrow (mhalcrow)]

If you want to use ext4 encryption with a subdirectory of your home directory and don't mind using the e4crypt or fscrypt tools to set it up, you shouldn't run into any real problems. If you want feature parity with how eCryptfs works in Ubuntu (i.e., automatically unlocking your home directory when logging in), I'd wait until 17.10 comes out with full support for that. fscrypt has all the functionality needed to implement unlock-on-login, but until 17.10 you'll have to go through the trouble of setting up PAM and what not yourself.