Cutting or copying files on btrfs to ecryptfs results in data loss

Bug #1305335 reported by lunomad
56
This bug affects 10 people
Affects Status Importance Assigned to Milestone
Linux
Unknown
Unknown
eCryptfs
Fix Released
Critical
Tyler Hicks
linux (Ubuntu)
Fix Released
Critical
Tyler Hicks
Nominated for Trusty by Alberto Salvia Novella
Nominated for Utopic by Alberto Salvia Novella
Nominated for Vivid by Alberto Salvia Novella

Bug Description

I've installed a fresh copy of 14.04 beta 2 on a brand-new SSD. I used btrfs as the filesystem.

I set up a Private folder usying ecryptfs-setup-private.

I copied a few folders into the Private folder, and the entire directory structure and subfolders are copied, including the files themselves, yet every file contains 0 bytes.

However, I can create and save new files in the Private directory.
Also, folders and files can be correctly copied if I use the command line cp -R.

Thanks,
Damon

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: nautilus 1:3.10.1-0ubuntu8
ProcVersionSignature: Ubuntu 3.13.0-23.45-generic 3.13.8
Uname: Linux 3.13.0-23-generic x86_64
ApportVersion: 2.14.1-0ubuntu1
Architecture: amd64
CurrentDesktop: Unity
Date: Wed Apr 9 18:12:56 2014
EcryptfsInUse: Yes
GsettingsChanges: b'org.gnome.nautilus.list-view' b'default-column-order' b"['name', 'size', 'type', 'date_modified', 'date_accessed', 'owner', 'group', 'permissions', 'mime_type', 'where']"
InstallationDate: Installed on 2014-04-05 (4 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Beta amd64 (20140326)
ProcEnviron:
 LANGUAGE=en_CA:en
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_CA.UTF-8
 SHELL=/bin/bash
SourcePackage: nautilus
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
lunomad (damon-metapaso) wrote :
Revision history for this message
lunomad (damon-metapaso) wrote :

I can see that this bug still exists in the final release 14.04.

To replicate this bug,
1.set up an SSD with BTRFS for root and home (all default settings).
2. Install 14.04.
3. Use ecryptfs-setup-private to install the Private subdirectory.
4. Create a text file on the desktop (or any directory outside Private) with any text inside.
5. Use nautilus to copy the file (either ctrl-c, ctrl-v, or right-click menus) to the private directory.
6. File will show as copied, but will contain no text (0 bytes).

I just checked and this problem also occurs if the file is cut/pasted, meaning that anyone who tries to cut/paste files into the Private directory will lose that data. I think that's a somewhat serious issue. Ouch.

Revision history for this message
Josh Hill (ingenium) wrote :

I just stumbled across this bug as well. Only noticed because I wasn't get thumbnails of an image I had just copied into my Private directory. I even tried using cp from the command line, so it's not a nautilus bug. Unfortunately I have no idea which (if any) files I lost as a result of this (I switched to btrfs over a month ago).

Unmounting and re-mounting the Private directory causes the files to show up, even after deleting (I added the file, saw it was corrupted, deleted it, unmounted, remounted, and it was there). It wasn't 0 bytes, so there was data there. However it was still corrupt, and the md5sum differed from the original.

I immediately copied everything out of my Private directory into a new directory. Everything went OK except for the corrupted file. It gave me input/output errors (on the original and the copies in .Trash), so I can only hope that the rest of the data is not corrupt.

This is a VERY serious bug causing data loss with no warning to the user that it's happening.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in nautilus (Ubuntu):
status: New → Confirmed
Revision history for this message
lunomad (damon-metapaso) wrote :

I have continued to use the private directory so far without hitch. Using nautilus and other programs, it seems as long as I create a file in the Private directory and don't move or copy it, i'm fine. I have my .thunderbird directory inside Private and everything seems to work just fine. I've only had this issue on copying or moving using nautilus.

Damon

Revision history for this message
djmaze (maze) wrote :

Same problem here. Using an ecryptfs folder on a btrfs filesystem. Running on Arch Linux though.

When copying files using nautilus or nemo into the ecryptfs folder, they often seem to get truncated to 0 bytes length. This does not happen when using "cp" from the terminal.

I get the following error when trying to read such a file:

    Valid eCryptfs headers not found in file header region or xattr region, inode xxxxx

I have to delete the file in order to get rid of the error message, just as described at http://blog.stephenn.com/2012/04/problem-with-corrupt-files-and-ecrypt.html.

Unmounting and remounting the encrypted filesystem makes broken files show with the correct filesize again. But opening them gives an I/O read error then. So I have to delete them finally.

Kernel 3.15.8-1-ARCH (x86_64)
ecryptfs-utils 103-2

Revision history for this message
Hannu Teulahti (teu) wrote :

Noticed this same problem.

When copying large files on btrfs nautilus seems to do "cp --reflink". When I copy a file from btrfs to same btrfs with ecryptfs-layer; the file seems to get corrupted. However when I do "file ECRYPTFS_FNEK_ENCRYPTED*" to the copied bare file under the ecryptfs-layer I can see the original contents.

So nautilus copies the file as plain with cp --reflink when instead it should do a proper full copy.

Revision history for this message
Rocko (rockorequin) wrote :

This sounds similar to a situation that I discovered using Ubuntu 14.10 and 15.04:

I have an encrypted home folder (in its own subvolume, in case that's relevant) and have created a symlink for Downloads to an external directory outside the subvolume. Files that I copy from this external directory into home (~) using "cp --reflink=auto" are always truncated to zero bytes without any error given during the file copy. It seems to be the symlink causing the confusion because files that I copy from other external directories are fine.

As I understand it, the reflink option should be smart enough to know when it needs to do a full copy (eg if you are copying to another file system) so the reflink option should be safe to use. So the problem is probably an issue occurring due to the combination of btrfs and ecryptfs rather than with nautilus/gnome-virtual-file-system.

stop (whoopwhoop)
Changed in ecryptfs:
status: New → Confirmed
Revision history for this message
stop (whoopwhoop) wrote :

So there is no possible way or workaround to write to a ecyptfs folder without changing/damaging the file when running btrfs?

Revision history for this message
stop (whoopwhoop) wrote :

Just found out this also broken when moving files.... This bug has data loss written all over it... Why is this being ignored? This should be critical yesterday...

summary: - nautilus fails to copy to ecryptfs Private folder on btrfs - all files 0
- bytes
+ Cutting or copying files on btrfs to ecryptfs results in data loss
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux (Ubuntu):
status: New → Confirmed
Brad Figg (brad-figg)
affects: linux-meta (Ubuntu) → linux (Ubuntu)
Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.19 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19-vivid/

tags: added: kernel-key
Changed in linux (Ubuntu):
importance: Undecided → High
importance: High → Critical
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Also, just to confirm, this issue only happens if you copy the files using Nautilus? It does not happen if you copy from a terminal?

Revision history for this message
lunomad (damon-metapaso) wrote :

Hi Joseph,

This bug freaked me out so much I re-formatted to EXT4 and started over. I don't have a system that I can test this on now.

The bug appeared on a completely fresh install of 14.04 beta 2 and the bug was still present in the release.

Unfortunately, I never had any experience with BTRFS prior so I can't tell you how it worked before. As I recall, I had success doing copies via the terminal command line using CP -R, so I assumed the bug was in how nautilus handled file copies. My experience also was that once I got a file INTO the private directory, I was able to open, edit, and save correctly without error.

Still, the whole thing sent shivers up my spine, and I'm not as technically-abled as others on this bug. I had to dump BTRFS as soon as I could.

Revision history for this message
stop (whoopwhoop) wrote :

I still have a running system with this issue.
It is not an ubuntu box but Arch 64. This is a fresh install.

I "stole" the disk layout from the ubuntu documentation for this system:
one btrfs partition with @ subvolume for / and @home subvolume for /home.
The system also uses a vfat ESP as it is an UEFI install.
Default kernel: currently 3.18.6-1-ARCH

I installed ecryptfs-utils (ecryptfs-utils 104-1) from the arch repo and did the following as configuration:
https://wiki.archlinux.org/index.php/ECryptfs#Without_ecryptfs-utils

The described setup above is currently working on an ext4 arch 64 bit system but not on btrfs system...

A copy or move with nautilus to Private folder results in file(s) with 0 bytes (on all file formats I have tested).
A cp or mv to Private folder results in proper amount of bytes but something is also wrong as
thumbnails for jpg are not showing in Private folder with nautilus although thumbnails are enabled. So maybe header is getting corrupted or something.
I also tested avi and tga but these did give me thumbnails. I have not tested other file formats at this time.

Revision history for this message
Rocko (rockorequin) wrote :

The 3.19 kernel still has the problem.

You can reproduce it by copying a test file from say /home into an encrypted home directory using --reflink:

echo test > /home/test
cp --reflink=always /home/test ~
ls -l ~/test # This shows 0 bytes

This command, however, copies the file correctly, assuming you haven't aliased cp to use reflink:

echo test > /home/test
cp /home/test ~
ls -l ~/test # This shows 5 bytes

nautilus is using the g_local_file_copy library command in glib, which since automatically applies reflink since 2.36.4 (libglib2.0-bin is at 2.43.4-1 in Vivid). See the comment at http://upstream.rosalinux.ru/changelogs/glib/2.36.4/changelog.html, where it lists "Btrfs clone/reflink ioctl support in g_local_file_copy " as one of the changes.

So one solution would seem to be to modify the copy functionality to disable reflink (or disallow the copy) if it detects it is copying into a mounted ecryptfs directory (eg using /proc/mounts) from a directory that is not in the same mount.

Presumably ecryptfs intercepts the copy request, so perhaps the check should be done in ecryptfs?

Revision history for this message
Rocko (rockorequin) wrote :

Just to note: in the example commands above, of course, /home needs to be write-accessible to the current user, or you need to use a command like "su root -c "echo test > /home/test".

Rocko (rockorequin)
tags: added: kernel-bug-exists-upstream
Revision history for this message
stop (whoopwhoop) wrote :

@Rocko: Do you have an explanation on why the thumbnail gets broken when I cp a jpg to the ecryptfs folder? The number of bytes are correct...

Revision history for this message
Rocko (rockorequin) wrote :

@whoop: If you mean that the thumbnail file length is correct but the image file is zero bytes, the reason is that the thumbnail file isn't copied from the non-ecryptfs folder to the ecryptfs folder. Instead, the system stores thumbnails in ~/.cache/thumbnails. So it creates a new thumbnail file for the newly-copied zero-byte image file in the ecryptfs folder, meaning the thumbnail file itself is valid. But the system tries to generate the thumbnail image from the zero-byte image file, which means that the thumbnail image is broken and so it appears broken in nautilus.

Revision history for this message
Rocko (rockorequin) wrote :
Download full text (3.2 KiB)

As a follow-up to comment #16: generally attempting to clone a file in a non-ecryptfs folder into a mounted ecryptfs folder appears to succeed but in fact creates a zero-length invalid target file, but going the other way (cloning from the mounted ecryptfs folder into the non-ecryptfs folder) fails as it should. (Both cp --reflink=auto and glib's file_copy_fallback try a clone operation and then try a non-clone operation if this fails, so this second case is fine.)

The attached script demonstrates both cases by creating a mounted ecryptfs system in /tmp/ecryptfs-test and using the folder 'private' to hold the 'lower' encrypted ecryptfs files and the folder 'plaintext' as the 'higher' mounted ecryptfs folder and then trying the clone operation in both directions. ( /tmp must be on a btrfs partition and you need at least ecryptfs-utils installed but you don't need an encrypted home folder.)

The btrfs_ioctl_clone command (in fs/btrfs/ioctl.c in the kernel source) is designed to only perform a clone if the source and dest files have the same VFS mountpoint and to fail with EXDEV (invalid cross-device link) if they don't:

ret = -EXDEV;
if (src_file.file->f_path.mnt != file->f_path.mnt)
  goto out_fput;

When trying to clone from the 'plaintext' folder into /tmp/ecryptfs-test, this test fails (the mountpoints are represented by two different dentries, '@' and '/').

However, when cloning into the 'plaintext' folder, btrfs_ioctl_clone is actually being asked to clone using the 'lower' ecryptfs file as the target (ie the target points to the 'private' folder, not the 'plaintext' folder), so the test passes. This is obviously wrong - ecryptfs is designed so that the information gets directed to the 'plaintext' folder, intercepted in the kernel, and stored encrypted instead in the 'private' folder.

To demonstrate this, I added this code just before the test:

 {
    #define BUFLEN 256
 char src_path[BUFLEN],dest_path[BUFLEN];
 int i;
 for (i=0;i<BUFLEN;++i) src_path[i]=dest_path[i]=0;

 printk(KERN_INFO "btrfs reflink src: %s [mnt %s], dest: %s [mnt %s]\n",
   dentry_path_raw(src_file.file->f_path.dentry, src_path, BUFLEN),
   src_file.file->f_path.mnt->mnt_root->d_iname,
   dentry_path_raw(file->f_path.dentry, dest_path, BUFLEN),
   file->f_path.mnt->mnt_root->d_iname
 );
    }

With that code included in the kernel, the syslog shows something like this after running the attached test script:

... btrfs reflink src: /@/tmp/ecryptfs-test/test [mnt @], dest: /@/tmp/ecryptfs-test/private/ECRYPTFS_FNEK_ENCRYPTED... [mnt @]

ie the target file passed to btrfs_ioctl_clone is the 'lower' file in 'private'. (Note that after the clone, trying to access this file results in an I/O error.)

I checked the cp code, and it calls:

return ioctl (dest_fd, BTRFS_IOC_CLONE, src_fd);

I used readlink to get the file path for dest_fd from /proc/self/fd/<file desriptor>, and it returned "/tmp/ecryptfs-test/plaintext/test", so desc_fd in the cp command is pointing to the correct path for the 'higher' file.

My guess is that ecryptfs intercepts this 'higher' path and converts it to the 'lower' path before it gets passed through to btrfs, but it should just do ...

Read more...

Revision history for this message
Rocko (rockorequin) wrote :

In case anyone still reads bugzilla.kernel.org, I reported this at https://bugzilla.kernel.org/show_bug.cgi?id=93691.

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Does anyone affected by this bug know if it is a regression? Was there a prior kernel version that did not exhibit the bug?

affects: nautilus → linux
tags: added: utopic vivid
Revision history for this message
Rocko (rockorequin) wrote :

I checked the mainline 3.11 kernel and it has the bug (as well as 3.13, 3.18 and 3.19, based on the other comments), so I would say it is not a regression.

Tyler Hicks (tyhicks)
Changed in linux (Ubuntu):
assignee: nobody → Tyler Hicks (tyhicks)
Changed in ecryptfs:
assignee: nobody → Tyler Hicks (tyhicks)
importance: Undecided → Critical
Revision history for this message
Tyler Hicks (tyhicks) wrote :

Thanks for the in-depth triage of this bug, Rocko. As you pointed out, I can easily reproduce this using cp's --reflink=always option.

I'm marking nautilus as Invalid since this is definitely an eCryptfs bug. I'll start determining the best way to fix this issue.

Changed in nautilus (Ubuntu):
status: Confirmed → Invalid
Revision history for this message
Tyler Hicks (tyhicks) wrote :

The `cp --reflink=always test test.bak` is always issuing the BTRFS_IOC_CLONE ioctl, even on non-btrfs filesystems. eCryptfs passes ioctls down to the lower filesystem but I suspect that it should stop doing that altogether or possibly only allow a white list of known good ioctls.

Revision history for this message
Rocko (rockorequin) wrote :

I think it is still useful for ecryptfs to support the btrfs clone ioctl for the case where both source and target higher files are in the same ecryptfs mount, since this saves disk space.

We might be able to handle this in fs/ecryptfs/file.c#ecryptfs_unlocked_ioctl, which gets passed the btrfs ioctrl clone command along with the the higher target file. It unconditionally converts it to the lower file and then passes it down to btrfs:

  struct file *lower_file = ecryptfs_file_to_lower(file);
   ...
  if (lower_file->f_op->unlocked_ioctl)
    rc = lower_file->f_op->unlocked_ioctl(lower_file, cmd, arg);

Adding code here that returns failure if the command is BTRFS_IOC_CLONE makes the cp --reflink=always command fail. (cp --reflink=auto then works because cp detects the failure and does a non-clone copy, and the same should go for the glib file copy).

However, note that:

1. When it fails to clone, ecryptfs still creates a valid but zero-byte file after the failed clone operation, so ecryptfs should probably remove this when it returns fail for the clone operation.

2. It shouldn't fail if the source and target files are both inside the same ecryptfs mount.

3. Do symlinks affect it, eg if the target is a symlink outside the ecryptfs mount that points to the higher ecryptfs file?

4. ecryptfs_compat_ioctl might possibly be affected as well since it does the same thing as ecryptfs_unlocked_ioctl.

Revision history for this message
Rocko (rockorequin) wrote :

ecryptfs_unlocked_ioctl gets passed the (higher) target file struct as the first argument, the command as the second, and the source file descriptor as the third argument. It looks like the source file descriptor has already been converted to the lower file if it is associated with a higher file inside an ecryptfs mount. This makes it harder to compare the mountpoints of the source and target to see if the clone should be allowed - you'd need to figure out if the source file descriptor is associated with a higher file then compare that higher file with the (higher) target file to see if they have the same mountpoint. I'm not sure how to do that - there aren't many comments in the ecryptfs code.

tags: added: kernel-da-key
removed: kernel-key
Revision history for this message
Tyler Hicks (tyhicks) wrote : Re: [Bug 1305335] Re: Cutting or copying files on btrfs to ecryptfs results in data loss
Download full text (3.5 KiB)

On 2015-02-24 04:32:21, Rocko wrote:
> I think it is still useful for ecryptfs to support the btrfs clone ioctl
> for the case where both source and target higher files are in the same
> ecryptfs mount, since this saves disk space.

I don't like the idea of eCryptfs supporting the clone ioctl by default.
It would allow an attacker to discover that the files (the original and
the clone) are the same. Sure, the decrypted file size of eCryptfs files
is stored in plaintext so an attacker would already know that the two
files are the same size but that doesn't mean that they contain the same
contents.

A core design aspect of eCryptfs is that a unique File Encryption Key
(FEK) is generated for each file, meaning that two matching plaintext
files will have entirely different ciphertext.

I'd consider allowing support for the clone ioctl down the road if the
user specified an eCryptfs mount option that implies that
performance and efficiency are more important than security.

Supporting sparse files in eCryptfs could fall under that same mount
option.

> We might be able to handle this in
> fs/ecryptfs/file.c#ecryptfs_unlocked_ioctl, which gets passed the btrfs
> ioctrl clone command along with the the higher target file. It
> unconditionally converts it to the lower file and then passes it down to
> btrfs:
>
> struct file *lower_file = ecryptfs_file_to_lower(file);
> ...
> if (lower_file->f_op->unlocked_ioctl)
> rc = lower_file->f_op->unlocked_ioctl(lower_file, cmd, arg);
>
> Adding code here that returns failure if the command is BTRFS_IOC_CLONE
> makes the cp --reflink=always command fail. (cp --reflink=auto then
> works because cp detects the failure and does a non-clone copy, and the
> same should go for the glib file copy).
>
> However, note that:
>
> 1. When it fails to clone, ecryptfs still creates a valid but zero-byte
> file after the failed clone operation, so ecryptfs should probably
> remove this when it returns fail for the clone operation.

This is a bug in the cp program, if anything. The same thing happens on
ext4. Here's what we see from strace:

$ strace cp --reflink=always test test.copy
...
open("test", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=5, ...}) = 0
open("test.copy", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4
fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
ioctl(4, BTRFS_IOC_CLONE, 0x3) = -1 ENOTTY (Inappropriate ioctl for device)
...

cp opens test, then creates test.copy, before calling ioctl() on
test.copy's file descriptor. The clone fails but then cp never unlinks
it.

> 2. It shouldn't fail if the source and target files are both inside the
> same ecryptfs mount.

I feel like it should fail for the reasons mentioned above.

If someone wants to add BTRFS clone support down the road, that'd be
nice. However, I think this bug is properly fixed by returning an error
when ecryptfs_{compat,unlocked}_ioctl() sees BTRFS_IOC_CLONE. That's the
only backportable fix.

In fact, I think eCryptfs should probably be returning error on any
filesystem specific ioctl commands. There are a handful ioctl commands
common across all filesystems that it should support and pass down to
the...

Read more...

Revision history for this message
Rocko (rockorequin) wrote :

> I don't like the idea of eCryptfs supporting the clone ioctl by default.
> It would allow an attacker to discover that the files (the original and
> the clone) are the same.

I agree with that reasoning.

In any case, I think that the btrfs clone operation should be disallowed in ecryptfs as a matter of urgency (upstream as well), since it can result in data loss, which is far worse than the (presumably small in practice) disk space savings available though using the clone.

no longer affects: nautilus (Ubuntu)
tags: added: kernel-fs
Changed in linux (Ubuntu):
status: Confirmed → Triaged
Revision history for this message
Rocko (rockorequin) wrote :

Cool, I see the patch is in 4.0-rc3 now (commit 6d65261a09adaa374c05de807f73a144d783669e).

I applied it to 3.19.1-generic and can confirm it works as advertised.

Revision history for this message
stop (whoopwhoop) wrote :

Seems to be working now on my arch machine.. Great work!!!

It noticed something though:
When I move a (large) file to ecryptfs folder this now seems to happen instantly. As I remember it (on ext4 machines) moving files would take a while and show a progress bar, like you see when moving to another partition.
I don't think files are being moved unencrypted but still I wonder, is there an explanation for this?

Revision history for this message
Pratomo Asta Nugraha (pratomoasta) wrote :

Linux pratomo-X450JB 4.5.0-040500rc7-generic #201603061830 SMP Sun Mar 6 23:33:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Revision history for this message
Rolf Leggewie (r0lf) wrote :

Can this ticket be closed?

Revision history for this message
Rocko (rockorequin) wrote :

It has been fixed for me since kernel 4.0, so yes, I think it can be closed now.

Changed in ecryptfs:
status: Confirmed → Opinion
Alexej (nebu0email.tg)
Changed in ecryptfs:
status: Opinion → Fix Released
Changed in linux (Ubuntu):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.