remounting breaks size reporting and rsync

[Benjamin Gemmill]

Sorry to resurrect an old bug, it looks like a fix applied a while back may end up breaking rsync.

For reference, it looks like the decision in this bug fix was to pass on the encrypted file size on disk to stat calls looking at the unencrypted mount:
https://bugs.launchpad.net/ecryptfs/+bug/390833

For reference, to reproduce:
mkdir /mnt/raw
mkdir /mnt/decrypted
dd if=/dev/urandom of=file.out2 bs=1MB count=100
mount -t ecryptfs -o ecryptfs_passthrough=n,no_sig_cache,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_xattr,ecryptfs_enable_filename_crypto=y,passphrase_passwd=SECRET,ecryptfs_fnek_sig=SECRET /mnt/raw /mnt/decrypted
rsync file.out2 /mnt/decrypted/

stat /mnt/decrypted/file.out2
# you get size 100000000

umount /mnt/decrypted/
mount -t ecryptfs -o ecryptfs_passthrough=n,no_sig_cache,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_xattr,ecryptfs_enable_filename_crypto=y,passphrase_passwd=SECRET,ecryptfs_fnek_sig=SECRET /mnt/raw /mnt/decrypted

stat /mnt/decrypted/file.out2
# you get size 100003840

What this means in practice is any additional rsyncs to that target after remounting will see the file size as different than the original, and issue a full new copy. My use case is ecryptfs over a remote file system, so rsync is no better than cp.

Would it be possible to make file size reporting consistent, and reflective of the size of the file as a reader would see it?

[Benjamin Gemmill (benjamin-gemmill)]

Running 16.04

$ uname -a
Linux spore 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[Benjamin Gemmill (benjamin-gemmill)]

After experimenting with different flags, it looks like the flag ecryptfs_xattr exposes the issue.

[Tyler Hicks (tyhicks)]

Verified in 4.8-rc2.

[Tyler Hicks (tyhicks)]

Yes, it is caused by the ecryptfs_xattr_metadata option. A workaround would be to not store the crypto metadata in the xattr and use the default of storing the metadata in the file contents.

The ecryptfs_xattr_metadata option is not the default and, therefore, is not as widely used/tested. This is a bug that needs to be fixed but I'd recommend not using ecryptfs_xattr_metadata.

[Benjamin Gemmill (benjamin-gemmill)]

Hi Tyler, thanks for commenting on this.

I'm trying to get encryptfs going over a remote filesystem, and xattrs are the only practical place to put crypto headers due to file reading speeds:
https://www.amazon.com/clouddrive/

And in particular, if you go to read any part of a file, you have to get the whole file:
https://developer.amazon.com/public/apis/experience/cloud-drive/content/nodes

You can imagine in a directory with many things, or even just a few huge things, an 'ls -al' can make the system start downloading a lot while also appearing to hang.

To get ecryptfs working practically on this kind of remote filesystem, we'd have to use xattrs.

Do you know what would be involved for a fix? I'd be happy to contribute if you could point me in the right direction.

[Tyler Hicks (tyhicks)]

The bug is in ecryptfs_read_and_validate_xattr_region(). It calls ecryptfs_getxattr_lower(), which is returning -34 (ERANGE). With ecryptfs_getxattr_lower() failing, it means that we can't retrieve the proper file size from the xattr metadata and can't set an an accurate value in the kstat buffer.

[Jason Xing (wlxing)]

It also affects me. I tested it on 14.04 (64-bit) and 16.04 (64-bit).

Verified in 4.2.0-27-generic and 4.8.0-36-generic.

[Jason Xing (wlxing)]

Also verified in 3.3.8

I compile the kernel (3.3.8) on Ubuntu 14.04 and Centos 7-1511.

After I mount ecryptfs with ecryptfs_xattr option, it will terminate if I copy some file in it. First time I copy files, it will be killed. Next time I copy files, it will hang and do nothing.

Test outputs:
$ sudo mount -t ecryptfs secure raw -o ecryptfs_xattr
[...]
$ cp file.out2 raw
Killed

I test 3.3.8 kernel on different machines including VM-ware, the outputs are all the same!

[Jason Xing (wlxing)]

I'm working on this bug issue, so I reassign to me. The maintainers of eCryptfs are very busy, if they want to write the patch, that would be great!

[Jason Xing (wlxing)]

Tyler, if you are writing the patch, please tell me, I will assign it to you.

[Jason Xing (wlxing)]

I tested different kernels and two filesystem (xfs and ext4). It turns out that we cannot read less than value_size (which is 81 in ecryptfs) buffer when doing ecryptfs_read_and_validate_xattr_region(). In ecryptfs_read_and_validate_xattr_region() function, "file_size" array is the buffer we're going to read from xatte region and write. The reasonable solution is read 4096 instead of 16 from xattr region.

Any thoughts?

[Jason Xing (wlxing)]

Hello Benjamin Gemmill,

I'm done with the patch (see the attachment). It can be applied in the latest linus' tree and 4.8.17 kernel.If you have any idea or advice, please tell me. Thanks.

[Benjamin Gemmill (benjamin-gemmill)]

Hi Jason, thanks for taking this. I'm happy to test in the coming week or so, we also need to get Tyler's thoughts since he's going to need to sign off too.

[Jason Xing (wlxing)]

Sure. But maybe we need to take more time since he is recently very busy. If this bug doesn't have much affect on you, we could wait. Otherwise, you can test it and apply it in your kernel. After all, it can solve your problem though it may not be the best way.