ecryptfs does not work properly over nfs, cifs, samba, WebDAV, or aufs

Reported by Dustin Kirkland  on 2008-10-03
226
This bug affects 34 people
Affects Status Importance Assigned to Milestone
eCryptfs
Wishlist
Unassigned
ecryptfs-utils (Ubuntu)
Wishlist
Unassigned

Bug Description

http://sourceforge.net/tracker/index.php?func=detail&aid=1639562&group_id=133988&atid=728799

ecryptfs over nfs: kernel BUG at fs/nfs/write.c:387!

Submitted By:
mir - mir07 Date Submitted:
2007-01-19 13:37
Last Updated By:
mhalcrow - Comment added Date Last Updated:
2007-06-08 16:57
Number of Comments:
3 Number of Attachments:
0
Category: (?)
ecryptfs (kernel module) Group: (?)
None
Assigned To: (?)
Michael Halcrow Priority: (?)
8
Status: (?)
Open Resolution: (?)
Accepted
Summary: (?)
ecryptfs over nfs: kernel BUG at fs/nfs/write.c:387! Private: (?)
No
after applying your patch from bug-report 1637031 to the 2.6.20-rc4-mm1
kernel we encounter the following kernel bug after copying about 1gb (9500
files).

Jan 19 14:53:20 hostname kernel: ------------[ cut here ]------------
Jan 19 14:53:20 hostname kernel: kernel BUG at fs/nfs/write.c:387!
Jan 19 14:53:20 hostname kernel: invalid opcode: 0000 [#1]
Jan 19 14:53:20 hostname kernel: SMP
Jan 19 14:53:20 hostname kernel: last sysfs file: /block/loop7/removable
Jan 19 14:53:20 hostname kernel: Modules linked in: cbc blowfish ecb
blkcipher ecryptfs nfs lockd nfs_acl sunrpc ipv6 button ac battery
dm_snapshot dm_mirror dm_mod loop tsdev floppy rtc parport_pc parport
serio_raw pcspkr i2c_piix4 i2c_core shpchp pci_hotplug psmouse intel_agp
agpgart evdev ext3 jbd mbcache ide_cd cdrom ide_disk generic uhci_hcd
usbcore pcnet32 mii piix ide_core thermal processor fan
Jan 19 14:53:20 hostname kernel: CPU: 0
Jan 19 14:53:20 hostname kernel: EIP: 0060:[<d0adfeda>] Not tainted
VLI
Jan 19 14:53:20 hostname kernel: EFLAGS: 00010246 (2.6.20-rc4-mm1 #1)
Jan 19 14:53:20 hostname kernel: EIP is at nfs_writepage_setup+0xea/0x3eb
[nfs]
Jan 19 14:53:20 hostname kernel: eax: ffffffef ebx: ffffffef ecx:
c75d9c00
edx: c72d03c0
Jan 19 14:53:20 hostname kernel: esi: c75d9c00 edi: ce7d9a34 ebp:
ce7d9b64
esp: ce17dbe4
Jan 19 14:53:20 hostname kernel: ds: 007b es: 007b fs: 00d8 gs: 0033
ss: 0068
Jan 19 14:53:20 hostname kernel: Process dirvish (pid: 2267, ti=ce17c000
task=c1275550 task.ti=ce17c000)
Jan 19 14:53:21 hostname kernel: Stack: 00000000 00001000 220424c6 912f9c05
00000000 c11b1120 ce7ecb00 00000000
Jan 19 14:53:21 hostname kernel: d09cd245 00000000 00001000 00000000
c11b1120 c11b1120 d0ae065f 00001000
Jan 19 14:53:21 hostname kernel: ce17dd00 c9703ff8 d09d63cc ce17dcf8
ce17dd08 00000000 cfaf6a40 ce7ecb00
Jan 19 14:53:21 hostname kernel: Call Trace:
Jan 19 14:53:21 hostname kernel: [<d09cd245>]
blkcipher_walk_done+0x12d/0x1dc [blkcipher]
Jan 19 14:53:21 hostname kernel: [<d0ae065f>] nfs_updatepage+0x13c/0x1a1
[nfs]
Jan 19 14:53:21 hostname kernel: [<d09d63cc>]
crypto_cbc_encrypt+0x132/0x146 [cbc]
Jan 19 14:53:21 hostname kernel: [<d0ad779a>] nfs_commit_write+0x26/0x35
[nfs]
Jan 19 14:53:21 hostname kernel: [<d0a9289d>]
ecryptfs_commit_lower_page+0x25/0x57 [ecryptfs]
Jan 19 14:53:21 hostname kernel: [<d0a944f3>]
ecryptfs_write_out_page+0x27/0x71 [ecryptfs]
Jan 19 14:53:21 hostname kernel: [<d0a95052>]
ecryptfs_encrypt_page+0x3db/0x449 [ecryptfs]
Jan 19 14:53:21 hostname kernel: [<d0a92fed>]
ecryptfs_commit_write+0x1f9/0x377 [ecryptfs]
Jan 19 14:53:21 hostname kernel: [<d0a92df4>]
ecryptfs_commit_write+0x0/0x377 [ecryptfs]
Jan 19 14:53:21 hostname kernel: [<c014aa44>]
generic_file_buffered_write+0x3e1/0x5ec
Jan 19 14:53:21 hostname kernel: [<c0128b59>] do_timer+0x5c1/0x711
Jan 19 14:53:21 hostname kernel: [<d0a70118>]
rpcauth_lookup_credcache+0x107/0x1f4 [sunrpc]
Jan 19 14:53:21 hostname kernel: [<c01249a1>] current_fs_time+0x4f/0x58
Jan 19 14:53:21 hostname kernel: [<c0277f3e>] tcp_v4_rcv+0x8c6/0x938
Jan 19 14:53:21 hostname kernel: [<c014b170>]
__generic_file_aio_write_nolock+0x521/0x59a
Jan 19 14:53:21 hostname kernel: [<c014b23e>]
generic_file_aio_write+0x55/0xc6
Jan 19 14:53:21 hostname kernel: [<c0163f84>] do_sync_write+0xc7/0x10a
Jan 19 14:53:21 hostname kernel: [<c015570b>]
__handle_mm_fault+0x74f/0x7a0
Jan 19 14:53:21 hostname kernel: [<c0131c1e>] wake_bit_function+0x0/0x44
Jan 19 14:53:21 hostname kernel: [<c0163ebd>] do_sync_write+0x0/0x10a
Jan 19 14:53:21 hostname kernel: [<c01647c3>] vfs_write+0xa8/0x154
Jan 19 14:53:21 hostname kernel: [<c0164dd0>] sys_write+0x41/0x67
Jan 19 14:53:21 hostname kernel: [<c0103d36>] sysenter_past_esp+0x5f/0x85
Jan 19 14:53:21 hostname kernel: =======================
Jan 19 14:53:21 hostname kernel: Code: d5 00 00 00 85 f6 0f 84 94 00 00 00
90 0f ba 6e 2c 00 19 c0 8b 56 18 8d 87 ec 00 00 00 89 f1 e8 67 76 6e ef 83
f8 ef 89 c3 75 04 <0f> 0b eb fe 85 c0 75 55 83 bf 00 01 00 00 00 75 2a 89
e8 e8 b2
Jan 19 14:53:21 hostname kernel: EIP: [<d0adfeda>]
nfs_writepage_setup+0xea/0x3eb [nfs] SS:ESP 0068:ce17dbe4

Dustin Kirkland  (kirkland) wrote :

Date: 2007-06-08 16:57
Sender: mhalcrowProject Admin
Logged In: YES
user_id=729064
Originator: NO

FYI, Erez Zadok recently confirmed a problem with stacked filesystems,
writepages, and NFS and has proposed a fix, which we need to evaluate.

Dustin Kirkland  (kirkland) wrote :

Date: 2007-02-23 00:06
Sender: mhalcrowProject Admin
Logged In: YES
user_id=729064
Originator: NO

Dmitriy Monakhov moved the O_LARGEFILE into the flags under all
circumstances. This may have fixed the problem.

Dustin Kirkland  (kirkland) wrote :

Date: 2007-01-19 14:49
Sender: mhalcrowProject Admin
Logged In: YES
user_id=729064
Originator: NO

This is an explicit call to BUG by NFS; while trying to do a radix insert
of an encrypted page, it looks like the page already exists in the NFS
inode page map when it shouldn't. If NFS were to just ignore the issue
rather than bug out, this may work fine. In any event, this is yet another
NFS corner case that we will need to scrutinize.

Changed in ecryptfs:
importance: Undecided → High
status: New → Confirmed
Changed in ecryptfs-utils:
importance: Undecided → High
status: New → Confirmed

Tyler-

What's the status of this bug in the upstream kernel code?

I seem to remember you saying that you had a handle on the fixes. Have those been merged upstream? If so, can you point to the git commits, in case others want to try and backport them to their kernels?

:-Dustin

Tyler Hicks (tyhicks) wrote :

Unfortunately, I haven't gotten to work on this bug much in the last month. I do not yet have a working patch set, but things were getting close for NFS support. I will try to devote some time this week to more work on NFS support.

Changed in ecryptfs-utils:
assignee: nobody → tyhicks
Changed in ecryptfs:
assignee: nobody → tyhicks
status: Confirmed → Triaged
Changed in ecryptfs-utils:
status: Confirmed → Triaged
Changed in ecryptfs:
status: Triaged → Confirmed
Changed in ecryptfs-utils (Ubuntu):
status: Triaged → Confirmed
tags: added: kernel
Dustin Kirkland  (kirkland) wrote :

Setting to in-progress.

Tyler has some working prototypes for this now.

Changed in ecryptfs:
status: Confirmed → In Progress
Changed in ecryptfs-utils (Ubuntu):
status: Confirmed → In Progress
GeekSmith (lixo-geeksmith) wrote :

Don't know if this helps, but I hit a different BUG (in inode.c) every time I try to use rsync/SSH to back up data to an ecryptfs directory. Trace is attached.

Paulo J. S. Silva (pjssilva) wrote :

I have found a workaround (at least for the last 5 minutes):

1) Uninstall ecryptfs-utils in the client machine and install sshfs
2) Mount Private using sshfs.

It works for me up to now.

summary: - ecryptfs does not work properly over nfs, cifs, or samba
+ ecryptfs does not work properly over nfs, cifs, samba, or aufs

I'm not sure if this is related, but I tried ecryptfs with .private on a Rackspace cloud storage filesystem (cloudfuse) and it would crash the file system every time (http://github.com/redbo/cloudfuse/issues#issue/4 ). I'm sure this is a problem on the file system side (which shouldn't just crash), but this obviously sounds related due to the observations for other remote file systems. At worst, consider it just another data point.

Let me know if I can help testing in any way.

Dan (danser) wrote :

Not sure if this is the same bug or a different one, but I find ecryptfs generally works smoothly over NFS, except that the second and subsequent "ls" in any folder doesn't return any files. I don't see anything in dmesg to suggest a problem.

This sounds like the bug reported two years ago here: http://<email address hidden>/msg00265.html

Corni (comaddcor) wrote :

I see the exact same behavior as Dan above me. In any folder the first ls (or any other command which requires file system data) works fine where the second and all subsequent calls fail. Nothing in any log file nor in dmesg as far as I can tell.
Reading through the comments I'm also not sure if this is a separate bug or not.

I'm running:
Client:
Kernel: 2.6.36-ARCH
nfs-utils: 1.2.2-3
ecryptfs-utils: 83

Server:
Kernel: 2.6.32-25-generic
nfs-utils: 1.2.0-4
ecrytpfs-utils 83

Hey Tyler,

What's the current state of your eCryptfs-on-NFS work-in-progress?

Dustin

Hello,

I tried running "ls" on a directory mounted over NFS. This works fine for the first time but for subsequent attempts "ls" does not show any output.

However, when I repeat the same operation over a mount over local filesystems, ls works fine.

Is this a known problem?

The ecryptfs README says that stability with NFS is still under progress. So is the problem with stability related to the stacking model that ecryptfs is based on, or is it due to the encryption/decryption complexity?

Thanks,
Pallavi

Confirming the problem as reported by Dan (#10), Corni (#11) and Pallavi (#13):
Mounting over NFS: First "ls" after the mount works, subsequent "ls" return an empty directory.

FYI, Tyler is getting closer to solving this bug, with a few patches
having been pushed to Linus' kernel tree.

Tyler, could you perhaps give us a brief update on where we are now?

--
:-Dustin

On the NFSv3 client front, I have all of the needed eCryptfs patches upstream (as of 2.6.39-rc5), but there is still an NFS client patch that needs to be accepted upstream. The NFS client maintainer has nack'ed it, but I plan on making a couple small changes and resubmitting it soon.

http://www.spinics.net/lists/linux-nfs/msg20820.html

summary: - ecryptfs does not work properly over nfs, cifs, samba, or aufs
+ ecryptfs does not work properly over nfs, cifs, samba, WebDAV, or aufs
Dustin Kirkland  (kirkland) wrote :

@Tyler, what's the status on that NFS client patch? Any progress?

Tyler Hicks (tyhicks) on 2011-12-13
Changed in ecryptfs:
status: In Progress → Triaged
importance: High → Wishlist
Tyler Hicks (tyhicks) wrote :

eCryptfs on top of NFS isn't going to happen in the short term. Nearly all of the eCryptfs fixes are now in place, except for one last sticking point. The NFS maintainer is requiring that eCryptfs fix up a struct nameidata and pass it down into NFS, but that can get ugly and is something that the VFS maintainer is against. The VFS maintainer has alluded to a patch set that he is working on to do away with the nameidata passing, but it is not currently posted anywhere. When that happens, eCryptfs has a much better chance at supporting NFS.

Changed in ecryptfs-utils (Ubuntu):
status: In Progress → Triaged
importance: High → Wishlist

@Tyler, thanks for your status update!

@Tyler: does this last remaining NFS issue also effect the other protocols like WebDAV or CIFS? If not, would it be possible to release a fix which supports all the other protocols except NFS?

On 2012-01-16 20:47:27, Chris wrote:
> @Tyler: does this last remaining NFS issue also effect the other
> protocols like WebDAV or CIFS? If not, would it be possible to release a
> fix which supports all the other protocols except NFS?

I'm not certain as my focus was just on NFS. CIFS looks to make heavy
use of nameidata, which is the basis of the NFS holdup. I have no idea
about WebDAV.

Tyler Hicks (tyhicks) on 2012-02-08
Changed in ecryptfs:
assignee: Tyler Hicks (tyhicks) → nobody
Changed in ecryptfs-utils (Ubuntu):
assignee: Tyler Hicks (tyhicks) → nobody
status: Triaged → Invalid
Don (do1) wrote :

Will this bug be fixed, or it is already fixed?

Tyler Hicks (tyhicks) wrote :

On 2012-12-13 19:10:06, Don wrote:
> Will this bug be fixed, or it is already fixed?

It isn't fixed. Most of the work for NFS is done, but there's still more
to do. It is marked wishlist because no one is working on it and it
isn't considered a priority at this time.

In the future, please check the bug status rather than asking if it is
fixed. Thanks!

Disconnect (ubbugs-sigkill) wrote :

FYI it does seem to work over afp/netatalk. It isn't a workaround for most uses (including mine unfortunately) but it might help some people.

zombi (tofvixok) wrote :

for me doing a bindfs mount on top of the webdav mount and then mounting the ecryptfs on the bindfs mount worked.

mount.davfs https://dav.box.com box
bindfs box/dav /crypt
mount -t ecryptfs /crypt /decrypt

Max Wolf (max-2) wrote :

Hi zombi

>for me doing a bindfs mount on top of the webdav mount and then mounting the ecryptfs on the bindfs mount worked.

does this work with "distributed editing"?
e.g
- mounting from device A
- mounting from device B
- edit on device A

are the changes properly reflected on device B ?

Johnny Patino (patino-johnny) wrote :

As a work around I successfully shared my encrypted partition using CIFS instead. I do have to share the partition with Windows boxes as well and so far everything seems to work fine. Performance is good and usability is seemless for both linux and windows clients. I just need to look into security, but the configurability has been simple so far.

Johnny Patino (patino-johnny) wrote :

By the way the title of this bug seems misleading as ecryptfs seems to work fine for cifs, samba, and even sshfs. Not sure about WebDAV or aufs .

KrautOS (krautos) wrote :

@Johnny: You mean eCryptFS runs better on SAMBA than on NFS on your side? Can you give us some informations about your setup?

Johnny Patino (patino-johnny) wrote :
Download full text (3.6 KiB)

Hello KrautOS. Here's some of my setup info, please let me know if you need more details. Thanks.

vmhost:
vSphere 5.5

vmmachine (nfs-server):
>uname -a
Linux xxx 3.11.0-18-generic #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 13.10
Release: 13.10
Codename: saucy
>pvdisplay
  --- Physical volume ---
  PV Name /dev/md0
  VG Name RaidVG
  PV Size 3.63 TiB / not usable 4.81 MiB
  Allocatable yes (but full)
  PE Size 4.00 MiB
  Total PE 951549
  Free PE 0
  Allocated PE 951549
  PV UUID EIJ1f2-N30Y-TRRk-dkW7-qtqC-MgIx-UXyx4n
>vgdisplay
  --- Volume group ---
  VG Name RaidVG
  System ID
  Format lvm2
  Metadata Areas 1
  Metadata Sequence No 2
  VG Access read/write
  VG Status resizable
  MAX LV 0
  Cur LV 1
  Open LV 1
  Max PV 0
  Cur PV 1
  Act PV 1
  VG Size 3.63 TiB
  PE Size 4.00 MiB
  Total PE 951549
  Alloc PE / Size 951549 / 3.63 TiB
  Free PE / Size 0 / 0
  VG UUID vfKSY2-RbHq-ZXih-EGXD-JCYw-IKEq-7J2DZZ
>lvdisplay
  --- Logical volume ---
  LV Path /dev/RaidVG/NFSLV
  LV Name NFSLV
  VG Name RaidVG
  LV UUID Ykdggh-D769-sQmp-ilzL-1bpL-a8eF-MEOvqw
  LV Write Access read/write
  LV Creation host, time nfs-server, 2014-03-02 12:54:24 -0700
  LV Status available
  # open 1
  LV Size 3.63 TiB
  Current LE 951549
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 252:2
>/etc/fstab
/dev/mapper/RaidVG-NFSLV /srv/nfs ext4 defaults 0 2
/srv/nfs /srv/nfs ecryptfs defaults 0 0
/dev/mapper/cryptswap1 none swap sw 0 0
>package versions (amd64):
mdadm 3.2.5-5ubuntu2
lvm2 2.02.98-6ubuntu1
ecryptfs 103-0ubuntu2
nfs-kernel-server 1:1.2.8-2ubuntu2
>/etc/exports
/srv/nfs 192.168.2.0/25(secure,rw,sync,wdelay,hide,no_subtree_check,secure_locks)
>exportfs
/srv/nfs 192.168.2.0/25
>syslog
kernel: [ 14.559732] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
kernel: [ 14.584691] NFSD: starting 90-second grace period (net ffffffff81cd2dc0)
rpc.mountd[1209]: Version 1.2.8 starting

nfs-client:
>/etc/fstab
nfs-server:/srv/nfs /mnt/nfs nfs4 rsize=8192,wsize=8192,timeo=14,intr 0 0
>package versions
nfs-common 1:1.2.5-3ubuntu3.1
>sudo showmount -e nfs-server
Export list for nfs-server:
/srv/nfs 192.168.2.0/25
>sudo mount /mnt/nfs
mount.nfs4: mounting nfs-server:/srv/nfs failed, reason given by server:
  No such file or directory
> nfs-server syslog
nfs-server rpc.mountd[1209]: qword_eol: fp...

Read more...

KrautOS (krautos) wrote :

I just done a server with NFS(w/o Kerberos)+NIS on basis of the upcoming Ubuntu 14.04 with server and desktop installations. eCryptfs seems to work, as i can log into an eCryptfs protected home directory on the graphical login screen (lightdm needs some config changes though).

At first i thought it would be just able with an extra bind mount like:
mount -t nfs4 -o proto=tcp,port=2049 SERVER:/home /home.nfs
mount -o bind /home.nfs /home

But it's also working with a direct mount of NFSv4 on /home, i only missed to install "ecryptfs-utils" on the second desktop i was testing on, as i seems to not be installed when you don't vote for encrypting the users home folder.

Now i need some rough test cases, as i just did login with a fresh account and that may not be the corner cases. Does somebody have any clues on?

@Johny: as i just looked at your setup it seems, you did something like a mixed thing between NFSv3 and NFSv4. That's you can export the folder is really strange as eCryptfs should be transparent to NFS, it's only that seems to have problems on top of NFS. For your setup have a look on https://help.ubuntu.com/community/NFSv4Howto

@Tyler: It's been a while since your last comment, so do you mind to give us some update on your point of view?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments