ecryptfs never releases the lower file

[Serge Hallyn]

Ecryptfs never releases the lower file. That means that no inotify events are ever issued for the underlying encrypted fs. This in turn means that if the encrypted directory is on Ubuntu One, changes are not synced until after a forced rescan or after a reboot.

Commit 332ab16f830f59e7621ae8eb2c353dc135a316f6 by tyhicks adds reference counting (resulting in a fput of the lower file on final umount) to the underlying files. This should allow an ecryptfs filesystem located under Ubuntu One to be properly synced. Could we get that into natty-proposed?

[Serge Hallyn (serge-hallyn)]

Unfortunately the patch may not be enough. This bug could be deemed depending on bug 723518.

[Tyler Hicks (tyhicks)]

Serge - the fix for this is simple. See the patch attached to bug 723518.

I don't think that it depends on Commit 332ab16f830f59e7621ae8eb2c353dc135a316f6, but I can't say for sure.

[Serge Hallyn (serge-hallyn)]

Quoting Tyler Hicks (<email address hidden>):
> Serge - the fix for this is simple. See the patch attached to bug
> 723518.
>
> I don't think that it depends on Commit
> 332ab16f830f59e7621ae8eb2c353dc135a316f6, but I can't say for sure.

Ok, thanks - I was holding of yesterday afternoon after you mentioned
that there appeared to still be a bug. I'll test that patch on
monday.

[Serge Hallyn (serge-hallyn)]

To give this a more repeatable test, here is a python file which can be used to verify. When I do

  (sudo bash)
  mkdir src tgt
  mount -t ecryptfs src tgt
  (in window 1):
  python inotify_test.py
  (in window 2):
  echo hi > hi
  echo there >> hi

I see:
root@ecryptfs-natty-amd64:~# python inotify_test.py
Create: /home/serge/src/hi
MOdified: /home/serge/src/hi
MOdified: /home/serge/src/hi

so the modified inotify events are sent, but the closed-write are not.

=========================
import os
from pyinotify import WatchManager, Notifier, ThreadedNotifier, EventsCodes, ProcessEvent

wm = WatchManager()

mask = EventsCodes.ALL_FLAGS['IN_DELETE'] | EventsCodes.ALL_FLAGS['IN_CREATE'] | EventsCodes.ALL_FLAGS['IN_MODIFY'] | EventsCodes.ALL_FLAGS['IN_CLOSE_WRITE'] # watched events

class PTmp(ProcessEvent):
    def process_IN_CREATE(self, event):
        print "Create: %s" % os.path.join(event.path, event.name)

    def process_IN_DELETE(self, event):
        print "Remove: %s" % os.path.join(event.path, event.name)

    def process_IN_MODIFY(self, event):
        print "MOdified: %s" % os.path.join(event.path, event.name)

    def process_IN_CLOSE_WRITE(self, event):
        print "closed write: %s" % os.path.join(event.path, event.name)

notifier = Notifier(wm, PTmp())

wdd = wm.add_watch('/home/serge/src/', mask, rec=True)

while True: # loop forever
    try:
        # process the queue of events as explained above
        notifier.process_events()
        if notifier.check_events():
            # read notified events and enqeue them
            notifier.read_events()
        # you can do some tasks here...
    except KeyboardInterrupt:
        # destroy the inotify's instance on this interrupt (stop monitoring)
        notifier.stop()
        break
========

[Serge Hallyn (serge-hallyn)]

With the cherrypicked commit from upstream, the results are:

root@ecryptfs-natty-amd64:~# python inotify_test.py
closed write: /home/serge/src/hi
closed write: /home/serge/src/
Create: /home/serge/src/yo
MOdified: /home/serge/src/yo
closed write: /home/serge/src/yo
MOdified: /home/serge/src/yo
closed write: /home/serge/src/yo
MOdified: /home/serge/src/yo
closed write: /home/serge/src/yo

[Serge Hallyn (serge-hallyn)]

I"ve marked this high priority since it prevents ecryptfs from being used with Ubuntu One, and there is no workaround - all you can do is to frequently reboot.

[Serge Hallyn (serge-hallyn)]

(of course if you disagree please do change the priority :)

[Serge Hallyn (serge-hallyn)]

Drat, running with my cherrypicked patch for some time now, but while inotify events are being sent, Ubuntu One does not seem to be syncing the changes.

[Andy Whitcroft (apw)]

@Serge -- has this progressed at all upstream? This patch is pretty big, but you imply its not sufficient yet?

[Serge Hallyn (serge-hallyn)]

@Andy

Hm, even normal files dont' seem to be getting pushed right now, so I'm afraid I can't confirm. The best I can confirm is that when I stop u1sdtool and the syncdaemon and then u1sdtool -c, ecryptfs backing files which have been modified then get pushed.

[Brad Figg (brad-figg)]

@Serge,

Is this issue fully resolved in Oneiric and Precise kernels? Is there still a desire to have it backported to Natty? As Andy stated, this is a fairly large patch to backport.

[Stefan Bader (smb)]

Setting to incomplete to mark this waiting on some feedback.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]