Cannot watch more than 8192 directories

[Michele Costantino Soccio]

If you have a large filesystem do backup (even just a huke home) TimeVault will say that it cannot watch so many directories.
Actually the message is not clear about that: does it mean the extra-directories will be backed up later (not respecting the time interval set in preferences) or does it mean the extra-directories will not be backed up?

[Ali Sattari (corelist)]

Hi,
I am on Gutsy and using TimeVault 0.7.5-1 having the same problem!
I get: [Too many directories (8680>8192): Some changes may be lost]
And it seems that TV doesn't take snapshots at all :(

[geekphreak (geekphreak)]

You can get around this by editing watcher.py in /usr/lib/python2.5/site-packages/TimeVault
By default, TV opens a file and reads a number from it, parses is as integer and uses that number as max num of dirs to watch

Instead of that, if you set
MAX_USER_WATCH = 1000000
and then in constructor's try block, comment out lines opening and closing the file with the number of dirs to watch, and simply have
self.maxUserWatches = MAX_USER_WATCH

TV will now be able to handle 1000000 files, or more, as long as you change the number to whatever you need.

[Daniel Stiner (danstiner)]

Geekphreak, are you sure this fixes the problem? It seems to me that this is a workaround in TV only. Wouldn't "inotify" still only be able to handle 8192 directories? Maybe I'm wrong, but this quote from www.beagle-project.org/Troubleshooting recommends editing the inotify file in /proc/ directly to upgrade the amount of watchable directories at the cost of increased ram usage. Maybe even where TimeVault sees this limit exceeded, it can ask the user for permission to increase the maximum watches.

The default number of watches is 8192. 16384 is a good value for most people, and 32768 is probably more than enough. Using additional watches does increase the amount of memory used inside the kernel, but increasing the number does not affect the amount of memory if they aren't used.

To change the limit:

sudo echo 16384 > /proc/sys/fs/inotify/max_user_watches

[Daniel Stiner (danstiner)]

As I am upgrading to the Hardy Heron beta, I noticed the following change in /etc/sysctl.conf:

# Increase inotify availability
fs.inotify.max_user_watches = 524288

So this bug will almost certainly be resolved by default Hardy Heron, and for anyone else, the above method of increasing /proc/sys/fs/inotify/max_user_watches works for sure, and the hack from geekphreak may work as well.

[geekphreak (geekphreak)]

Daniel,

I looked briefly at /proc/sys/fs/inotify/max_user_watches before I came up with the watcher.py change and I actually made a change to inotify file, however, right after saving changes, I did a less on that file only to find the number go back to 8192. I may have done something wrong there.
You're probably right about the inotify's ability to watch files, but I am glad that this will be resolved in Hardy.
Is TV going to continue as an active project? Haven't heard much about it other than complaints lately. I love the idea of TimeMachine-like backup, but I was afraid that people have given up on TimeVault. Do you know anything about the future of TV?

[Daniel Stiner (danstiner)]

I wish I could tell you TimeVault is active and ready to be included in Hardy Heron, but sadly the project indeed seems to be dying (last edit of trunk was 2007-10-27). I haven't been involved in the project at all yet, but this may be a good excuse to do so. I think if the interface was just cleaned up a bit (maybe a simple page under System->Preferences somewhat like Time Machine's), TimeVault would be good enough to be included in the next version of Ubuntu by default. I won't have a whole lot of time to help, but it would be a good excuse to learn python, backup my computer, etc.

To clarify this bug though, the echo into /proc/sys/fs/inotify/max_user_watches trick worked for me after restarting the timevault service, but it /will not/ persist when rebooting, it must be repeated each time. Though in hardy heron, I now have a watch limit of 524288.

[geekphreak (geekphreak)]

Yeah, I noticed that, too. I can't really promise much of my time either, other than testing, perhaps, but I love Ubuntu and it's great to see how much it matured over such a short period of time. I use partimage for disk imaging and I hoped that TimeVault would be a good solution for individual files and folders, sort of an addition to partimage in everyone's backup arsenal.
Perhaps this should be raised in forums or other appropriate places. I already voted it up at Ubuntu Brainstorm site.

[Daniel Stiner (danstiner)]

A fix was committed into the new Hardy Heron release for more than 8192 inotify watches.