handle reaching inotify max watches better

Bug #761578 reported by David Ayers
30
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Nautilus
New
Undecided
Unassigned
Ubuntu One Client
Confirmed
Medium
Unassigned
ubuntuone-client (Ubuntu)
Confirmed
Medium
Roberto Alsina

Bug Description

Binary package hint: ubuntuone-client

I'm testing the maverick proposed version of the client to test if the crash of bug #744383 if fixed. Since I now have 20GB instead of the 2GB I had to move some folders with many files and some very large files into the Ubuntu one directory. The sync seemed to stop an a tail -f on the sync deamon log failed with no space left on device yet the disks still have enough space and enough inodes. Searching the web indicates that it seems that kernel memory for watchers was exhausted:

ayers@schiefer:~$ sudo sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192
ayers@schiefer:~$ sudo sysctl fs.inotify.max_user_watches=16384
fs.inotify.max_user_watches = 16384
After executing this the tail worked again.

I suggest some mechanism to limit the amount of watchers needed or to at least notify the user when the limit is exceeded as this effects other processes because system wide resources get exhausted.

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: ubuntuone-client 1.4.6-0ubuntu3
ProcVersionSignature: Ubuntu 2.6.35-28.50-generic 2.6.35.11
Uname: Linux 2.6.35-28-generic x86_64
Architecture: amd64
Date: Fri Apr 15 11:47:11 2011
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Release amd64 (20101007)
PackageArchitecture: all
ProcEnviron:
 PATH=(custom, user)
 LANG=de_AT.utf8
 SHELL=/bin/bash
SourcePackage: ubuntuone-client
UbuntuOneSyncdaemonExceptionsLog:

UbuntuOneUserSyncdaemonConfig:
 [bandwidth_throttling]
 read_limit = 2097152
 write_limit = 2097152
 on = False

Revision history for this message
David Ayers (ayers) wrote :
Revision history for this message
Lucio Torre (lucio.torre) wrote :

In a default ubuntu install i get:

$ sudo sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 524288

Which is enough for syncdaemon to work.

It is true that we could do something better than just fail when the user runs out of watches, so i am marking this as whishlist.

Changed in ubuntuone-client (Ubuntu):
status: New → Triaged
importance: Undecided → Wishlist
tags: added: chicharra
Revision history for this message
David Ayers (ayers) wrote :

I get 8192 on Lucid 32-bit, Maverick 32-bit, Maverick 64-bit and Natty 32-bit.

Possibly the value is dependent on some resource of which you have more?

Let me also not that ever since my testing and the tripped value, I have removed that directory from my Ubuntu One folder in the hope that syncing would resume normally, but it didn't.

So there is the potential that tripping this value can put Ubuntu One in a state that doesn't allow it to sync correctly anymore.

summary: - sync daemon need to many watchers
+ handle reaching inotify max watches better
Revision history for this message
Duane Hinnen (duanedesign) wrote :

David,
You can increase that number of watches with the following.

Open /etc/sysctl.conf for editing:
gksudo gedit /etc/sysctl.conf

Add this line to the bottom of the file (If the line exists just change the number):
 fs.inotify.max_user_watches = 524288

thank you,
duanedesign

Revision history for this message
David Ayers (ayers) wrote :

duanedesign, thanks for the hint. For testing purposes I have updated the value and rebooted my system.

Before I had backed up my files, de-registered from my account, wiped Ubuntu one from the system incl. all local ubuntuone related files I could identify, reinstalled ubuntu one, re-registered, waited for the sync from the server to complete.

I then tried to restore the files from the backup into my Ubuntu One directory. The corresponding Sync is taking quite some time but I'll let it run for a few hours.

Revision history for this message
Szymon Danielczyk (danielczyk-szymon) wrote :

Hi guys just to let you know
I've experienced problems with running "less" command because of inotify limit which was
8192 on my machine
It was hard to diagnose as a message I got from "less" was very misleading
like
tail: cannot watch `catalina.out': No space left on device
there was plenty of space and the real reason was inotify watchers had been eaten by ubuntuone ;-)
This can affect many users and a lot of others applications
i so dropbox is displaying a message to the user when inotify limit is reached
I think you should do the same to let the users know what is going on
to avoid unnecessary frustration and save their time

Thanks for making ubuntuone
Szymon

Revision history for this message
shankao (shankao) wrote :

Same thing here on raring.

shankao@maldita:~/www/aftertime$ sudo sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192

And it affects other programs of the system (i.e. tail reverts to polling).

I suggest to either increase the value automatically when installing ubuntuone to accomodate its requirements, decrease these requirements or, at least, notify the users somehow of the situation and the workarounds that they can try.

Roman Yepishev (rye)
tags: added: u1-support
Revision history for this message
shankao (shankao) wrote :

I suggest to raise the importance of this bug regardless of how simple the workaround would be, as it affects other main components in Ubuntu that rely on inotify not exhausted. For example:

- The already commented example of tail (on comment #6).
- Nautilus stops updating the folders state automatically and needs to be done manually (F5)

Julien Funk (jaboing)
tags: added: u1-by-support
Revision history for this message
David Ayers (ayers) wrote :

As the original reporter, I'd also suggest to inform the user about hitting this limit with a pointer to the workaround. I'm syncing about 40K+ files with Ubuntu One (10K+ directories) and if I forget to set this value on one of those machines, U1 of course "silently" does not sync those files that didn't get a watcher. (I say silently since it does report the issue in it's log file, but unless you are looking you will most certainly miss it).

tags: removed: u1-support
Revision history for this message
Julien Funk (jaboing) wrote :

I am raising the priority as this seems like something that affects a number of people.

Changed in ubuntuone-client (Ubuntu):
status: Triaged → Confirmed
tags: added: u1-on-production u1-qa-escalated
Changed in ubuntuone-client (Ubuntu):
importance: Wishlist → Medium
assignee: nobody → Roberto Alsina (ralsina)
Julien Funk (jaboing)
Changed in ubuntuone-client:
status: New → Confirmed
importance: Undecided → Medium
dino99 (9d9)
tags: added: raring
removed: maverick
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.