indicator-messages-service is waking up every 4 seconds adding inotifies on paths that don't exist

[Colin Ian King]

I've observed that an "idle" indicator-messages-services is waking up every 4 seconds on a poll() and doing two inotify_add_watch() calls on paths that don't exist, which wastes power on devices such as phones.

strace shows:

clock_gettime(CLOCK_MONOTONIC, {10117, 224455428}) = 0
poll([{fd=8, events=POLLIN}, {fd=9, events=POLLIN}], 2, 3991) = 0 (Timeout)
clock_gettime(CLOCK_MONOTONIC, {10121, 224516469}) = 0
inotify_add_watch(9, "/usr/local/share/applications", IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF|IN_UNMOUNT|IN_ONLYDIR) = -1 ENOENT (No such file or directory)
inotify_add_watch(9, "/usr/share/ubuntu-touch/applications", IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF|IN_UNMOUNT|IN_ONLYDIR) = -1 ENOENT (No such file or directory)
clock_gettime(CLOCK_MONOTONIC, {10121, 232085521}) = 0
poll([{fd=8, events=POLLIN}, {fd=9, events=POLLIN}], 2, 3984) = 0 (Timeout)
clock_gettime(CLOCK_MONOTONIC, {10125, 222593684}) = 0
inotify_add_watch(9, "/usr/local/share/applications", IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF|IN_UNMOUNT|IN_ONLYDIR) = -1 ENOENT (No such file or directory)
inotify_add_watch(9, "/usr/share/ubuntu-touch/applications", IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF|IN_UNMOUNT|IN_ONLYDIR) = -1 ENOENT (No such file or directory)
clock_gettime(CLOCK_MONOTONIC, {10125, 229094524}) = 0

Can these redundant polling checks be reduced or removed? Polling every 4 seconds is a little bit heavy handed IMHO.

[Colin Ian King (colin-king)]

This shows up in the daily testing too: http://ci.ubuntu.com/power/eventstat/image/3138/machine/6/task/indicator-messages-service/details/

[Sebastien Bacher (seb128)]

Thanks Colin, that's the same issue than bug #1296233

Without improving/fixing/replace inotify we mostly have workaround solution. One would be to create those directories on the default image...

[Colin Ian King (colin-king)]

OK, I understand. My concern is that we do have a lot of processes doing this kind of polling, so the cumulative effect of the same root issue is causing a lot of unwanted wakeups on a "so called" idle system.

[Sebastien Bacher (seb128)]

Right, that's a valid concern. The issue is that some code needs to watch for new files/directories (like the desktop needs to pick new applications when they get installed), and we don't have anything better than inotify to do that at the moment.

Our options seem to:
- work on something better than inotify and start using that (not likely to happen soon I guess)
- accept the wakeups and their cost
- keep working around the issues, either by making code use less watching (with the side effect of gettings bugs/not being notified of the environment changes) or by adding hacks (like creating empty directories that are not needed on the default installation)

[Oliver Grawert (ogra)]

how about you make the indicator (ab)use upstart for this, it has file watchers already and could send an event the indicator could pick up (hoping that systemd offers something equivalent indeed)

[Sebastien Bacher (seb128)]

@Oliver, it would be surprising if upstart wouldn't get the same issue. The problem here is that the mechanism provided by the kernel (inotify) is not handling those usecases in a nice way, what client is using the interface is not going to change that, you are just going to move the wakeups to the upstart job (upstart might be able to reduce/share the polling if several processes watch the same location though, but I'm unsure that's the case here)

[Oliver Grawert (ogra)]

what i meant is that the upstart file watcher is running anyway ... so you could as well just use it and not have the indicator duplicate the file watching functionality

[Sebastien Bacher (seb128)]

What "upstart file watcher"? Upstart doesn't magically watch all files, you have specify what you want to be notified about, and I don't think we have a job watching e.g "/usr/local/share/applications" at the moment

[Charles Kerr (charlesk)]

seb128, I think the question should be, what's a realistic timeline for a fix in GIO?

I was sympathetic to workarounds for the sake of Trusty (and, for that reason, did one in indicator-datetime). But now that the window for landing things in Trusty has passed, workarounds lose a lot of their raison d'être. I think we should look to actually fixing this in GIO.