nepomukfilewatch leaks memory

[LarsIvarIgesund]

Binary package hint: virtuoso-opensource

This is a recent regression (last week or so). After the system has been on for a while (12-24 hours), it becomes totally unresponsive.

At this point htop shows two instances of nepomukfilewatch to use 26-44% of the physical memory each. Sometimes when this happen (when the numbers above are in the lower range), there's also 3 instances of nepomukstrigiservice using up to 8% each, and a dbus-deamon instance using up 26-27%. All in all, I'm well into swap usage on a laptop with 4GB memory.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: virtuoso-nepomuk 6.1.0-0ubuntu3
ProcVersionSignature: Ubuntu 2.6.32-21.32-generic 2.6.32.11+drm33.2
Uname: Linux 2.6.32-21-generic i686
NonfreeKernelModules: fglrx
Architecture: i386
Date: Sun Apr 25 12:33:11 2010
InstallationMedia: Kubuntu 9.10 "Karmic Koala" - Release i386 (20091028.5)
ProcEnviron:
 LANGUAGE=nn_NO:nn:no_NO:no:nb_NO:nb:en
 LANG=nn_NO.UTF-8
 SHELL=/usr/bin/zsh
SourcePackage: virtuoso-opensource

[Phull (phull)]

Version: unknown (using 4.4.00 (KDE 4.4.0) "release 222", KDE:KDE4:Factory:Desktop / openSUSE_11.2)
Compiler: gcc
OS: Linux (x86_64) release 2.6.31.12-0.1-desktop

During initial indexing of my home directory, nepomukservicestub nepomukfilewatch grows to well over 2GByte of memory, rendering the system unusable due to constant swapping.

[Ub71a5bwcb7ku-martin-b69y0hv8hgzde (ub71a5bwcb7ku-martin-b69y0hv8hgzde)]

I can confirm that this service is eating a lot of memory during indexing.
At this time I can't say if this returns to normal if the indexing task is finished.

[Paul Leopardi (paul-leopardi-iinet)]

Can also confirm:

# ps -e -o args -o vsz | grep nepomuk
kdeinit4: nepomukserver [kd 315304
/usr/bin/nepomukservicestub 408108
/usr/bin/nepomukservicestub 186376
/usr/bin/nepomukservicestub 176132
/usr/bin/nepomukservicestub 690664
/usr/bin/nepomukservicestub 2763740
/usr/bin/nepomukservicestub 218376
/usr/bin/nepomukservicestub 254672
grep nepomuk 7396
/usr/bin/akonadi_nepomuk_co 222776

See also http://forum.kde.org/viewtopic.php?f=154&t=85457&start=10

[Paul Leopardi (paul-leopardi-iinet)]

# uname -a
Linux linfinit 2.6.31.12-0.1-desktop #1 SMP PREEMPT 2010-01-27 08:20:11 +0100 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/SuSE-release
openSUSE 11.2 (x86_64)
VERSION = 11.2
# rpm -q -f /usr/bin/nepomukservicestub
kdebase4-runtime-4.4.1-192.1.x86_64

[Iggy-mf (iggy-mf)]

why is this problem not confirmed? can reproduce this here. everytime strigi is indexing it eats up a lot of memory here and my system has to swap.

[Trueg (trueg)]

This bugs has been confirmed by many people and I can reproduce it, too. I suspect that you mean that you want the bug status to change. So I am doing that.

[Smithjd15 (smithjd15)]

I actually seemed to stumble across some interesting interactions and behaviour in running a 'find / sudo' after enabling compcache after every bootup.

Access times seem to be updated and the entire filesystem 'ledger' is stored in compressed ram. What I did notice is reduced ram requirements in clementine as versus Amarok and of course in the filewatch daemon.

Probably this is similar in scope to prelinking on Gentoo as far as feel goes, snake-oil-like. It may expose some workarounds for kernel behaviour and / or in Nepomuk behavior.

[Smithjd15 (smithjd15)]

(In reply to comment #6)
> I actually seemed to stumble across some interesting interactions and behaviour
> in running a 'find / sudo' after enabling compcache after every bootup.
>
> Access times seem to be updated and the entire filesystem 'ledger' is stored in
> compressed ram. What I did notice is reduced ram requirements in clementine as
> versus Amarok and of course in the filewatch daemon.
>
> Probably this is similar in scope to prelinking on Gentoo as far as feel goes,
> snake-oil-like. It may expose some workarounds for kernel behaviour and / or in
> Nepomuk behavior.

Improvements are to be noticed in shared memory mostly. Amarok isn't privy to shared memory on file handles anyway and so it doesn't show much improvement there..not being inclined to patch or dig any deeper into what calls are at issue, I used Amarok as a hardline reference and played around with Nepomuk / Clementine for a bit to assess file-handle economics. I'm still surprised how well compcache functions as a 'phreak'-like application; I haven't felt like I was hacking as such since my Atari ST days.

[Jonathan Thomas (echidnaman)]

Known problem with Nepomuk: https://bugs.kde.org/show_bug.cgi?id=226676

[Tommy_CZ (t-kijas)]

I can confirm this bug does affect me too.
Linux tom-laptop 2.6.33-020633-generic #020633 SMP Thu Feb 25 10:10:03 UTC 2010 x86_64 GNU/Linux
Kubuntu 10.04

[Jan Baron (bartonjan)]

I can confirm memory leak of nepomukservicestub too, often with virtuoso-t. It started after upgrading my system to Lucid Lynx

KDE Platform Version: 4.4.2 (KDE 4.4.2)
Qt Version: 4.6.2
Operating System: Linux 2.6.32-22-generic i686
Distribution: Ubuntu 10.04 LTS

[Vangelis Tasoulas (cyberang3l)]

Can confirm this bug too....

[Getaceres-h (getaceres-h)]

Confirmed on Kubuntu 10.04 with KDE 4.4.2

[Bugs-kde-3 (bugs-kde-3)]

Might be useful:

http://thread.gmane.org/gmane.comp.kde.devel.core/63824

A mailing list thread where Sebastian asks for help in diagnosing the memory leak.

Sebastian, did you try massif like suggested over there by Andreas Hartmetz?

[Bugs-kde-3 (bugs-kde-3)]

BTW, Sebastian, could it be related to the dangling metadata graphs that you wrote about (http://trueg.wordpress.com/2010/02/01/dangling-meta-data-graphs-caution-very-technical/)?

On my machine, the query shows that I have 11717 such graphs:

$ nepomukcmd query 'select count(?mg) where { ?mg nrl:coreGraphMetadataFor ?g . OPTIONAL { graph ?g { ?s ?p ?o . } . } . FILTER(!BOUND(?s)) . }'
callret-0 -> "11717"^^<http://www.w3.org/2001/XMLSchema#int>
Total results: 1
Execution time: 00:00:08.58

Is it a large number?

[Smithjd15 (smithjd15)]

http://trac.pcbsd.org/changeset/7147

A BSD changeset related to CPU usage. While it's all accounted for, who can say what whale jumps?

Degressing, how is this issue seen differently on BSD?

[Trueg (trueg)]

It seems the memory leak was magically fixed by some changes unrelated to the search for this bug. Too bad we never figured out the real reason. It is still good to know the leak is gone.

[Trueg (trueg)]

*** Bug 231409 has been marked as a duplicate of this bug. ***

[Trueg (trueg)]

*** Bug 235377 has been marked as a duplicate of this bug. ***

[Hrogge (hrogge)]

Unfortunately it seems to be back in KDE 4.7 (or in Kubuntu 11.10 to be more correct).

nepomukservicestub just runs my 4GB ram system at work out of memory, had do disable Nepomuk completely to get the memory back.

[Trueg (trueg)]

Git commit 4f7a5d19da26af282f640c913afccad26000a388 by Sebastian Trueg.
Committed on 24/10/2011 at 17:47.
Pushed by trueg into branch 'KDE/4.7'.

Run the MetaDataMover with an event loop.

It is using the exact same approach as the file indexer does: a new
thread is created and started and the MetaDataMover is then
QObject::moveToThread'ed to it.

This fixes mem leaks caused by DBus events that are not cleaned up.

BUG: 226676
FIXED-IN: 4.7.3

M +70 -63 nepomuk/services/filewatch/metadatamover.cpp
M +13 -10 nepomuk/services/filewatch/metadatamover.h
M +9 -10 nepomuk/services/filewatch/nepomukfilewatch.cpp
M +3 -1 nepomuk/services/filewatch/nepomukfilewatch.h

http://commits.kde.org/kde-runtime/4f7a5d19da26af282f640c913afccad26000a388

[Trueg (trueg)]

Git commit 90afde5b3a488f89be2349085971655e6497f1bc by Sebastian Trueg.
Committed on 24/10/2011 at 17:00.
Pushed by trueg into branch 'master'.

Run the MetaDataMover with an event loop.

It is using the exact same approach as the file indexer does: a new
thread is created and started and the MetaDataMover is then
QObject::moveToThread'ed to it.

This fixes mem leaks caused by DBus events that are not cleaned up.

BUG: 226676

M +70 -63 services/filewatch/metadatamover.cpp
M +13 -10 services/filewatch/metadatamover.h
M +9 -10 services/filewatch/nepomukfilewatch.cpp
M +3 -1 services/filewatch/nepomukfilewatch.h

http://commits.kde.org/nepomuk-core/90afde5b3a488f89be2349085971655e6497f1bc

[Trueg (trueg)]

Git commit 0f511184ae25364618ba244f6afda5570b02c388 by Sebastian Trueg.
Committed on 24/10/2011 at 17:47.
Pushed by trueg into branch 'master'.

Run the MetaDataMover with an event loop.

It is using the exact same approach as the file indexer does: a new
thread is created and started and the MetaDataMover is then
QObject::moveToThread'ed to it.

This fixes mem leaks caused by DBus events that are not cleaned up.

BUG: 226676

M +70 -63 nepomuk/services/filewatch/metadatamover.cpp
M +13 -10 nepomuk/services/filewatch/metadatamover.h
M +9 -10 nepomuk/services/filewatch/nepomukfilewatch.cpp
M +3 -1 nepomuk/services/filewatch/nepomukfilewatch.h

http://commits.kde.org/kde-runtime/0f511184ae25364618ba244f6afda5570b02c388

[Jonathan Thomas (echidnaman)]

Fixed with KDE 4.7.3.

[Jure Repinc (jlp)]

Looks like this is back in 4.9 (I'm currently using RC1+ from git here).

[Jure Repinc (jlp)]

Created attachment 72428
massif output

This is the massif output after running a few minutes

[Dennis Schridde (devurandom)]

(In reply to comment #22)
> Looks like this is back in 4.9 (I'm currently using RC1+ from git here).
I opened bug #304476 against KDE 4.9.0.

[Me-ngeefk4xayt3t4u9watah405ve (me-ngeefk4xayt3t4u9watah405ve)]

(In reply to comment #23)
> Created attachment 72428 [details]
> massif output
>
> This is the massif output after running a few minutes

Thanks a lot. It helped in diagnosing the problem, and I think I found another way I can reduce the memory footprint by about 10 mb.

Could you please test the patch given over here - https://bugs.kde.org/show_bug.cgi?id=304476