virtuoso consumes large amounts of memory (>1G)

[Craig Magina]

Looking at top I see a virt of ~2G and res of ~1G shortly after booting and it continues to grow slowly but can end up consuming multiple gigs of memory. I have a large IMAP e-mail account, multiple calendars and contact syncs, and RSS feeds configured in kontact.

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: virtuoso-minimal 6.1.6+repack-0ubuntu2
ProcVersionSignature: Ubuntu 3.8.0-18.28-generic 3.8.6
Uname: Linux 3.8.0-18-generic x86_64
ApportVersion: 2.9.2-0ubuntu8
Architecture: amd64
Date: Mon Apr 15 10:46:29 2013
EcryptfsInUse: Yes
InstallationDate: Installed on 2013-03-07 (38 days ago)
InstallationMedia: Kubuntu 13.04 "Raring Ringtail" - Alpha amd64 (20130306)
MarkForUpload: True
PackageArchitecture: all
SourcePackage: virtuoso-opensource
UpgradeStatus: No upgrade log present (probably fresh install)

[Piedro Kulman (piedro)]

After virtuoso-t running below my desktop without downtime for more than 24 hours it starts consuming 3 to 4 times more memroy. (from 50mB now after 32 hours now at 190mB)

Before it ran up to 517mB of memory after no downtime for a few days. Is this still not fixed?

Reproducible: Always

Steps to Reproduce:
1. start kubuntu desktop with KDE 4.8.9 from the Beta Kubuntu PPA
2. check memory usage of the process virtuoso-t and leave your system running for more than 24h
3. check memory usage again :-(

Seems like virtuoso-t is still leaking.

thx for reading,
piedro

Actual Results:
Memory usage at least triples within 24h

Expected Results:
memory usage should stay near to equal during long uptimes

[Me-ngeefk4xayt3t4u9watah405ve (me-ngeefk4xayt3t4u9watah405ve)]

Could you please make sure you're running virtuoso 6.1.4 and NOT 6.1.5.

6.1.5 has a nasty bug.

[Piedro Kulman (piedro)]

thx for the hint, but it's 6.1.4-3127 ...

[Ingo Ratsdorf (ingo-envirology)]

I have KDE 4.9 from kubuntu backports PPA and nepomukservicestub is consuming all available memory 1.5GB out of 2.0GB, makes the system totally unusable. Killing the servicestub just starts a new instance and the same procedure begins again, starting with some 50MB and then steadyly ever increasing until 1.5GB again.
Does not take 24hrs but rather 10 minutes.
Disabling Nepomuk Semantic Desktop in KDE settings "solves" the issue with the result of not being able to use CATEGORIES for contacts or calendar events any more.
I did delete all nepomuk system settings and the databases etc. Same procedure. No difference.
This is the most broken Nepomuk version ever.

4:4.9.0a-0ubuntu1~precise1~ppa1 32bit

[Kollix (kollix)]

I'm here at KDE-4.10 on openSuse 12.2 64bit and since a rather short time (probably since yesterday or so where I did a search in kmail) virtuoso uses 1.2GB !!
Even if I shut down nepomuk and akonadi completely and restart nepomuk (via systemsettings), virtuoso-t again uses 1.2GB RAM.
What information shall I provide or what can I check to find out why it uses that much RAM ?

[Me-ngeefk4xayt3t4u9watah405ve (me-ngeefk4xayt3t4u9watah405ve)]

(In reply to comment #4)
> I'm here at KDE-4.10 on openSuse 12.2 64bit and since a rather short time
> (probably since yesterday or so where I did a search in kmail) virtuoso uses
> 1.2GB !!
> Even if I shut down nepomuk and akonadi completely and restart nepomuk (via
> systemsettings), virtuoso-t again uses 1.2GB RAM.

Are you sure virtuoso still consumes 1.2 Gb ram even after restarting Nepomuk? Does it consume that much the moment it starts? How long does it take to go up to the 1.2gb?

Maybe Nepomuk didn't get started? You should try doing it manually via -

$ qdbus org.kde.NepomukServer /nepomukserver quit
-- Wait for it to end - check via `ps aux | grep nepomuk`
$ nepomukserver

[Kollix (kollix)]

It did consume 1.2GB after I activated nepomuk again in systemsettings. It did not take 1.2GB immediately after start but after I guess 2 minutes or so.

However I think it has to do with the last made search in kmail.
To verify, I did a new search inside my empty junk folder, and voila, virtuoso-t now only needs 70MB (I think I still needed to restart nepomuk to reduce the memory consumption).

If you can not verify this behaviour I'll need some more time to try to reproduce it

[Me-ngeefk4xayt3t4u9watah405ve (me-ngeefk4xayt3t4u9watah405ve)]

(In reply to comment #6)
> It did consume 1.2GB after I activated nepomuk again in systemsettings. It
> did not take 1.2GB immediately after start but after I guess 2 minutes or so.
>
> However I think it has to do with the last made search in kmail.
> To verify, I did a new search inside my empty junk folder, and voila,
> virtuoso-t now only needs 70MB (I think I still needed to restart nepomuk to
> reduce the memory consumption).
>
> If you can not verify this behaviour I'll need some more time to try to
> reproduce it

Take your time :)

We unfortunately cannot, but it would be great if you could check the queries running by doing this - http://vhanda.in/blog/2012/02/virtuoso-going-crazy-/

[Franciscoadriansanchez (franciscoadriansanchez)]

I have noticed this bug after updating to 4.10. But I have noticed something else: the extensions excluded from indexing, are indexed anyway. I saw this in Nepomuk configurations, when is shows the current indexing.

I haven't this already filled, could it be related to this bug?

[Craig Magina (craig.magina)]

I am experiencing this bug on Kubuntu 13.04 amd64 running KDE 4.10.2, virtuoso 6.1.6. Its a slow leak, but over the course of a day the virtuoso-t process grows from hundreds of megs to multiple gigs of memory (after 2 hours of running it is using 1.6G of memory). I have a large downloaded IMAP e-mail account, multiple google calendar's and a facebook sync, multiple google contacts and a ldap contact sync, plus a bunch of RSS feeds.

[Me-ngeefk4xayt3t4u9watah405ve (me-ngeefk4xayt3t4u9watah405ve)]

(In reply to comment #9)
> I am experiencing this bug on Kubuntu 13.04 amd64 running KDE 4.10.2,
> virtuoso 6.1.6. Its a slow leak, but over the course of a day the virtuoso-t
> process grows from hundreds of megs to multiple gigs of memory (after 2
> hours of running it is using 1.6G of memory). I have a large downloaded IMAP
> e-mail account, multiple google calendar's and a facebook sync, multiple
> google contacts and a ldap contact sync, plus a bunch of RSS feeds.

Urgh. Temporary solution - if virtuoso consumes too much memory just kill it. It will automatically respawn and continue.

I think I might just implement some kind of process to monitor virtuoso and kill it if it starts consuming too much memory.

[Craig Magina (craig.magina)]

I did that and it doesn't seem to start growing again. It did appear to stop growing at 1.66G. Is there any information I can gather the next time this occurs to help debug the problem? Any tools I can run on the process?

[Ernesto Manriquez (alejandronova)]

@Craig Please, watch out for potential Akonadi database corruptions. There were changes in the Akonadi DB format. I was suffering from this bug, until I ran this, in succession, in my KDE active session.

$ akonadictl fsck
$ akonadictl vacuum
$ nepomukcleaner

After that, and restarting my session, the memory leaks were gone, even after enabling the Akonadi Nepomuk Feeder per bug 315985 .

[Ernesto Manriquez (alejandronova)]

@Craig Also, please turn off the Facebook resource. It will continuously fetch updates from your timeline in realtime, and every update is a call to Akonadi Nepomuk Feeder. Also, since the Facebook resource is experimental, we can think it could be a source of the bugs.

[Franciscoadriansanchez (franciscoadriansanchez)]

(In reply to comment #12)
> @Craig Please, watch out for potential Akonadi database corruptions. There
> were changes in the Akonadi DB format. I was suffering from this bug, until
> I ran this, in succession, in my KDE active session.
>
> $ akonadictl fsck
> $ akonadictl vacuum
> $ nepomukcleaner
>
> After that, and restarting my session, the memory leaks were gone, even
> after enabling the Akonadi Nepomuk Feeder per bug 315985 .

Thanks a lot, Alejandro, for the tip,

regards.

[Craig Magina (craig.magina)]

I removed the facebook resource, ran the akonadi commands and I am still experiencing large memory growth for virtuoso-t. It happens after almost every boot. The process tends to grow to over 1.5G relatively fast and then the growth slows down, but it does continue to grow. If I kill the process, it comes back and doesn't repeat the memory growth. I have akonadi resources for imap, ldap, a couple Google accounts setup, and owncloud.

[Ernesto Manriquez (alejandronova)]

Hmmm... Please, upgrade again. Sorry for being like a broken record, but KDE 4.10.3 has a rewritten Akonadi-Nepomuk feeder, and that can make the difference here. Check you have Akonadi 1.9.2. After that, run the command set and let the Nepomuk Cleaner work until it successfully ends.

After you upgrade, and run those commands, restart your computer.

[Dan Barnett (simplemobiledan)]

Hello, Just Installed kubuntu 13.04 amd 64 with all updates I can confirm the nepomuck memory leak. I have only one google resource and nothing else yet virtuoso consumes over 2GB of memory. The only relief to the system is to completely disable the resource and disable indexing of everything. This is a big blow to kde 4.10. I really enjoy using kontact but if this memory leak isn't fixed I imagine I will have to move back to cinnamon. I really don't want to either because kde 4.10 looks fantastic.
Tried commands per alejandro's advice to no avail.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[iamsergio (iamsergio)]

I'm willing to profile it if anyone can give-me a test case, or a way to reproduce.
Here it takes about 130MB.

[Kollix (kollix)]

I've currently again the situation that virtuoso-t is using approximately 724.6 MB of memory.
Here's what I find by doing what's written at http://vhanda.in/blog/2012/02/virtuoso-going-crazy-/

There are 2 queries:
/nepomukqueryservice/query139
/nepomukqueryservice/query95

qdbus org.kde.nepomuk.services.nepomukqueryservice /nepomukqueryservice/query139 queryString
select distinct ?r ?reqProp1 where { { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/18b6518b-a964-43cd-86d0-e9056800bd51> . ?r <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#plainTextMessageContent> ?v1 . FILTER(bif:contains(?v1, "'virt'")) . OPTIONAL { ?r <http://akonadi-project.org/ontologies/aneo#akonadiItemId> ?reqProp1 . } } . }

Wow ... the second is large:

qdbus org.kde.nepomuk.services.nepomukqueryservice /nepomukqueryservice/query95 queryString
select distinct ?r ?reqProp1 where { { ?r <http://akonadi-project.org/ontologies/aneo#akonadiItemId> ?reqProp1 . { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/1f280755-fb31-42ae-93c3-0d00c73de706> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/b849ca47-f5cb-4a1b-8e66-f492e0c74ea1> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/e0007b88-19a8-4a18-b8fe-9f52db773429> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/b2cc492d-a6c5-4bd8-b8b5-9a1220518f3c> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/796549f2-ba2e-4270-a8f6-018dbafb42dc> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/b4665ccb-1d70-4e67-b968-3b6ca0a5ee40> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/2414de6b-3f9b-4636-ba7b-2de8e2ef0e89> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/abbb5463-e29e-4150-8b98-ecc80f9f13b2> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/b7c9e1e8-0777-43af-9c8b-dc2653413a13> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/adebc26c-c255-4a8e-b62d-ff4afd6800c5> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/9449d58f-a9d4-42e7-b54a-7c41a299cee9> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/62830ceb-a9cc-41d5-ac9d-fc9b486a602d> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/f894f9e2-c6a0-4d33-8978-222b1e0d04b5> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/0ef02f61-0e6f-4e67-aba8-393af9d38676> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/cca75cb2-0051-45e8-80eb-ce9fadec438f> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/4236263b-f2f4-400b-9d63-5ce241958297> . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2...

[Kollix (kollix)]

And BTW: I'm using KDE 4.10.5 on openSuse 12.3, 64bit

[Kollix (kollix)]

So now I've already upgraded to KDE 4.11 and currently virtuoso-t is again using approximately 938.5 MB of memory.

Interestingly qdbus org.kde.nepomuk.services.nepomukqueryservice shows now not a single query:

/
/KDebug
/backupmanager
/datamanagement
/nepomukontologyloader
/nepomukqueryservice
/nepomukstorage
/resourcewatcher
/resourcewatcher/watch10
/resourcewatcher/watch14
/resourcewatcher/watch22
/resourcewatcher/watch6
/resourcewatcher/watch8
/servicecontrol

I killed it and it restarted at 12:50 and it used about 50MB RAM.
It's now 15:40 and the memory is already up to approximately 139.3 MB

[Donolwenn (donolwenn)]

Since 4.11 on Archlinux, virtuoso-t eats more and more memory and KMail is slow.
dbus-qt4 org.kde.nepomuk.services.nepomukqueryservice
/
/backupmanager
/datamanagement
/nepomukontologyloader
/nepomukqueryservice
/nepomukstorage
/resourcewatcher
/resourcewatcher/watch1
/resourcewatcher/watch2
/resourcewatcher/watch3
/resourcewatcher/watch37
/resourcewatcher/watch5
/resourcewatcher/watch6
/resourcewatcher/watch7
/resourcewatcher/watch8
/resourcewatcher/watch9
/servicecontrol

[Stecue (stecue)]

I updated from openSUSE 12.3 (KDE 4.10.5) to openSUSE 13.1 (KDE 4.11) and now virtuoso-t keeps eating memory (to more than 6 GB and eventually hangs everything). I only use nepomuk for email indexing (because Kmail requires). The size of all mail boxes are about 7GB. I hope it dose not need 7GB memory to index 7GB emails? Anyway, I have to use a script to monitor the memory use and kill virtuoso-t as suggested.

$ qdbus org.kde.nepomuk.services.nepomukqueryservice
/
/KDebug
/backupmanager
/datamanagement
/nepomukontologyloader
/nepomukqueryservice
/nepomukstorage
/resourcewatcher
/resourcewatcher/watch1
/resourcewatcher/watch2
/resourcewatcher/watch3
/resourcewatcher/watch5
/resourcewatcher/watch6
/servicecontrol

[alexqwesa (alexo-veto)]

kde 4.11.4 x64
memory still leaking(and it's leak fast! ~30Mb per minute), i tried:
$ akonadictl fsck
$ akonadictl vacuum
$ nepomukcleaner
it didn't help

qdbus org.kde.nepomuk.services.nepomukqueryservice
/
/backupmanager
/datamanagement
/nepomukontologyloader
/nepomukqueryservice
/nepomukstorage
/resourcewatcher
/resourcewatcher/watch1
/resourcewatcher/watch14
/resourcewatcher/watch15
/resourcewatcher/watch2
/resourcewatcher/watch3
/servicecontrol

[alexqwesa (alexo-veto)]

kde 4.11.5 x64 still leaking...

[alexqwesa (alexo-veto)]

in addition to 16GB RAM I create 28GB swap and allowed nepomuk to work few weeks( also i set ulimit to 12Gb in order to don't freeze my system) - look like it fix itself...

[Ingo Ratsdorf (ingo-envirology)]

Don't get me wrong, I don't want to sound cynical but I have no 16GB RAM, I have a laptop with 2GB and the reason I use Akonadi and Nepomuk is TO FIND STUFF FAST, not wait some weeks.
Is that a wrong expectation?

On my system, I now have nepomuk enabled and it seem to work, however the contact auto completion in KMail is now not working. If I switch nepomuk off, the autocompletion is working again, but I have no categories in KAddressbook... Aaaaaaahhhhhh.....
So either Address categories OR address autocompletion.

[Me-ngeefk4xayt3t4u9watah405ve (me-ngeefk4xayt3t4u9watah405ve)]

Thank you for taking the time to file a bug report.

The Nepomuk project is no longer included in the KDE Software Compilation. With Plasma 5, we have replaced most of the underlying technology with Baloo and other components. Hopefully this will have addressed your concern.

We encourage you to try out Plasma 5 (+Baloo) and let us know if your problem persists.