virtuoso-t eats my cpu, should be nice

[aexl]

on kubuntu lucid there is a process "virtuoso-t" that eats my cpu (it is said to help nepomuk with indexing).
this background process has niceness 0 according to ps.
i think as a background indexer it should have maximum niceness of 19.

[Philip Muškovac (yofel)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. This bug did not have a package associated with it, which is important for ensuring that it gets looked at by the proper developers. You can learn more about finding the right package at https://wiki.ubuntu.com/Bugs/FindRightPackage. I have classified this bug as a bug in virtuoso-opensource.

When reporting bugs in the future please use apport, either via the appropriate application's "Help -> Report a Problem" menu or using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

[tdn (spam-thomasdamgaard)]

I can confirm this bug on fully updated kubuntu 10.04.

[aexl (aexl)]

added kdebase-runtime to affected projects, because the calling process of virtuoso-t ist nepomukservicestub, which belongs to this project.

[aexl (aexl)]

did not get lp to add this possible upstream link: https://bugs.kde.org/show_bug.cgi?id=228081

[Philip Muškovac (yofel)]

Upstream watch set and reassigned.

[Scott Kitterman (kitterman)]

We enabled indexing by default this cycle and I can confirm my CPU is periodically eaten as well.

[Jonathan Thomas (echidnaman)]

Technically indexing was broken until this cycle. ;)

[Scott Kitterman (kitterman)]

Yes, but we tried this with strigi before so the idea isn't new.

[Jeff Trull (jetrull)]

This is fairly frustrating. Hope something can be done about it. Suddenly my fan turns on... and it's "virtuoso-t" with 98%.

[Jonathan Thomas (echidnaman)]

The solution for now is to disable indexing by default. It has not been enabled in previous releases, and we can address it in kubuntu-default-settings in a way that users who currently have it enabled will not lose their settings.

The underlying issue of virtuoso not having a low enough priority still needs to be addressed, but I'm not as hopeful for an SRU for that bit.

[Jonathan Thomas (echidnaman)]

For default settings in maverick, we should try a "wait and see" approach and see if things improve.
I'll attach a patch for kubuntu-default-settings for lucid in a bit here.

[Martin Pitt (pitti)]

Accepted kubuntu-default-settings into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Rohan Garg (rohangarg)]

Hi
With the latest kubuntu-default-settings i can confirm this has been resolved.Strigi is disabled by default and does not cause high CPU loads

[Scott Kitterman (kitterman)]

The Lucid fix for this issue is verification-done (the wrong task was milestoned). It should be copied to -updates and included in 10.04.1.

[Jonathan Riddell (jr)]

Copied to lucid-updates

[Martin L. (heinrich20)]

Version: 4.1 (using KDE 4.4.5)
OS: Linux

Normal working with my notebook. Writing mails with KMail. Then suddenly the application is blocked totally. Nothing happenes. I can see that virtuoso uses more than 50% CPU. Together with KMail using more than 20% and I do not knowhow much memory Kmail is blocked and my system is blocked - annoying!

I did not switch of strigi but switched of indexing of multimedia files. No idea if this will help. Anyway, to work productively with my notebook shall I switch of this whole strigi/nepomuk stuff?

Reproducible: Always

Steps to Reproduce:
no way to reproduce it.

Actual Results:
The problem occurs during normal work.

OS: Linux (x86_64) release 2.6.32-5-amd64
Compiler: cc

[Loacoon (loacoon)]

Same here, virtuoso-t often uses 50 to 70% of each cores of my CPU for around 2 or 3 minutes.

[Trueg (trueg)]

It would be really interesting to track down in which situations Virtuoso takes this much CPU.
Does this only happen when using KMail or also if KMail is not running?

[Loacoon (loacoon)]

I don't use Kmail at all so this can't be it. It seems to happen when "playing" with files. Moving, erasing or copying a certain amount of files seems to trigger this.

[Trueg (trueg)]

Is this still on 4.4.x or 4.5.0?

[Loacoon (loacoon)]

4.5.0

[Trueg (trueg)]

can you try a patch for kdebase?

[Loacoon (loacoon)]

I'm using a Gentoo overlay ebuild, so I can try to patch it, but I'm not sure I'll manage to do it (it doesn't look complicated though).
By the way, I just switched to 4.5.1, in case it changes something to your patch.

[Nicolas Réau (kolia)]

KDE 4.5.1 here and noticing the same behavior. It was not indexing at the moment. The only particular thing I can think of when it happened is that it started just after I made a svn checkout of kdelibs (thus lots of files written suddenly).

Regards,

[HinzundKunz (martin-tlustos)]

I'm running Kubuntu 10.04 with the kde4.5 testing packages and virtuoso still peaks high, even when indexing is disabled. It probably has to do with akonadi being used (I'm trying out the akonadi resources like imap and stuff).
Anyway, I use cpulimit to prevent my cpu from overheating, so it doesn't threaten my computer.

[Christopher-tanner (christopher-tanner)]

Running a version called 4.5.68 on Mandriva Cooker, and am noticing the same behaviour. CPU usage is not dependent on whether or not kmail is running. Anything that I can look for to help debug this?

[Rudd-O (rudd-o)]

The absolute exact same thing happens to me. It makes KDE completely unusable. When you strace it, all you can see is a metric ton of futex calls and very little work actually being done. I really wish I could use KDE but the fact of the matter is, I can't let this destroy my laptop battery life.

Please already confirm this and put some real effort in fixing this problem, OK?

[Loacoon (loacoon)]

You still can use KDE, you just have to disable Strigi and Nepomuk. No more CPU problems, and ~150Mo or RAM saved (which is also way too much).

[Rudd-O (rudd-o)]

I fail to see the point of using KDE without these KDE technologies. I admit, there is not that much those technologies are useful for, right now, but saying "Turn off nepomuk and strigi" is sort of like saying "Well, you do without desktop search and semantic information, which is an advertised feature of KDE". You can drive your car... but forget about turning the radio and A/C on.

[Loacoon (loacoon)]

KDE is not only this, and as you said, it's still not quite useful. But you're right, Linux gives this choice, and there are many desktop environments much more stable than KDE.

[AlexHarrowell (a-harrowell)]

I've got this problem as well, w/KDE4.4.4 and OpenSUSE11.3/2GB RAM/Intel T1700.

Typically, it's associated with KMail, which will happily spend 15 minutes hammering the disk I/O and basically hanging every time it starts up. During that period you either have KMail pulling 25-30% CPU and a *lot* of disk activity, or else Virtuoso running up to 90~% CPU. Interestingly, it is NOT REPEAT NOT hitting the swap - during these events there is typically well over 1GB of RAM available.

This has recently started to be a problem - I don't know whether this is as a result of an update or whether there's some critical folder or mailbox size.

I take the point that you could turn off desktop search, but I don't *have* desktop search and haven't since KDE3.5.7 and good old kerry beagle. I'd like to have a desktop search application rather than KFind. Strigi is disabled through System Settings.

I am at a loss to see the purpose of the whole complex of Akonadi/Strigi/Nepomuk/Virtuoso. It just appears to be a selection of processes that burn enormous quantities of system resources and do nothing whatsoever of any use, except mangling my data now and again.

[Mark Greenwood (fatgerman)]

I've got this problem on Maverick, with KDE 4.5.3. With file indexing enabled virtuoso-t keeps the CPU at 99% indefinitely *even* if I then disable file indexing again. I have to reboot after disabling indexing.
Also, if I enable the option in digikam to store metadata in nepomuk, the same thing happens.
However I've found that if increase the memory available to nepomuk (from System Settings) this problem goes away. I have, however, had to allow it to use 400MB of RAM, which is way unacceptable.
I think, if you only index a very small number of files the default amount of memory is probably fine, but I now have 18,000 files in my database (it has indexed all my MP3s and Photos, which is what I wanted it to do).

[Bernhard Rosenkraenzer (berolinux)]

This just happened for me with 4.6.0-beta2:

- Installed 4.6.0-beta2 with a clean ~/.kde
- Opened kmail and imported mails from our IMAP server and contacts and calendars from SOGo
- Quit kmail and did completely unrelated things

Suddenly the system turned sluggish and virtuoso-t consumed 450% CPU (this is a 4-core system with hyperthreading, so the 450% number is ok).
After waiting for about half an hour (assuming it might be related to having imported thousands of messages from an imap server - akonadi is still running even after exiting kmail), virtuoso-t went down to 247%, where it has remained stable for the last couple of hours.

The systemsettings Nepomuk configuration tool says "File indexer is idle", so it's not indexing new files.

As with the other people reporting this, strace shows virtuoso-t doing virtually nothing but futex calls.

There are 164GB spread across 10380 files in my home directory (but again the indexer claims to be idle).

Running processes that may be in some way related:

akonadi_control
akonadiserver
akonadi_agent_server
akonadi_nepomuk_calendar_feeder
akonadi_nepomuk_email_feeder
akonadi_nepomuk_contact_feeder
akonadi_maildispatcher_agent
akonadi_imap_resource
akonadi_davgroupware_resource
akonadi_davgroupware_resource (running twice with different identifiers)
nepomukserver
nepomukservicestub nepomukremovablestorageservice
nepomukservicestub nepomukstorage
nepomukservicestub nepomukstrigiservice
nepomukservicestub nepomukbackupsync
nepomukservicestub nepomukqueryservice
nepomukservicestub nepomukfilewatch
nepomukservicestub nepomukremovablestorageservice

There's no extraordinary spewage in .xsession-errors or the likes.

[Bernhard Rosenkraenzer (berolinux)]

I killed the related processes manually one by one.
virtuoso-t insanity stopped when I killed

nepomukservicestub nepomukstorage

Incidentally, this is the last process I killed, so the others are quite probably not to blame.

[AlexHarrowell (a-harrowell)]

On Tuesday 14 December 2010 19:06:59 Bernhard Rosenkraenzer wrote:
> https://bugs.kde.org/show_bug.cgi?id=246678
>

The problem has been resolved for me; I wonder if a recent update changed
anything?

>
> Bernhard Rosenkraenzer <email address hidden> changed:
>
> What |Removed |Added
> ---------------------------------------------------------------------------
> - CC| |<email address hidden>
>
>
>
>
> --- Comment #15 from Bernhard Rosenkraenzer <bero arklinux org> 2010-12-14
> 20:06:55 --- This just happened for me with 4.6.0-beta2:
>
> - Installed 4.6.0-beta2 with a clean ~/.kde
> - Opened kmail and imported mails from our IMAP server and contacts and
> calendars from SOGo
> - Quit kmail and did completely unrelated things
>
> Suddenly the system turned sluggish and virtuoso-t consumed 450% CPU (this
> is a 4-core system with hyperthreading, so the 450% number is ok).
> After waiting for about half an hour (assuming it might be related to
> having imported thousands of messages from an imap server - akonadi is
> still running even after exiting kmail), virtuoso-t went down to 247%,
> where it has remained stable for the last couple of hours.
>
> The systemsettings Nepomuk configuration tool says "File indexer is idle",
> so it's not indexing new files.
>
> As with the other people reporting this, strace shows virtuoso-t doing
> virtually nothing but futex calls.
>
> There are 164GB spread across 10380 files in my home directory (but again
> the indexer claims to be idle).
>
> Running processes that may be in some way related:
>
> akonadi_control
> akonadiserver
> akonadi_agent_server
> akonadi_nepomuk_calendar_feeder
> akonadi_nepomuk_email_feeder
> akonadi_nepomuk_contact_feeder
> akonadi_maildispatcher_agent
> akonadi_imap_resource
> akonadi_davgroupware_resource
> akonadi_davgroupware_resource (running twice with different identifiers)
> nepomukserver
> nepomukservicestub nepomukremovablestorageservice
> nepomukservicestub nepomukstorage
> nepomukservicestub nepomukstrigiservice
> nepomukservicestub nepomukbackupsync
> nepomukservicestub nepomukqueryservice
> nepomukservicestub nepomukfilewatch
> nepomukservicestub nepomukremovablestorageservice
>
>
> There's no extraordinary spewage in .xsession-errors or the likes.

[Aitor Pazos (aitorpazos)]

The same to me here.
On system preferences it says Strigi is waiting, but virtuoso-t process is taking 159% CPU (2 Cores)

Running KDE 4.6 RC1 on x86_64
Nopomuk Details:
  34369 files in index
  5.3 GB store size

Akonadi precesses:
 3370 ? 00:00:01 akonadi_control
 3372 ? 00:00:53 akonadiserver
 3409 ? 00:00:00 akonadi_agent_l
 3410 ? 00:00:00 akonadi_agent_l
 3411 ? 00:00:00 akonadi_birthda
 3412 ? 00:00:04 akonadi_davgrou
 3413 ? 00:00:00 akonadi_agent_l
 3414 ? 00:00:01 akonadi_imap_re
 3415 ? 00:00:24 akonadi_imap_re
 3416 ? 00:01:27 akonadi_imap_re
 3417 ? 00:00:05 akonadi_imap_re
 3418 ? 00:00:04 akonadi_maildis
 3419 ? 00:00:00 akonadi_microbl
 3420 ? 00:00:00 akonadi_microbl
 3421 ? 00:00:00 akonadi_mixedma
 3422 ? 00:00:00 akonadi_nepomuk
 3423 ? 00:00:00 akonadi_nepomuk
 3424 ? 00:00:06 akonadi_nepomuk
 3425 ? 00:00:00 akonadi_nepomuk
 3426 ? 00:00:03 akonadi_agent_l
 3544 ? 00:00:00 akonadi_agent_s

Nepomuk services:
3422 ? 00:00:00 akonadi_nepomuk
 3423 ? 00:00:00 akonadi_nepomuk
 3424 ? 00:00:06 akonadi_nepomuk
 3425 ? 00:00:00 akonadi_nepomuk
 3466 ? 00:00:00 nepomukserver
 3496 ? 00:00:44 nepomukservices
 3671 ? 00:00:00 nepomukservices
 3672 ? 00:00:08 nepomukservices
 3673 ? 00:00:00 nepomukservices
 3675 ? 00:00:00 nepomukservices
 3676 ? 00:00:33 nepomukservices

[Bernhard Rosenkraenzer (berolinux)]

Happened again here today.
When I got into the office, my work box (which I already updated to 4.6.0-rc2 yesterday) was using up almost all available CPU power in virtuoso-t, and once again, killing the nepomukstorage process fixed it.

So the problem is not fixed, though maybe it's less common than before.

[Lbeltrame (lbeltrame)]

If you're using PIM 4.6 (the betas), the problem can be workarounded by disabling the Nepomuk Search runner in krunner.

[Martin L. (heinrich20)]

Thank you a lot! Can you explain this briefly?

Am Mittwoch, 5. Januar 2011, um 12:35:33 schrieb Luca Beltrame:
> If you're using PIM 4.6 (the betas), the problem can be workarounded by
> disabling the Nepomuk Search runner in krunner.

[Lbeltrame (lbeltrame)]

This is not the only cause of the problem, but according to http://lists.kde.org/?t=129170918700001&r=1&w=2 some KRunner queries hang virtuoso-t. Hence, opening up KRunner and disabling the search plugin (wrench -> uncheck "Nepomuk Search") will work around the problem.

It is *not* a fix, it merely prevents the issue from appearing.

[Martin L. (heinrich20)]

Thank you, Luca but I do not understand a word! krunner is a kde script to
start applications, correct? How do I inactivate processes, etc. in this way?

Martin

Am Mittwoch, 5. Januar 2011, um 14:58:05 schrieb Luca Beltrame:
> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #22 from Luca Beltrame <einar heavensinferno net> 2011-01-05
> 14:58:02 --- This is not the only cause of the problem, but according to
> http://lists.kde.org/?t=129170918700001&r=1&w=2 some KRunner queries hang
> virtuoso-t. Hence, opening up KRunner and disabling the search plugin
> (wrench -> uncheck "Nepomuk Search") will work around the problem.
>
> It is *not* a fix, it merely prevents the issue from appearing.

[mutlu_inek (mutlu-inek)]

@ Martin: Open Krunner (Alt+F2), then click on the wrench symbol to the very left of Krunner. Then disable the Nepomuk Desktop Search runner plugin and click "apply".

[Trueg (trueg)]

*** Bug 258039 has been marked as a duplicate of this bug. ***

[Martin L. (heinrich20)]

Am Mittwoch, 5. Januar 2011, um 15:50:06 schrieb mutlu inek:
> @ Martin: Open Krunner (Alt+F2), then click on the wrench symbol to the
> very left of Krunner. Then disable the Nepomuk Desktop Search runner
> plugin and click "apply".

Hey Mutlu, thankx a lot! ;-)

[Martin L. (heinrich20)]

Starting up my system including KMail without Desktop Search this morning was
even slower than yesterday! :-(

Result: There are really serious problems with KMail since KDE 4.x and noone
really has an idea what the reason might be and even more: noone cares! Sorry
for that!

/"Off-Topic"
To no longer be blamed that I am constantly running a Debian test system I
changed to kubuntu 10.10. No big difference! Yes Kubuntu normally is faster
when starting up but the problems within KDE persist!
/End

Am Mittwoch, 5. Januar 2011, um 21:24:49 schrieb Martin L ü c h e m:
> [Bug 246678] virtuoso: Usage of CPU is much too high

[Trueg (trueg)]

*** Bug 228081 has been marked as a duplicate of this bug. ***

[Trueg (trueg)]

*** Bug 228884 has been marked as a duplicate of this bug. ***

[Trueg (trueg)]

*** Bug 230400 has been marked as a duplicate of this bug. ***

[Kdebugs-99-urgwurz (kdebugs-99-urgwurz)]

*** This bug has been confirmed by popular vote. ***

[Mueller-kde (mueller-kde)]

Sebastian, whats the status here? this is the only remaining 4.6.0 blocker.

[Edney Matias da Silva (edney)]

Running KDE 4.6.0 RC 1 with Nepomuk krunner disabled and performance it's ok. Didn't give a try enabling it.

[X-wstephenson (x-wstephenson)]

Created attachment 56212
Patch 1/2 kdelibs patch by trueg

This is the patch that Sebastian proposed, which has the drawback that it creates a lot of CPU usage on first run.

[X-wstephenson (x-wstephenson)]

Created attachment 56213
Patch 2/2 kdebase patch by trueg

[GDR! (gdr.name)]

virtuoso_t uses all CPU cores and all I/O when looking at iotop, making the system unresponsive and unusable. Can't it just run nice and ionice? I'm using 10.10.

[Trueg (trueg)]

The patch Will proposes introduces two optimizations. I could probably reduce the first-run issue if I removed one of the optimizations. I will have to test that though.

[Mueller-kde (mueller-kde)]

I would be fine with a mitigation.

it seems I can change the default of this to false:

[Service-nepomukstrigiservice]
autostart=false

or to disable the krunner plugin. which one actually helps?

[Mueller-kde (mueller-kde)]

how do I disable the krunner plugin btw (I mean the default of it being disabled)?

[Rdieter-math (rdieter-math)]

Will,Sebastian, patch in comment #35 (Patch 2/2 kdebase) seems to be a dup of that from comment #34 (Patch 1/2 kdelibs).

Do either of you want this tested more? If so, could you repost the kdebase patch?

[HinzundKunz (martin-tlustos)]

@ Dirk (Comment #38): ALT-F2 and then disable the plugin in the options dialogue

[X-wstephenson (x-wstephenson)]

Created attachment 56242
Patch 2/2 kdebase patch by trueg

[8ebastkzk (8ebastkzk)]

SVN commit 1215973 by trueg:

Backport:
* Fixed the handling of quotes and keywords such as "AND", "OR", and "NOT" in LiteralTerm.
  Now correct bif:contains or regex filters are created for values.
* Made the query parser merge LiteralTerms into a single one to improve query performance.
  While merging two LiteralTerms into one does not yield the exact same query (when merged
  both literal tokens need to appear in the same property value while with separate LiteralTerms
  the tokens can appear in different properties) it should cover close to all typical use cases
  while increasing the performance significantly and getting rid of the nasty "Virtuoso goes
  crazy when I use KRunner" bug.

CCBUG: 246678

 M +91 -48 literalterm.cpp
 M +40 -5 queryparser.cpp

WebSVN link: http://websvn.kde.org/?view=rev&revision=1215973

[Trueg (trueg)]

Created attachment 56252
Patch which introduces the new query terms (kdelibs/nepomuk)

Could someone please test this and the next patch on kde 4.6 and confirm that it fixes the virtuoso going crazy bug.

[Trueg (trueg)]

Created attachment 56253
Patch that creates the data necessary for the new query terms in nepomukquery

And this is the second patch to test in combination with the former one on kde 4.6.

[Trueg (trueg)]

Created attachment 56254
Updated patch which contains missing files (kdebase/runtime/nepomuk)

[Trueg (trueg)]

Created attachment 56258
Updated patch which adds faster pim data conversion (kdebase/runtime/nepomuk)

[Trueg (trueg)]

Created attachment 56259
yet another update for kdebase/runtime/nepomuk

[T-bracken (t-bracken)]

Hi,

This message is for all who want to get rid
of the nepomuk process which is using up all the
CPU cycles on the Linux KDE desktop.

I found out where the nepomuk is started up from.

Its in the /usr/share/autostart directory ,
under the file nepomukserver.desktop .

As someone wrote below,

http://forums.opensuse.org/applications/418846-how-do-i-get-rid-nepomuk-virus.html

renaming this file (/usr/share/autostart/nepomukserver.desktop)
with a non .desktop extension and
clearing out the $HOME/.kde4/share/apps/nepomuk directory
seems to stop this service from running on startup.

Cheers!

Tom Bracken

[Martin L. (heinrich20)]

Thanks Thomas but shouldn't there be an easier way? I mean KDE 4 was developed
comletely new and there is no way to prevent this system from running a
process easier? I would assume a simple switch in kcontrol!!!

Am Donnerstag, 20. Januar 2011, um 22:55:06 schrieb Thomas Bracken:
> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #48 from Thomas Bracken <t bracken shaw ca> 2011-01-20
> 22:54:57 --- Hi,
>
> This message is for all who want to get rid
> of the nepomuk process which is using up all the
> CPU cycles on the Linux KDE desktop.
>
> I found out where the nepomuk is started up from.
>
> Its in the /usr/share/autostart directory ,
> under the file nepomukserver.desktop .
>
> As someone wrote below,
>
> http://forums.opensuse.org/applications/418846-how-do-i-get-rid-nepomuk-vir
> us.html
>
> renaming this file (/usr/share/autostart/nepomukserver.desktop)
> with a non .desktop extension and
> clearing out the $HOME/.kde4/share/apps/nepomuk directory
> seems to stop this service from running on startup.
>
> Cheers!
>
> Tom Bracken

[X-wstephenson (x-wstephenson)]

@Martin: Yes, there is. Look in 'Desktop Search' in systemsettings. This setting is per-user though.

@Thomas: There is no need to take a crowbar to the autostart .desktop file to prevent Nepomuk starting. This will prevent any user from locally configuring it to start in the future, and breaks the integrity of the package (rpm -V kdebase4-workspace to see what I mean). There is an elegant way (which I've just been using to configure Nepomuk and Strigi indexing default settings for openSUSE 11.4).

If you look inside the nepomukserver.desktop autostart there is

X-KDE-autostart-condition=nepomukserverrc:Basic Settings:Start Nepomuk:true

This means if, per user
$HOME/.kde4/share/config/nepomukserverrc
or system wide
/etc/kde4/share/config/nepomukserverrc

contain

[Basic Settings]
Start Nepomuk=false

the autostart desktop file evaluates this file and does not start nepomukserver.

[Trueg (trueg)]

Created attachment 56273
Another update for Will S. - cutting down initial update time again (kdebase/runtime/nepomuk)

[Trueg (trueg)]

Created attachment 56274
Updating patches like crazy: this is probably the best I can do for the initial update. (kdebase/runtime/nepomuk)

[X-wstephenson (x-wstephenson)]

Re patch in c52: It's still not terminating. Something else is going on here. I can make the database available to you privately if you want. For now I will try with a newly-created non-Akonadi-data database.

[Ernesto Manriquez (alejandronova)]

With pre-release KDE 4.6.0 packages (Rex Dieter repo):

- The first search in KRunner works perfectly.
- The second search in KRunner eats one core.
- The third search in KRunner eats the other core.

[Ernesto Manriquez (alejandronova)]

That was with the Nepomuk file search KRunner Plugin. Now, disabling that, restarting Nepomuk, and enabling Contacts KRunner Plugin...

Works perfectly! I can send mail to contacts instantly, without a wait, without thrashing a core. Kudos!

[Trueg (trueg)]

Created attachment 56434
Updated patch: yet another improvement.

[X-wstephenson (x-wstephenson)]

@Sebastian [/space/kde/installs/4_6/bin/nepomukservicestub] nepomukstorage(10378)/nepomuk (storage service) CrappyInferencer2::UpdateAllResourcesThread::run: 4 resources updated. Elapsed: 31.703 sec

using my pathological 90000 PIM items database on the core2 1.8GHz machine.

And I can't trigger the busy cpu now with the nepomuk runner, so I'm happy.

[Trueg (trueg)]

Created attachment 56522
Updated patch which adds subtype inference (kdebase/runtime/nepomuk)

Hey Will, please test this patch. I am not at home and cannot test it with real data but am very curious.

[X-wstephenson (x-wstephenson)]

I don't think the new code in CrappyInferencer2::UpdateAllResourcesThread even runs. I don't see the Elapsed debug output, neither is there a noticeable CPU spike.

[Trueg (trueg)]

@Will: ok, could you please do me a last favor and provide the output of

nepoukcmd list "" "" "" "<urn:crappyinference2:inferredtriples>"

(This is after the run with the latest patch)

[X-wstephenson (x-wstephenson)]

Total Results: 0

[X-wstephenson (x-wstephenson)]

I just logged in and the conversion took place on login. Elapsed time: 41s. I couldn't make it happen by starting nepomukserver in a running session.

[0cs935kb51-marc-wz6bkyhu4u (0cs935kb51-marc-wz6bkyhu4u)]

*** Bug 259517 has been marked as a duplicate of this bug. ***

[Charlie Figura (cfigura)]

I'm regularly seeing nepomuk-t consume at least 50% CPU. I saw the thread above (e.g. comment #24) and checked and found that the nepomuk plugin in krunner was *already* disabled. So virtuoso-t is *still* burning up my CPU.

[Charlie Figura (cfigura)]

Sorry, I left out that I'm running the released 4.6 version.

[Sebastian Kügler (sebasje)]

@#64: Did you also disable the Contacts runner?

[Álvaro M. Recio (naproxeno)]

I can confirm the same problem as Charles Figura. I'm using Kubuntu and I don't have a Contacts runner, only Nepomuk.

However, the problem only occurs when using the new Akonadi-enabled PIM suite with Akonadi 1.4.95. Disabling Akonadi and restarting Nepomuk makes virtuoso-t behave as a normal process.

[0cs935kb51-marc-wz6bkyhu4u (0cs935kb51-marc-wz6bkyhu4u)]

I agree with Álvaro. I have gone back to KDEPIM 4.4.10 and virtuoso is no longer getting crazy.

Maybe the new version triggers something wrong with nepomuk?

Regards

[moenchmeyer (rm-anracon)]

(In reply to comment #68)
> I agree with Álvaro. I have gone back to KDEPIM 4.4.10 and virtuoso is no
> longer getting crazy.
>

I can confirm that the problem started with the newer Akonadi/Nepomuk versions.

I, too, have meanwhile given up KDE 4.6 Beta 1/Beta2 and have gone back to KDEPIM 4.4.10. Everything is normal there.

However, Nepomuk/Akonadi consume permanently 100% of one CPU core in KDE 4.6 Beta 1 and Beta 2.

[Ernesto Manriquez (alejandronova)]

I can confirm the problems with KDE PIM/Akonadi. But my own experience is a little different. 2 different machines, one running Ubuntu, the other one Fedora.

1. I began with almost empty databases, because of the known problems with migration between KDE PIM 4.4.x and KDE PIM/Akonadi. With Fedora, I couldn't trigger the bug, but with Ubuntu I triggered the Virtuoso bug more than once. I narrowed it to the Akonadi-Nepomuk Email Feeder.

2. To workaround the issues I was experiencing with Kubuntu, I tried to remove and recreate the resources with Akonadi Console, and after a try, mail began to flow and Virtuoso core-wasting somewhat stopped. The behavior of Virtuoso this time was different: it was using ~70% of one core, while akonadi_nepomuk_email_feeder was shown as running. Certainly, akonadi_nepomuk_email_feeder can hog a system just like (or more than) Strigi, and this flew under everyone's radar.

3. I simply left akonadi_nepomuk_email_feeder index all the mail. After half an hour, akonadi_nepomuk_email_feeder seemed to finish, because there wasn't more mail to index, and my Virtuoso (hence, NEPOMUK) CPU usage went to the single digits and stayed there.

With Fedora the behavior was more straightforward, but similar. After no recreating resources, but simply using what I had, akonadi_nepomuk_email_feeder pushed my CPU to 70% out of 200%. After a while, all mail was indexed, and NEPOMUK CPU usage went to the single digits.

I filed a separate bug against akonadi_nepomuk_email_feeder (bug 249828) because that process doesn't suspend on batteries like Strigi does. Also, its interaction with Virtuoso and NEPOMUK should be monitored.

This also means that, for everyone affected, a spot-on workaround should be to remove the Nepomuk E-Mail Feeder resource from your Akonadi pool.

[Ernesto Manriquez (alejandronova)]

I forgot one important thing: the size of the mail-folders:

- Folder A: ~1,000 POP3 resource, connected.
- Folder B: ~20,000 IMAP resource, GMail, connected.

[Álvaro M. Recio (naproxeno)]

After reading about akonadi_email_feeder I was going to write that in my case that wouldn't be the cause of the problem because I have a very small email folder. To prove it, I even removed it from Akonadi and restarted KDE.

Of course, virtuoso-t was again using 100% of a core but to double check I opened Akonadi Console and removed "Nepomuk EMail Feeder". Nothing changed.

But then I noticed "Nepomuk Contact Feeder" and "Nepomuk Calendar Feeder". Indeed, I have some calendars in Akonadi so I tried removing those feeders.

Then, virtuoso-t started behaving. Barely 1% of CPU usage and Nepomuk was indeed working, as I could use Dolphin search and Bangarang.

So, to summarize, there seems to be something in those Nepomuk feeders that is triggering the 100% CPU usage bug. At least, that's what it was causing the problem for me.

[Tamás Németh (nice)]

http://nice.nyme.hu/c60cg.pdf

This PDF files makes a nepomukservices process consuming 100% CPU virtually forever if strigi is enabled.

[Álvaro M. Recio (naproxeno)]

In my machine it seems Strigi/Nepomuk managed to index the PDF without problems.

I downloaded it, searched for words contained in the PDF using Dolphin and it found it without hanging.

I'm using the packages for KDE SC 4.6 from the Kubuntu Backports PPA (I don't know how to get the version of Strigi I'm running).

[Max (mlinchits)]

Virtuoso no longer displays this cpu-sucking behavior. It'd behaved
very well since kde 4.5

On Tue, Feb 1, 2011 at 1:37 PM, Alvaro Manuel Recio Perez
<email address hidden> wrote:
> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #74 from Alvaro Manuel Recio Perez <amrecio gmail com>  2011-02-01 19:37:08 ---
> In my machine it seems Strigi/Nepomuk managed to index the PDF without
> problems.
>
> I downloaded it, searched for words contained in the PDF using Dolphin and it
> found it without hanging.
>
> I'm using the packages for KDE SC 4.6 from the Kubuntu Backports PPA (I don't
> know how to get the version of Strigi I'm running).
>
> --
> Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.
>

[Ernesto Manriquez (alejandronova)]

Your PDF is "only" 918 pages long.

I have the ultimate hell for the NEPOMUK text indexing engine: a collection of 11 PDFs with ~1,200 pages each of pure text, containing the History of the Chilean Constitution. That's really long, and those PDFs use 41 MB. Believe me: 41 MB of PURE TEXT with NO GRAPHICS and NO FONTS is really too much. Of course, those PDFs clogged the Virtuoso engine here, but only for a minute, and half of that time it wasn't actually Virtuoso what was forcing my CPU, but PDF2Text.

NEPOMUK guys, if you wish, you can download those PDFs directly from http://www.bcn.cl/lc/cpolitica/actas_oficiales-r . It doesn't matter that you don't understand Spanish, those PDFs would be great to torture NEPOMUK and to watch the real impact of small optimizations in the text indexing engine.

[Sebastian Kügler (sebasje)]

@all: This thread is NOT a collection of "everything that makes Nepomuk eat CPU". Please file individual bugreports if you find strigi / Nepomuk choking on a file. That's a completely different issue from this one though, and only makes it harder to get the useful information out of *this* bugreport.

This bugreports is about Nepomuk eating CPU due to a hanging query. Those hanging queries are, for example causes by the contacts or Nepomuk runners. As workaround, these have been disabled in 4.6.0, and we're looking for a structural fix in 4.6.1.

Please, before reacting to this bugreport, makes sure it's this exact issue. If it's not -> file another bugreport (after checking for possible dupes). That makes it much easier to track this bug, and fix it. Thanks.

[Ernesto Manriquez (alejandronova)]

@Sebastian: Sorry about helping somewhat to derail this thread.

I only wanted to report that, in my system, the Nepomuk E-Mail feeder can trigger the same behavior (a hanging query). Alvaro reports that the Nepomuk Contacts feeder can also trigger the same behavior.

There's a pattern here: just like KRunner plugins, those feeders exercise the same parts of NEPOMUK and can produce the same bugs. So, if KRunner plugins are fixed, those runners must be fixed too as part of this bug.

[Sebastian Kügler (sebasje)]

@Alejandro: The runners are only the symptom (and the akonadi feeders other symptoms, apparently).

There's fix available as attachment to this thread (be sure to pick the latest version) which needs some testing. It would be very helpful if you could try this patch and confirm if it fixes your email feeder symptoms as well. (And how long the conversion takes.)

[0cs935kb51-marc-wz6bkyhu4u (0cs935kb51-marc-wz6bkyhu4u)]

I thought I had reported my experience with the patches but apparently I forgot. Well, I tried both patches and they didn't help a bit with the kdepim 4.6 beta3. In fact, the behaviour becomes much worse :S. Without the akonadi version of kdepim everything runs just fine though.

Regards

[Trueg (trueg)]

@Marc: just to be clear: you patched both kdelibs and kdebase, reinstalled, and then restarted Nepomuk and the problem still persists?

[0cs935kb51-marc-wz6bkyhu4u (0cs935kb51-marc-wz6bkyhu4u)]

(In reply to comment #81)
> @Marc: just to be clear: you patched both kdelibs and kdebase, reinstalled, and
> then restarted Nepomuk and the problem still persists?

I did so ( and I can tell you it takes some time in my near 5 years old laptop). Not only restarted but rebooted the machine (ya know, just in case).

I still have the packages I made ready to install again. I can test them again tonight or probably tomorrow morning (both with kdepim 4.4.10 and kdepim 4.6beta3).

Regards

[Trueg (trueg)]

@Marc: could I please get some feedback with the latest patches?

[0cs935kb51-marc-wz6bkyhu4u (0cs935kb51-marc-wz6bkyhu4u)]

@Sebastian sorry for the delay.

I've tried the patches and everything seems to be fine as long as it's a new user/config.

When I tried with my regular user account virtuoso went crazy for over one and a half hours... untill I killed the processes and started fresh. I guess I should try it more hours to be sure all is ok. I'll report back here soon

[Trueg (trueg)]

@Marc: could you please while virtuoso is going crazy check its status via:

> isql localhost:<PORT> dba dba
> status();

where <PORT> is the port virtuoso was started on. You can get this by checking the config file you see in "ps aux|grep virtuoso"

[0cs935kb51-marc-wz6bkyhu4u (0cs935kb51-marc-wz6bkyhu4u)]

@Sebastian: I'm sorry to tell you that I started from zero with my own user and haven't found the virtuoso problem yet. (this is with your patches applied)

My apologies :(

Regards

[Trueg (trueg)]

@Marc: isn't this a good thing? Does that mean you cannot reproduce? Or does it still happen if you use your existing data?

[0cs935kb51-marc-wz6bkyhu4u (0cs935kb51-marc-wz6bkyhu4u)]

@Sebastian: it is for me! I just thought you wanted to debug why happened in the first place while upgrading to a new version. I've run into some trouble with the akonadi kdepim version but the virtuoso-t eating all of my cpu cycles is here no more! :)

Thanks for the work!

[Trueg (trueg)]

Great. So I can apply the patch for KDE 4.6.1. Thanks for all the testing guys.

[Trueg (trueg)]

Patches are commited to git master and KDE/4.6. 4.6.1 will contain the fixes.

[moenchmeyer (rm-anracon)]

Can confirm that everything is OK now with
Opensuse 11.4, KDE 4.6.1 and a Cyrus imap server.

Good job !

[Ernesto Manriquez (alejandronova)]

Kudos! Sorry about not testing the fix, but I didn't know how to pull a remote branch directly using git until now.

This bugfix is seriously lacking publicity and I tried my best to counter that. Again, thanks!

[Edney Matias da Silva (edney)]

Is this patch already part of KDE SC 4.6.1?

Thank you for the fix.

[Ettore Atalan (atalanttore)]

This problem still exists in Kubuntu 11.04.

[Ettore Atalan (atalanttore)]

Created attachment 60798
syslog

This problem still exists in KDE 4.6.2 (running Kubuntu 11.04).

[Meyerm-3 (meyerm-3)]

Same behaviour on KDE 4.6.4 and KDEPIM 4.6.0. The runners are disabled, strigi not running (disabled in system settings; nepomuk enabled of course). virtuoso-t is using up one of my two cores. Second to virtuoso-t concerning the totally used up cpu time on my computer is now dbus-daemon (not before the update) and third nepomukservicestub.

[Meyerm-3 (meyerm-3)]

Oh, sorry. "nepomukservicestub nepomukstorage" to be precise ;-)

[moenchmeyer (rm-anracon)]

The same problem here.

Today, after a long time I tested KDEPIM 4.6.0 again with KDE 4.6.4 on an Opensuse 11.4 system. (Basic KDE 4.6.4 components from Opensuse's KDE factory repository and KDEPIM 4.6.0 from an KDE-"Unstable" repository of Opensuse.)

Kmail was connected to a cyrus imap server with tons of mails. All the imap folders were nevertheless built up and filled in Kmail/Akonadi surprisingly rapidly. However, when starting KDE itself again nepomuk and virtuoso-t ran amok.

I have an Intel i7 950 quadcore with hyperthreading (8 threads). The cpu consumption of vituoso-t and nepomuk appear to be distributed over all cpu threads. However, in sum and on average the load of virtuoso-t consumes at least one cpu thread completely. According to ksysguard the following processes determine the cpu load (8x100%) :

virtuoso-t : 96% - 126 % (i.e. on average more than one cpu thread of 8 is used completely)

nepomukservices : 8% - 12 % (kind of a leading nepomukservices process)

dbus-daemon : 2% - 8%

several other nepomukservices processes: each with 2% - 4%

akonadi-nepomuk: 2% - 4%

This pattern was active over 20 minutes with no end in sight. The same after a new start of the system. I gave up then and went back to KDEPIM 4.4.11 (with Kmail 1.13.7).

So my impression is that the problem is not fixed or resolved.

Regarding my comment #91:
After looking into some logs I now think I reported wrongly. My positive impression at that time probably came from an Opensuse system that mixed KDE 4.6 with KDEPIM 4.4. Sorry for any confusion comment #91 may have caused.

Off topic: KDEPIM 4.6.0 apparently does not offer any chance to connect to Open-Xchange calendars or OX address books. So OX users can't use it productively, anyway.

[Andre Heinecke (aheinecke)]

Hi,
I have gotten reports from multiple users of KDEPIM on Windows that they see the same behavior (CPU Spiking of Nepomuk for a short time and blocking the system)
The Version they use is latest 4.6 branches.
We are probably talking about another bug here then the one fixed for kde 4.6 but there are still issues with the cpu consumption at least when kdepim is used.

Is there any advice on how to get better debug information on this? Like turn on some Virtuoso logging to see what is going on?

With the reports from #97 #95 and #94 I've reopened this bug, I hope that was ok or should i also have then removed the kde-4.6.0-blocker keyword. I just want to avoid that this bug gets lost as resolved.

Regards

[Andre Heinecke (aheinecke)]

I was able to reproduce a bug that specific to the complaints i've gotten and opened bug 276620 for this which is more specific (Marking Mails with KMail causes virtuoso to spike in cpu usage):
 https://bugs.kde.org/show_bug.cgi?id=276620

Apologies for reopening this bug i am marking it as resolved again because the issue mentioned for most of this report is appearantly fixed and i am seeing a different virtuoso cpu spike.

[rabauke (sven-burmeister)]

Unfortunately the bug does seem to exist in KDE 4.6.4. It was said that one can test whether it really got stuck by disabling nepomuk. If virtuoso-t stops some time after that as well, everything is "ok" since it was just working on something and not just stuck. But now it seems that virtuoso-t is stuck in some loop, i.e. it uses one full core even 10 minutes after nepomuk was disabled and strace does not show any activity as well.

ps aux | grep virtuos
user 2454 1.6 4.1 250936 84940 ? SNl Jun29 135:54 /usr/bin/virtuoso-t +foreground +configfile /tmp/virtuoso_nn2440.ini +wait

strace -p 2454 -tt -s 1000
Process 2454 attached - interrupt to quit
15:04:42.952129 futex(0x108a424, FUTEX_WAIT_PRIVATE, 2439, NULL

and no change after minutes.

Attaching gdb to the process is somehow not possible, i.e. it reports:

Attaching to process 2454
/usr/bin/virtuoso-t (deleted): file or directory not found.

I can see the process in the list though and that file exists.

[Edney Matias da Silva (edney)]

Hi there.

I can confirm the strange behavior but also can confirm that disabling nepomuk, virtuoso-t stops some time later. Is it possible to find out where he is stuck, what he is doing?

Anyway I will let it do its thing.

Thanks.

[moenchmeyer (rm-anracon)]

Follow up of comment #97:

Meanwhile, I have upgraded from KDE 4.6 to KDE 4.7. And for KDE 4.7 I cannot reproduce the strange CPU consuming behavior of Akonadi/Nepomuk/virtuoso-t anymore.

I use Kmail 4.7. I connected it via an Akonadi resource to an IMAP server with tons of mails. After the initial Akonadi update with all IMAP mail information from some hundred mailfolders there was indeed some background activity of Nepomuk/virtuoso-t.

But: It occured mainly during periods of low system activity and it did not last forever. It stopped after some time - maybe 10 minutes. No comparison to KDE 4.6.x. So, with KDE 4.7 I see a much better and maybe more intelligent performance regarding Nepomuk/virtuoso-t.

Now, when starting KDE I sometimes see some minor Nepomuk activity for a short period of time (< 30 seconds). But this is not really worth mentioning.

This is something I can live with. Good job.

Off topic: Kontact 4.7 still does not work with Open-Xchange 5 servers. Only with Open-Xchange 6 servers. But for OX 6 it really works very well.

[Marcin Gil (marcin.gil)]

Still have this appearing in Oneiric.

[Richard Bacchetta (richb1908)]

Same happening with KDE 4.8 RC2 in Kubuntu 11.10. Virtuoso-t consumes 50% cpu during any type of computer use. It happens every time Nepomuk is enabled.
Disabling Nepomuk while this is happening keeps virtuoso-t a running process indefinitely. If I kill it, it just restarts. Logging out and back in the session after disabling Nepomuk corrects the problem.

[Ernesto Manriquez (alejandronova)]

@<email address hidden>: Some things you can do.

There are 2 things that will make your Virtuoso install to use 100% CPU in KDE 4.8. Both are normal.

- Virtuoso is indexing files.
- Virtuoso is indexing mail.

1. To counter the heat, run the following command as root:

echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load

That will stop your processor speed to scale up, if it gets used by Virtuoso.

2. See what Virtuoso is doing.

- If you have a "&" like icon in your system tray, Virtuoso is indexing files.
- If you don't have that, run akonadiconsole and look for a resource named "Akonadi Nepomuk Feeder". If it's indexing mail, leave it alone. It will stop working when you are using the computer and it will resume indexing when you leave.

The third cause (Virtuoso eating CPU like crazy) shouldn't exist in KDE 4.8. Check if you have Akonadi 1.6.90 or higher.

[Richard Bacchetta (richb1908)]

The third case may be the problem. I have akonadi-server 1.6.2-0ubuntu1. It
is the only version available through the beta ppa for KDE 4.8
Richard

On Sat, Jan 14, 2012 at 12:20 PM, Alejandro Nova <email address hidden>wrote:

> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #104 from Alejandro Nova <alejandronova gmail com> 2012-01-14
> 17:20:51 ---
> @<email address hidden>: Some things you can do.
>
> There are 2 things that will make your Virtuoso install to use 100% CPU in
> KDE
> 4.8. Both are normal.
>
> - Virtuoso is indexing files.
> - Virtuoso is indexing mail.
>
> 1. To counter the heat, run the following command as root:
>
> echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load
>
> That will stop your processor speed to scale up, if it gets used by
> Virtuoso.
>
> 2. See what Virtuoso is doing.
>
> - If you have a "&" like icon in your system tray, Virtuoso is indexing
> files.
> - If you don't have that, run akonadiconsole and look for a resource named
> "Akonadi Nepomuk Feeder". If it's indexing mail, leave it alone. It will
> stop
> working when you are using the computer and it will resume indexing when
> you
> leave.
>
> The third cause (Virtuoso eating CPU like crazy) shouldn't exist in KDE
> 4.8.
> Check if you have Akonadi 1.6.90 or higher.
>
> --
> Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.
>

[Ernesto Manriquez (alejandronova)]

Please, file a bug against Kubuntu. Historically, there have been problems with 3 packages essential to Nepomuk/Akonadi: strigi; soprano and akonadi. Strangely, they are not covered by Kubuntu official PPAs, and if those packages are not updated to their latest releases, you WILL face problems. Currently:

- Strigi is at 0.7.7.
- Akonadi is at 1.6.90 (for KDE SC 4.8)
- Soprano is at 2.7.4 (2.7.5 is imminent).

[Richard Bacchetta (richb1908)]

Thanks Your suggestion:

echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load

Has kept the cpu and temps down.

I will file the bug report.
Richard

On Sun, Jan 15, 2012 at 2:27 PM, Alejandro Nova <email address hidden>wrote:

> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #106 from Alejandro Nova <alejandronova gmail com> 2012-01-15
> 19:27:09 ---
> Please, file a bug against Kubuntu. Historically, there have been problems
> with
> 3 packages essential to Nepomuk/Akonadi: strigi; soprano and akonadi.
> Strangely, they are not covered by Kubuntu official PPAs, and if those
> packages
> are not updated to their latest releases, you WILL face problems.
> Currently:
>
> - Strigi is at 0.7.7.
> - Akonadi is at 1.6.90 (for KDE SC 4.8)
> - Soprano is at 2.7.4 (2.7.5 is imminent).
>
> --
> Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.
>

[Ernesto Manriquez (alejandronova)]

Already filed it:

https://bugs.launchpad.net/kubuntu-packaging/+bug/916903

Please, vote for that report!

[Edney Matias da Silva (edney)]

Hi there!

Right now running Fedora 16 with KDE 4.8 RC 2 from Red Hat KDE Repository. These are my packages version

strigi-libs-0.7.7-1.fc16.x86_64
kdegraphics-strigi-analyzer-4.7.97-1.fc16.x86_64
akonadi-1.6.90-1.fc16.x86_64
kdepimlibs-akonadi-4.7.97-1.fc16.x86_64
soprano-2.7.4-1.fc16.x86_64

There's no '&' on my tray, it's hidden and stating that the file indexer is idle. Akonadi Nepomuk Feeder states "system busy, indexing suspended". Virtuoso is at 99.3% of my 4 core CPU, since i activated it again. My machine was shut off once since then because of overheating.

[Edney Matias da Silva (edney)]

...since my last comment, virtuoso is at 99.4% of my CPU.

[Martin L. (heinrich20)]

Hi all,

Directly translated from German we would say: "I am loosing all my hopes for KDE"

Ok, sorry, not very quailfied! But anyway I wonder what really happens in this area. I am working with a 4 years old dual-core Notebook and this problem really hurts. But Edney's machine is much faster and the system brings the machine down anyway. Is this system design or the rollout strategy for virtuoso, nepomuk, akonadi and all the rest of this nicely designed system framework?

Regards, Martin
-------- Original-Nachricht --------
> Datum: Mon, 16 Jan 2012 16:29:12 +0000
> Von: Edney Matias <email address hidden>
> An: <email address hidden>
> Betreff: [Bug 246678] virtuoso: Usage of CPU is much too high

> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #110 from Edney Matias <edneymatias gmail com> 2012-01-16
> 16:29:10 ---
> ...since my last comment, virtuoso is at 99.4% of my CPU.
>
> --
> Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You reported the bug.

[rabauke (sven-burmeister)]

If you want to "use" nepomuk you have to remove its database from time to time. You will lose your tags etc. and you might have high CPU usage because it re-indexes all files – but at least that cpu usage ends at some point. And to be honest, nepomuk is of no use as desktop search anyway, compared to state-of-the-art desktop searches.

Sadly it happens very often that at some point or after an upgrade from KDE 4.x to 4.y virtuoso uses the cpu constantly, never comes to an end.

There are some hints at http://kdeatopensuse.wordpress.com/2011/11/09/debugging-nepomukvirtuosos-cpu-usage/ on how to debug.

But as I mentioned, the quickest way of getting around that buggy piece of software is to just remove its database from time to time. Does not sound nice and I wish it was different, but these issues have been around for years and are still not resolved.

[rabauke (sven-burmeister)]

Forgot the most important part:

As user issue rm -rf ~/.kde4/share/apps/nepomuk or use dolphin to remove that folder and get rid of all nepomuk data.

If you want to play nice you can disable nepomuk before doing so and re-enable it afterwards.

For some distros it might be ~/.kde/…

And BTW, please do not confuse nepomuk with akonadi which did not have years yet to stabilise and did in fact get quite stable within a short time and works very well in 4.8 – unlike nepomuk.

[Ernesto Manriquez (alejandronova)]

@S.Burmeister: from my experience, there have been multiple super-bugs with Nepomuk, and now all of them lie in the Akonadi Nepomuk Feeder. Make the test; open akonadiconsole, disable the Akonadi Nepomuk Feeder and in all cases your CPU use will decrease to zero in a matter of minutes (if you have some files left to index, your HDD will get used, but Nepomuk will scale up and down quickly).

If there is a way to permanently disable the Akonadi Nepomuk Feeder, that would bring an end to all of this.

[rabauke (sven-burmeister)]

Nope, sorry. The feeder did/does have its bugs but its not the sole/main cause of all nepomuk issues and virtuoso cpu hogging.

[Martin L. (heinrich20)]

Hi Folks,

the Akonadi framework is the main reason for me to quit KMail. Sinse
1999 I have been using KMail, most of the time I thought, this was one
of the best mail clients to get. Easy to use and working!

But with KDE 4 and "I don't now what" corresponding version of PIM/KMail
(4.?) the situation ended up getting catastrophically. Know I will move
to thunderbird and one step further after more than ten years of using
Linux/KDE might be to move to Windows 7. Not a fault of M$! I might
leave a platform that I thought it really was an alternative one.

Regards, Martin

PS: Last bug I found was in kdewallet, ending up to become very slow. I
wonder which application on my system really works.

m 16.01.2012 20:10, schrieb S. Burmeister:
> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #115 from S. Burmeister<sven burmeister gmx net> 2012-01-16 19:10:24 ---
> Nope, sorry. The feeder did/does have its bugs but its not the sole/main cause
> of all nepomuk issues and virtuoso cpu hogging.
>

[rabauke (sven-burmeister)]

Please do not spam this bug with unrelated issues but only info that helps to fix or work around this bug!

Please do only quote text you refer to.

Akonadi/kwallet/whatsoever bugs belong into another bug report and everything OT belongs on some personal blog, forum or mailinglist but not the bug tracker.

[Martin L. (heinrich20)]

Hi Sven,

would you kindly have a look at the history of this issue? I know what I
did! I myself started this bug report! How long ago? Was this problem
really solved? Maybe, maybe not! Noone looks at this situation from a
global perspective that tells us that KDE really runs out of acceptance
because if one bug is fixed two others are opened.

Maybe this is Spam and there is noone to be blamed because nowadays
there are really not enough developers to solve all the problems that
are the result of a ambitious project. A pity!

Martin

Am 16.01.2012 23:47, schrieb S. Burmeister:
> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #117 from S. Burmeister<sven burmeister gmx net> 2012-01-16 22:47:15 ---
> Please do not spam this bug with unrelated issues but only info that helps to
> fix or work around this bug!
>
> Please do only quote text you refer to.
>
> Akonadi/kwallet/whatsoever bugs belong into another bug report and everything
> OT belongs on some personal blog, forum or mailinglist but not the bug tracker.
>

[Richard Bacchetta (richb1908)]

Martin,
Yous is an editorial comment and does not further the solution of this problem. And I just editorialized again.

To get back on track, The suggestions made in an earlier post

echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load

Has "corrected' the problem for me in KDE 4.8 RC2. Corrected in quotes as it has prevented high cpu and overheating. Also the latest strigi, akonadi and soprano packages were not included in the upgrade to KDE that my distro provided and may just be a packaging issue. So it is still an open issue.

Let's give information to the developers that resolves the problem please.

[Edney Matias da Silva (edney)]

Hi there.

Writing to let you know that after many hours virtuoso now sit quite on my process list. This already happened before and an update to the system put it crazy again.

Thank you.

[Ernesto Manriquez (alejandronova)]

Basically this issue connects several bugs. Some of the most memorable ones are:

- Virtuoso being locked up with strigi, one core sleeping - killed with KDE 4.6.1.
- Various memory leaks, some with Virtuoso, some with nepomukstorage, and even some with dbus-daemon. The last one of those was fixed with Soprano 2.7.3.
- Virtuoso being locked up with akonadi-nepomuk-email-feeder, one core wasted. That one haunted KDE 4.7.x until ~4.7.4.
- Virtuoso being locked up with akonadi-nepomuk-feeder, one core wasted. That is the latest incarnation of this bug, and is what <email address hidden> was experiencing.

All of these real issues were amplified by more downstream issues.

- Strigi lacks a real release policy. During most of 2010 and part of 2011, Debian shipped Strigi 0.7.2, an extremely buggy and ancient release.
- Akonadi and Soprano are not updated with KDE, in KDE dependent distros.

So, you can't really fix this, unless:

- You require a Virtuoso + Soprano + Akonadi + Strigi + Shared Desktop Ontologies stack, and maintain it with a KDE 4.x release cycle.
- You assign a real maintainer to akonadi-nepomuk-feeder (that package is somewhat orphan, unlike Nepomuk).
- You REQUIRE distros shipping KDE to UPDATE their packages. That can be made easily through CMakeFiles (if you ship KDE 4.8, then you must have certain versions of Soprano, Akonadi, Strigi and shared-desktop-ontologies)
- You MAKE a "Update my Ontologies" app like what Bangarang has. That's a necessity.

Once you have all of this sorted out, you can really begin with bug triaging and reporting. Reporting bugs without all of these requirements unmet will be a waste of time for the reporter and for the developer.

About the Nepomuk database erasing: if I read correctly Sebastian Trueg's blogs and dev history, KDE SC 4.8 is going to be the last release that will require erasing everything to work well. The Akonadi Nepomuk Feeder is fresh code, made with something called DMS (AFAIK, replacing hand tuned SQL queries with queries generated automatically)

Please, be more constructive about how to fix this. I really hate Nepomuk bugs, I really want the thing to work, but we won't get anywhere if we just kill the thing. Remember that Nepomuk is the very thing (WinFS) that Microsoft FAILED to implement, and Sebastian Trueg has made tremendous strides to do what Microsoft failed to do, with no resources, and (as of late) with no money.

[Martin L. (heinrich20)]

Hi Alejandro,

thanks a lot for the information! Some of it I do understand, some I don't. I myself have the project lead in a commercial development project where not all the time I do understand the dependencies. The whole KDE 4.x - project, is there anyone who understands the dependencies and the release plan (if there is one)?

Talking about this issue: What shall we do, reopen it or keep it running in parallel?

Only someone who really understands how the different parts work together could track an issue like that. I myself do not have time and (no longer) patience to do so.

I am a KDE/Linux user who tried to help but stopped writing bug reports because there are too many. I have no idea where to start and where this might end!

Regards, Martin
-------- Original-Nachricht --------
> Datum: Tue, 17 Jan 2012 00:35:10 +0000
> Von: Alejandro Nova <email address hidden>
> An: <email address hidden>
> Betreff: [Bug 246678] virtuoso: Usage of CPU is much too high

> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #121 from Alejandro Nova <alejandronova gmail com> 2012-01-17
> 00:35:08 ---
> Basically this issue connects several bugs. Some of the most memorable
> ones
> are:
>
> - Virtuoso being locked up with strigi, one core sleeping - killed with
> KDE
> 4.6.1.
> - Various memory leaks, some with Virtuoso, some with nepomukstorage, and
> even
> some with dbus-daemon. The last one of those was fixed with Soprano 2.7.3.
> - Virtuoso being locked up with akonadi-nepomuk-email-feeder, one core
> wasted.
> That one haunted KDE 4.7.x until ~4.7.4.
> - Virtuoso being locked up with akonadi-nepomuk-feeder, one core wasted.
> That
> is the latest incarnation of this bug, and is what <email address hidden> was
> experiencing.
>
> All of these real issues were amplified by more downstream issues.
>
> - Strigi lacks a real release policy. During most of 2010 and part of
> 2011,
> Debian shipped Strigi 0.7.2, an extremely buggy and ancient release.
> - Akonadi and Soprano are not updated with KDE, in KDE dependent distros.
>
> So, you can't really fix this, unless:
>
> - You require a Virtuoso + Soprano + Akonadi + Strigi + Shared Desktop
> Ontologies stack, and maintain it with a KDE 4.x release cycle.
> - You assign a real maintainer to akonadi-nepomuk-feeder (that package is
> somewhat orphan, unlike Nepomuk).
> - You REQUIRE distros shipping KDE to UPDATE their packages. That can be
> made
> easily through CMakeFiles (if you ship KDE 4.8, then you must have certain
> versions of Soprano, Akonadi, Strigi and shared-desktop-ontologies)
> - You MAKE a "Update my Ontologies" app like what Bangarang has. That's a
> necessity.
>
> Once you have all of this sorted out, you can really begin with bug
> triaging
> and reporting. Reporting bugs without all of these requirements unmet will
> be a
> waste of time for the reporter and for the developer.
>
> About the Nepomuk database erasing: if I read correctly Sebastian Trueg's
> blogs
> and dev history, KDE SC 4.8 is going to be the last release that will
> require
> erasing everything to work well. The Akonadi Nepomuk Feeder is fresh code,
> made
> with something called DMS (AFAIK...

[rabauke (sven-burmeister)]

Martin, would you please stop quoting the full comment of somebody else!

Have a look at https://bugs.kde.org/show_bug.cgi?id=246678#c122 if you do not know what I'm talking about!

If virtuoso shows high CPU usage you have to debug it, find the queries that causes the cpu usage etc. A start would be http://kdeatopensuse.wordpress.com/2011/11/09/debugging-nepomukvirtuosos-cpu-usage/ as already pointed out.

[Martin L. (heinrich20)]

Sven,

I can see, what you pointed out. This is, what happened within the GMX client and I did not take care about. Sorry for that! But anyway, when we start to discuss about quoting like that or not, the way we communicate has reached a quality that not at all helps to solve the problem.

Two things where I wonder, how to improve the situation out of the persepctive (=role) of a user:

1. Debugging as described by your link is far too complicated to make that happen by a user.

2. How an where would I start? When would I start debugging? Which application to debug first. This is not cynism, I do think this to be a real problem for users.

/Off-Topic
Even though we are "only" users of KDE/Linux we do a real hard job. We stay on a platform (in case of KDE 4.x) that is shipped with Ubuntu but is really far away from being stable. If e.g. I try to write bug reports I even run into problems with that because I c a n n o t describe the bug itself appearing in a very complex system environment that is far to complecated. See 1. So by the time you wonder what you could do and find out: nothing!
/

Regards, Martin

[Edney Matias da Silva (edney)]

Hi there.

I tried to follow the instructions to debug the problem but couldn't find the isql-vt tool neither the package that contains it. I know it's a distribution problem, but it's just another one that sums up. Running Fedora 16 here.

Regards.

[Rdieter-math (rdieter-math)]

on fedora,
yum install virtuoso-opensource-utils

look in /usr/libexec/virtuoso/
for isql, isqlw, inifile

I'm not sure what isql-tw is supposed to refer to, but I'm guessing just the 'isql' tool (maybe suse renamed it or something).

[Edney Matias da Silva (edney)]

Hi there!

Trying to exec this line isql-vt -H localhost -P 1111 -U dba -P dba from the tutorial available here http://kdeatopensuse.wordpress.com/2011/11/09/debugging-nepomukvirtuosos-cpu-usage/ gives me that on Fedora 16:

[matias@padme ~]$ /usr/libexec/virtuoso/isql -H localhost -P 1114 -U dba -P dba

*** Error IM002: [iODBC][Driver Manager]Data source name not found and no default driver specified. Driver could not be loaded
at line 0 of Top-Level:

Any clues?

My virtuoso is crazy again after a restart on akonadi server from the control painel. :S

top command shows:

top - 14:38:57 up 2 days, 4:46, 3 users, load average: 1.67, 1.59, 1.10
Tasks: 198 total, 2 running, 196 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.4%us, 1.0%sy, 49.6%ni, 46.6%id, 0.0%wa, 0.3%hi, 0.1%si, 0.0%st
Mem: 3943656k total, 3776300k used, 167356k free, 52392k buffers
Swap: 4194300k total, 340968k used, 3853332k free, 990336k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1704 matias 39 19 1514m 513m 3888 S 198.5 13.3 317:24.94 virtuoso-t

Thank you

[Edney Matias da Silva (edney)]

two minutes after posting my last comment virtuoso shut my machine off! :S

[Rdieter-math (rdieter-math)]

comment #127 highlights a problem in fedora's packaging that I'm fixing now (we ship only the iodbc variants of these utilities, which apparently do not work in this context).

[Ernesto Manriquez (alejandronova)]

@Edney Matias: Run the command I posted. It won't do anything about your Virtuoso problems, but it will protect your computer from overheating.

[Kent57 (kent5700)]

Didn't have this problem in Oneiric until the latest update. Now virtuoso-t pegs cpu at 100%.

[wimo (pfuentesor)]

After update my kubuntu 11.10 t0 kde 4.8 i have the same problem.

[Scott Kitterman (kitterman)]

On Thursday, January 26, 2012 07:26:57 PM you wrote:
> After update my kubuntu 11.10 t0 kde 4.8 i have the same problem.

These issues should be filed as a new bug (similar symptoms don't necessarily
mean the same root cause).

[Edney Matias da Silva (edney)]

Hi!

I'm writing to let you know that virtuoso is fine now for a few days. I'm running Fedora 16 and here are my versions:

[matias@padme ~]$ rpm -qa | grep virtuoso
redland-virtuoso-1.0.14-1.fc16.x86_64
virtuoso-opensource-6.1.4-4.fc16.x86_64
virtuoso-opensource-utils-6.1.4-4.fc16.x86_64

[matias@padme ~]$ rpm -qa | grep akonadi
kdepimlibs-akonadi-4.8.0-1.fc16.x86_64
pykde4-akonadi-4.8.0-1.fc16.x86_64
akonadi-1.7.0-1.fc16.x86_64

[matias@padme ~]$ rpm -qa | grep kdelibs
kdelibs-devel-4.8.0-1.fc16.x86_64
kdelibs-common-4.8.0-1.fc16.x86_64
kdelibs-4.8.0-1.fc16.x86_64

[matias@padme ~]$ rpm -qa | grep kde-workspace
kde-workspace-4.8.0-7.fc16.x86_64
kde-workspace-libs-4.8.0-7.fc16.x86_64
kde-workspace-devel-4.8.0-7.fc16.x86_64

Anything else?

Thank you.

[Ernesto Manriquez (alejandronova)]

kde-runtime-4.8.0-3 ;)

Check for it.

[sanette (sanette-linux)]

kde 4.9.3, fresh install on a new laptop
(kubuntu 12.10)

* virtuoso-t is regularly eating 100% of one of my CPUs each time I do something with kmail... had to stop using kmail

* even without kmail, virtuoso-t uses 435 Mo of RAM, even though it seems to do nothing
(and in the systemsettings I have set to 50Mo the max RAM usage for nepomuk

[sanette (sanette-linux)]

ps: see attached graphics for the CPU usage (yellow curve)
this is a typical virtuoso-t burst when kmail is open (and I do nothing)

[bamyasi (iadzhubey)]

Kubuntu 12.10, KDE 4.9.4 fresh install on a 240GB Intel 520 SSD

I have installed a new 4TB data drive in my system today and started transferring files from my old (1TB) drive onto it, using rsync. Note: the new drive is mounted as /disk/; my old drive is mounted under /mnt/. None of the filesystems involved has any relation to my /home partition. All through the file transferring process, virtuosi-t process keeps eating 15-50% of CPU and iotop shows enormous amounts of disk I/O generated by 6-16 copies of virtuosi-t process run in parallel. The desktop is occasionally completely locked up for 1-2 seconds and unresponsive otherwise.

File indexing is configured to include only a handful of subdirectories under my /home directory. Still, virtuoso-t keeps trashing disk I/O subsystem during any file transfers, regardless of whether their source/target is withing the directories configured for indexing or not.

I consider this as a very serious and extremely annoying bug.

[maxis11 (ya-maxis11)]

virtuoso-t eats 50% of cpu in 4.11.2 and in dmesg

[39851.714273] virtuoso-t[12371]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f4b143aad50 error 7 in virtuoso-t[400000+b09000]
[39861.327093] virtuoso-t[12388]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f840fddbd50 error 7 in virtuoso-t[400000+b09000]
[39872.752638] virtuoso-t[12416]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f501f336d50 error 7 in virtuoso-t[400000+b09000]
[39882.341785] virtuoso-t[12452]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f538e8fdd50 error 7 in virtuoso-t[400000+b09000]
[39894.538014] virtuoso-t[12691]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fb4dc1c8d50 error 7 in virtuoso-t[400000+b09000]
[39904.505139] virtuoso-t[12728]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f42807ebd50 error 7 in virtuoso-t[400000+b09000]
[39914.684411] virtuoso-t[12773]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f9af605ad50 error 7 in virtuoso-t[400000+b09000]
[39924.794001] virtuoso-t[12800]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f0df1aaad50 error 7 in virtuoso-t[400000+b09000]
[39934.491591] virtuoso-t[12837]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fc61eabdd50 error 7 in virtuoso-t[400000+b09000]
[39944.314952] virtuoso-t[12853]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f724b51bd50 error 7 in virtuoso-t[400000+b09000]
[39953.429060] virtuoso-t[12881]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fecb1702d50 error 7 in virtuoso-t[400000+b09000]
[39962.463541] virtuoso-t[12910]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f21cf69dd50 error 7 in virtuoso-t[400000+b09000]
[39972.012866] virtuoso-t[12932]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f11b9f3bd50 error 7 in virtuoso-t[400000+b09000]
[39982.649058] virtuoso-t[12971]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fce9a230d50 error 7 in virtuoso-t[400000+b09000]
[39991.769957] virtuoso-t[12992]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fe2aa716d50 error 7 in virtuoso-t[400000+b09000]
[40002.143040] virtuoso-t[13026]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fc1e09b4d50 error 7 in virtuoso-t[400000+b09000]
[40012.900378] virtuoso-t[13057]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fed11668d50 error 7 in virtuoso-t[400000+b09000]
[40022.103676] virtuoso-t[13091]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f7dbf637d50 error 7 in virtuoso-t[400000+b09000]
[40032.376944] virtuoso-t[13130]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f70b8d63d50 error 7 in virtuoso-t[400000+b09000]
[40042.807790] virtuoso-t[13146]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f9d3a718d50 error 7 in virtuoso-t[400000+b09000]
[40052.312900] virtuoso-t[13173]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f3d6dd58d50 error 7 in virtuoso-t[400000+b09000]
[40062.013635] virtuoso-t[13203]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f857cea9d50 error 7 in virtuoso-t[400000+b09000]
[40073.095813] virtuoso-t[13241]: segfault at ffffffffffffffff ip 000...