virtuoso-t eats my cpu, should be nice

[aexl]

on kubuntu lucid there is a process "virtuoso-t" that eats my cpu (it is said to help nepomuk with indexing).
this background process has niceness 0 according to ps.
i think as a background indexer it should have maximum niceness of 19.

[Philip Muškovac (yofel)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. This bug did not have a package associated with it, which is important for ensuring that it gets looked at by the proper developers. You can learn more about finding the right package at https://wiki.ubuntu.com/Bugs/FindRightPackage. I have classified this bug as a bug in virtuoso-opensource.

When reporting bugs in the future please use apport, either via the appropriate application's "Help -> Report a Problem" menu or using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

[tdn (spam-thomasdamgaard)]

I can confirm this bug on fully updated kubuntu 10.04.

[aexl (aexl)]

added kdebase-runtime to affected projects, because the calling process of virtuoso-t ist nepomukservicestub, which belongs to this project.

[aexl (aexl)]

did not get lp to add this possible upstream link: https://bugs.kde.org/show_bug.cgi?id=228081

[Philip Muškovac (yofel)]

Upstream watch set and reassigned.

[Scott Kitterman (kitterman)]

We enabled indexing by default this cycle and I can confirm my CPU is periodically eaten as well.

[Jonathan Thomas (echidnaman)]

Technically indexing was broken until this cycle. ;)

[Scott Kitterman (kitterman)]

Yes, but we tried this with strigi before so the idea isn't new.

[Jeff Trull (jetrull)]

This is fairly frustrating. Hope something can be done about it. Suddenly my fan turns on... and it's "virtuoso-t" with 98%.

[Jonathan Thomas (echidnaman)]

The solution for now is to disable indexing by default. It has not been enabled in previous releases, and we can address it in kubuntu-default-settings in a way that users who currently have it enabled will not lose their settings.

The underlying issue of virtuoso not having a low enough priority still needs to be addressed, but I'm not as hopeful for an SRU for that bit.

[Jonathan Thomas (echidnaman)]

For default settings in maverick, we should try a "wait and see" approach and see if things improve.
I'll attach a patch for kubuntu-default-settings for lucid in a bit here.

[Martin Pitt (pitti)]

Accepted kubuntu-default-settings into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Rohan Garg (rohangarg)]

Hi
With the latest kubuntu-default-settings i can confirm this has been resolved.Strigi is disabled by default and does not cause high CPU loads

[Scott Kitterman (kitterman)]

The Lucid fix for this issue is verification-done (the wrong task was milestoned). It should be copied to -updates and included in 10.04.1.

[Jonathan Riddell (jr)]

Copied to lucid-updates

[Martin L. (heinrich20)]

Version: 4.1 (using KDE 4.4.5)
OS: Linux

Normal working with my notebook. Writing mails with KMail. Then suddenly the application is blocked totally. Nothing happenes. I can see that virtuoso uses more than 50% CPU. Together with KMail using more than 20% and I do not knowhow much memory Kmail is blocked and my system is blocked - annoying!

I did not switch of strigi but switched of indexing of multimedia files. No idea if this will help. Anyway, to work productively with my notebook shall I switch of this whole strigi/nepomuk stuff?

Reproducible: Always

Steps to Reproduce:
no way to reproduce it.

Actual Results:
The problem occurs during normal work.

OS: Linux (x86_64) release 2.6.32-5-amd64
Compiler: cc

[Loacoon (loacoon)]

Same here, virtuoso-t often uses 50 to 70% of each cores of my CPU for around 2 or 3 minutes.

[Trueg (trueg)]

It would be really interesting to track down in which situations Virtuoso takes this much CPU.
Does this only happen when using KMail or also if KMail is not running?

[Loacoon (loacoon)]

I don't use Kmail at all so this can't be it. It seems to happen when "playing" with files. Moving, erasing or copying a certain amount of files seems to trigger this.

[Trueg (trueg)]

Is this still on 4.4.x or 4.5.0?

[Loacoon (loacoon)]

4.5.0

[Trueg (trueg)]

can you try a patch for kdebase?

[Loacoon (loacoon)]

I'm using a Gentoo overlay ebuild, so I can try to patch it, but I'm not sure I'll manage to do it (it doesn't look complicated though).
By the way, I just switched to 4.5.1, in case it changes something to your patch.

[Nicolas Réau (kolia)]

KDE 4.5.1 here and noticing the same behavior. It was not indexing at the moment. The only particular thing I can think of when it happened is that it started just after I made a svn checkout of kdelibs (thus lots of files written suddenly).

Regards,

[HinzundKunz (martin-tlustos)]

I'm running Kubuntu 10.04 with the kde4.5 testing packages and virtuoso still peaks high, even when indexing is disabled. It probably has to do with akonadi being used (I'm trying out the akonadi resources like imap and stuff).
Anyway, I use cpulimit to prevent my cpu from overheating, so it doesn't threaten my computer.

[Christopher-tanner (christopher-tanner)]

Running a version called 4.5.68 on Mandriva Cooker, and am noticing the same behaviour. CPU usage is not dependent on whether or not kmail is running. Anything that I can look for to help debug this?

[Rudd-O (rudd-o)]

The absolute exact same thing happens to me. It makes KDE completely unusable. When you strace it, all you can see is a metric ton of futex calls and very little work actually being done. I really wish I could use KDE but the fact of the matter is, I can't let this destroy my laptop battery life.

Please already confirm this and put some real effort in fixing this problem, OK?

[Loacoon (loacoon)]

You still can use KDE, you just have to disable Strigi and Nepomuk. No more CPU problems, and ~150Mo or RAM saved (which is also way too much).

[Rudd-O (rudd-o)]

I fail to see the point of using KDE without these KDE technologies. I admit, there is not that much those technologies are useful for, right now, but saying "Turn off nepomuk and strigi" is sort of like saying "Well, you do without desktop search and semantic information, which is an advertised feature of KDE". You can drive your car... but forget about turning the radio and A/C on.

[Loacoon (loacoon)]

KDE is not only this, and as you said, it's still not quite useful. But you're right, Linux gives this choice, and there are many desktop environments much more stable than KDE.

[AlexHarrowell (a-harrowell)]

I've got this problem as well, w/KDE4.4.4 and OpenSUSE11.3/2GB RAM/Intel T1700.

Typically, it's associated with KMail, which will happily spend 15 minutes hammering the disk I/O and basically hanging every time it starts up. During that period you either have KMail pulling 25-30% CPU and a *lot* of disk activity, or else Virtuoso running up to 90~% CPU. Interestingly, it is NOT REPEAT NOT hitting the swap - during these events there is typically well over 1GB of RAM available.

This has recently started to be a problem - I don't know whether this is as a result of an update or whether there's some critical folder or mailbox size.

I take the point that you could turn off desktop search, but I don't *have* desktop search and haven't since KDE3.5.7 and good old kerry beagle. I'd like to have a desktop search application rather than KFind. Strigi is disabled through System Settings.

I am at a loss to see the purpose of the whole complex of Akonadi/Strigi/Nepomuk/Virtuoso. It just appears to be a selection of processes that burn enormous quantities of system resources and do nothing whatsoever of any use, except mangling my data now and again.

[Mark Greenwood (fatgerman)]

I've got this problem on Maverick, with KDE 4.5.3. With file indexing enabled virtuoso-t keeps the CPU at 99% indefinitely *even* if I then disable file indexing again. I have to reboot after disabling indexing.
Also, if I enable the option in digikam to store metadata in nepomuk, the same thing happens.
However I've found that if increase the memory available to nepomuk (from System Settings) this problem goes away. I have, however, had to allow it to use 400MB of RAM, which is way unacceptable.
I think, if you only index a very small number of files the default amount of memory is probably fine, but I now have 18,000 files in my database (it has indexed all my MP3s and Photos, which is what I wanted it to do).

[GDR! (gdr.name)]

virtuoso_t uses all CPU cores and all I/O when looking at iotop, making the system unresponsive and unusable. Can't it just run nice and ionice? I'm using 10.10.

[Ettore Atalan (atalanttore)]

This problem still exists in Kubuntu 11.04.

[moenchmeyer (rm-anracon)]

The same problem here.

Today, after a long time I tested KDEPIM 4.6.0 again with KDE 4.6.4 on an Opensuse 11.4 system. (Basic KDE 4.6.4 components from Opensuse's KDE factory repository and KDEPIM 4.6.0 from an KDE-"Unstable" repository of Opensuse.)

Kmail was connected to a cyrus imap server with tons of mails. All the imap folders were nevertheless built up and filled in Kmail/Akonadi surprisingly rapidly. However, when starting KDE itself again nepomuk and virtuoso-t ran amok.

I have an Intel i7 950 quadcore with hyperthreading (8 threads). The cpu consumption of vituoso-t and nepomuk appear to be distributed over all cpu threads. However, in sum and on average the load of virtuoso-t consumes at least one cpu thread completely. According to ksysguard the following processes determine the cpu load (8x100%) :

virtuoso-t : 96% - 126 % (i.e. on average more than one cpu thread of 8 is used completely)

nepomukservices : 8% - 12 % (kind of a leading nepomukservices process)

dbus-daemon : 2% - 8%

several other nepomukservices processes: each with 2% - 4%

akonadi-nepomuk: 2% - 4%

This pattern was active over 20 minutes with no end in sight. The same after a new start of the system. I gave up then and went back to KDEPIM 4.4.11 (with Kmail 1.13.7).

So my impression is that the problem is not fixed or resolved.

Regarding my comment #91:
After looking into some logs I now think I reported wrongly. My positive impression at that time probably came from an Opensuse system that mixed KDE 4.6 with KDEPIM 4.4. Sorry for any confusion comment #91 may have caused.

Off topic: KDEPIM 4.6.0 apparently does not offer any chance to connect to Open-Xchange calendars or OX address books. So OX users can't use it productively, anyway.

[Andre Heinecke (aheinecke)]

Hi,
I have gotten reports from multiple users of KDEPIM on Windows that they see the same behavior (CPU Spiking of Nepomuk for a short time and blocking the system)
The Version they use is latest 4.6 branches.
We are probably talking about another bug here then the one fixed for kde 4.6 but there are still issues with the cpu consumption at least when kdepim is used.

Is there any advice on how to get better debug information on this? Like turn on some Virtuoso logging to see what is going on?

With the reports from #97 #95 and #94 I've reopened this bug, I hope that was ok or should i also have then removed the kde-4.6.0-blocker keyword. I just want to avoid that this bug gets lost as resolved.

Regards

[Andre Heinecke (aheinecke)]

I was able to reproduce a bug that specific to the complaints i've gotten and opened bug 276620 for this which is more specific (Marking Mails with KMail causes virtuoso to spike in cpu usage):
 https://bugs.kde.org/show_bug.cgi?id=276620

Apologies for reopening this bug i am marking it as resolved again because the issue mentioned for most of this report is appearantly fixed and i am seeing a different virtuoso cpu spike.

[rabauke (sven-burmeister)]

Unfortunately the bug does seem to exist in KDE 4.6.4. It was said that one can test whether it really got stuck by disabling nepomuk. If virtuoso-t stops some time after that as well, everything is "ok" since it was just working on something and not just stuck. But now it seems that virtuoso-t is stuck in some loop, i.e. it uses one full core even 10 minutes after nepomuk was disabled and strace does not show any activity as well.

ps aux | grep virtuos
user 2454 1.6 4.1 250936 84940 ? SNl Jun29 135:54 /usr/bin/virtuoso-t +foreground +configfile /tmp/virtuoso_nn2440.ini +wait

strace -p 2454 -tt -s 1000
Process 2454 attached - interrupt to quit
15:04:42.952129 futex(0x108a424, FUTEX_WAIT_PRIVATE, 2439, NULL

and no change after minutes.

Attaching gdb to the process is somehow not possible, i.e. it reports:

Attaching to process 2454
/usr/bin/virtuoso-t (deleted): file or directory not found.

I can see the process in the list though and that file exists.

[Edney Matias da Silva (edney)]

Hi there.

I can confirm the strange behavior but also can confirm that disabling nepomuk, virtuoso-t stops some time later. Is it possible to find out where he is stuck, what he is doing?

Anyway I will let it do its thing.

Thanks.

[moenchmeyer (rm-anracon)]

Follow up of comment #97:

Meanwhile, I have upgraded from KDE 4.6 to KDE 4.7. And for KDE 4.7 I cannot reproduce the strange CPU consuming behavior of Akonadi/Nepomuk/virtuoso-t anymore.

I use Kmail 4.7. I connected it via an Akonadi resource to an IMAP server with tons of mails. After the initial Akonadi update with all IMAP mail information from some hundred mailfolders there was indeed some background activity of Nepomuk/virtuoso-t.

But: It occured mainly during periods of low system activity and it did not last forever. It stopped after some time - maybe 10 minutes. No comparison to KDE 4.6.x. So, with KDE 4.7 I see a much better and maybe more intelligent performance regarding Nepomuk/virtuoso-t.

Now, when starting KDE I sometimes see some minor Nepomuk activity for a short period of time (< 30 seconds). But this is not really worth mentioning.

This is something I can live with. Good job.

Off topic: Kontact 4.7 still does not work with Open-Xchange 5 servers. Only with Open-Xchange 6 servers. But for OX 6 it really works very well.

[Marcin Gil (marcin.gil)]

Still have this appearing in Oneiric.

[Richard Bacchetta (richb1908)]

Same happening with KDE 4.8 RC2 in Kubuntu 11.10. Virtuoso-t consumes 50% cpu during any type of computer use. It happens every time Nepomuk is enabled.
Disabling Nepomuk while this is happening keeps virtuoso-t a running process indefinitely. If I kill it, it just restarts. Logging out and back in the session after disabling Nepomuk corrects the problem.

[Ernesto Manriquez (alejandronova)]

@<email address hidden>: Some things you can do.

There are 2 things that will make your Virtuoso install to use 100% CPU in KDE 4.8. Both are normal.

- Virtuoso is indexing files.
- Virtuoso is indexing mail.

1. To counter the heat, run the following command as root:

echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load

That will stop your processor speed to scale up, if it gets used by Virtuoso.

2. See what Virtuoso is doing.

- If you have a "&" like icon in your system tray, Virtuoso is indexing files.
- If you don't have that, run akonadiconsole and look for a resource named "Akonadi Nepomuk Feeder". If it's indexing mail, leave it alone. It will stop working when you are using the computer and it will resume indexing when you leave.

The third cause (Virtuoso eating CPU like crazy) shouldn't exist in KDE 4.8. Check if you have Akonadi 1.6.90 or higher.

[Richard Bacchetta (richb1908)]

The third case may be the problem. I have akonadi-server 1.6.2-0ubuntu1. It
is the only version available through the beta ppa for KDE 4.8
Richard

On Sat, Jan 14, 2012 at 12:20 PM, Alejandro Nova <email address hidden>wrote:

> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #104 from Alejandro Nova <alejandronova gmail com> 2012-01-14
> 17:20:51 ---
> @<email address hidden>: Some things you can do.
>
> There are 2 things that will make your Virtuoso install to use 100% CPU in
> KDE
> 4.8. Both are normal.
>
> - Virtuoso is indexing files.
> - Virtuoso is indexing mail.
>
> 1. To counter the heat, run the following command as root:
>
> echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load
>
> That will stop your processor speed to scale up, if it gets used by
> Virtuoso.
>
> 2. See what Virtuoso is doing.
>
> - If you have a "&" like icon in your system tray, Virtuoso is indexing
> files.
> - If you don't have that, run akonadiconsole and look for a resource named
> "Akonadi Nepomuk Feeder". If it's indexing mail, leave it alone. It will
> stop
> working when you are using the computer and it will resume indexing when
> you
> leave.
>
> The third cause (Virtuoso eating CPU like crazy) shouldn't exist in KDE
> 4.8.
> Check if you have Akonadi 1.6.90 or higher.
>
> --
> Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.
>

[Ernesto Manriquez (alejandronova)]

Please, file a bug against Kubuntu. Historically, there have been problems with 3 packages essential to Nepomuk/Akonadi: strigi; soprano and akonadi. Strangely, they are not covered by Kubuntu official PPAs, and if those packages are not updated to their latest releases, you WILL face problems. Currently:

- Strigi is at 0.7.7.
- Akonadi is at 1.6.90 (for KDE SC 4.8)
- Soprano is at 2.7.4 (2.7.5 is imminent).

[Richard Bacchetta (richb1908)]

Thanks Your suggestion:

echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load

Has kept the cpu and temps down.

I will file the bug report.
Richard

On Sun, Jan 15, 2012 at 2:27 PM, Alejandro Nova <email address hidden>wrote:

> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #106 from Alejandro Nova <alejandronova gmail com> 2012-01-15
> 19:27:09 ---
> Please, file a bug against Kubuntu. Historically, there have been problems
> with
> 3 packages essential to Nepomuk/Akonadi: strigi; soprano and akonadi.
> Strangely, they are not covered by Kubuntu official PPAs, and if those
> packages
> are not updated to their latest releases, you WILL face problems.
> Currently:
>
> - Strigi is at 0.7.7.
> - Akonadi is at 1.6.90 (for KDE SC 4.8)
> - Soprano is at 2.7.4 (2.7.5 is imminent).
>
> --
> Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.
>

[Ernesto Manriquez (alejandronova)]

Already filed it:

https://bugs.launchpad.net/kubuntu-packaging/+bug/916903

Please, vote for that report!

[Edney Matias da Silva (edney)]

Hi there!

Right now running Fedora 16 with KDE 4.8 RC 2 from Red Hat KDE Repository. These are my packages version

strigi-libs-0.7.7-1.fc16.x86_64
kdegraphics-strigi-analyzer-4.7.97-1.fc16.x86_64
akonadi-1.6.90-1.fc16.x86_64
kdepimlibs-akonadi-4.7.97-1.fc16.x86_64
soprano-2.7.4-1.fc16.x86_64

There's no '&' on my tray, it's hidden and stating that the file indexer is idle. Akonadi Nepomuk Feeder states "system busy, indexing suspended". Virtuoso is at 99.3% of my 4 core CPU, since i activated it again. My machine was shut off once since then because of overheating.

[Edney Matias da Silva (edney)]

...since my last comment, virtuoso is at 99.4% of my CPU.

[Martin L. (heinrich20)]

Hi all,

Directly translated from German we would say: "I am loosing all my hopes for KDE"

Ok, sorry, not very quailfied! But anyway I wonder what really happens in this area. I am working with a 4 years old dual-core Notebook and this problem really hurts. But Edney's machine is much faster and the system brings the machine down anyway. Is this system design or the rollout strategy for virtuoso, nepomuk, akonadi and all the rest of this nicely designed system framework?

Regards, Martin
-------- Original-Nachricht --------
> Datum: Mon, 16 Jan 2012 16:29:12 +0000
> Von: Edney Matias <email address hidden>
> An: <email address hidden>
> Betreff: [Bug 246678] virtuoso: Usage of CPU is much too high

> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #110 from Edney Matias <edneymatias gmail com> 2012-01-16
> 16:29:10 ---
> ...since my last comment, virtuoso is at 99.4% of my CPU.
>
> --
> Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You reported the bug.

[rabauke (sven-burmeister)]

If you want to "use" nepomuk you have to remove its database from time to time. You will lose your tags etc. and you might have high CPU usage because it re-indexes all files – but at least that cpu usage ends at some point. And to be honest, nepomuk is of no use as desktop search anyway, compared to state-of-the-art desktop searches.

Sadly it happens very often that at some point or after an upgrade from KDE 4.x to 4.y virtuoso uses the cpu constantly, never comes to an end.

There are some hints at http://kdeatopensuse.wordpress.com/2011/11/09/debugging-nepomukvirtuosos-cpu-usage/ on how to debug.

But as I mentioned, the quickest way of getting around that buggy piece of software is to just remove its database from time to time. Does not sound nice and I wish it was different, but these issues have been around for years and are still not resolved.

[rabauke (sven-burmeister)]

Forgot the most important part:

As user issue rm -rf ~/.kde4/share/apps/nepomuk or use dolphin to remove that folder and get rid of all nepomuk data.

If you want to play nice you can disable nepomuk before doing so and re-enable it afterwards.

For some distros it might be ~/.kde/…

And BTW, please do not confuse nepomuk with akonadi which did not have years yet to stabilise and did in fact get quite stable within a short time and works very well in 4.8 – unlike nepomuk.

[Ernesto Manriquez (alejandronova)]

@S.Burmeister: from my experience, there have been multiple super-bugs with Nepomuk, and now all of them lie in the Akonadi Nepomuk Feeder. Make the test; open akonadiconsole, disable the Akonadi Nepomuk Feeder and in all cases your CPU use will decrease to zero in a matter of minutes (if you have some files left to index, your HDD will get used, but Nepomuk will scale up and down quickly).

If there is a way to permanently disable the Akonadi Nepomuk Feeder, that would bring an end to all of this.

[rabauke (sven-burmeister)]

Nope, sorry. The feeder did/does have its bugs but its not the sole/main cause of all nepomuk issues and virtuoso cpu hogging.

[Martin L. (heinrich20)]

Hi Folks,

the Akonadi framework is the main reason for me to quit KMail. Sinse
1999 I have been using KMail, most of the time I thought, this was one
of the best mail clients to get. Easy to use and working!

But with KDE 4 and "I don't now what" corresponding version of PIM/KMail
(4.?) the situation ended up getting catastrophically. Know I will move
to thunderbird and one step further after more than ten years of using
Linux/KDE might be to move to Windows 7. Not a fault of M$! I might
leave a platform that I thought it really was an alternative one.

Regards, Martin

PS: Last bug I found was in kdewallet, ending up to become very slow. I
wonder which application on my system really works.

m 16.01.2012 20:10, schrieb S. Burmeister:
> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #115 from S. Burmeister<sven burmeister gmx net> 2012-01-16 19:10:24 ---
> Nope, sorry. The feeder did/does have its bugs but its not the sole/main cause
> of all nepomuk issues and virtuoso cpu hogging.
>

[rabauke (sven-burmeister)]

Please do not spam this bug with unrelated issues but only info that helps to fix or work around this bug!

Please do only quote text you refer to.

Akonadi/kwallet/whatsoever bugs belong into another bug report and everything OT belongs on some personal blog, forum or mailinglist but not the bug tracker.

[Martin L. (heinrich20)]

Hi Sven,

would you kindly have a look at the history of this issue? I know what I
did! I myself started this bug report! How long ago? Was this problem
really solved? Maybe, maybe not! Noone looks at this situation from a
global perspective that tells us that KDE really runs out of acceptance
because if one bug is fixed two others are opened.

Maybe this is Spam and there is noone to be blamed because nowadays
there are really not enough developers to solve all the problems that
are the result of a ambitious project. A pity!

Martin

Am 16.01.2012 23:47, schrieb S. Burmeister:
> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #117 from S. Burmeister<sven burmeister gmx net> 2012-01-16 22:47:15 ---
> Please do not spam this bug with unrelated issues but only info that helps to
> fix or work around this bug!
>
> Please do only quote text you refer to.
>
> Akonadi/kwallet/whatsoever bugs belong into another bug report and everything
> OT belongs on some personal blog, forum or mailinglist but not the bug tracker.
>

[Richard Bacchetta (richb1908)]

Martin,
Yous is an editorial comment and does not further the solution of this problem. And I just editorialized again.

To get back on track, The suggestions made in an earlier post

echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load

Has "corrected' the problem for me in KDE 4.8 RC2. Corrected in quotes as it has prevented high cpu and overheating. Also the latest strigi, akonadi and soprano packages were not included in the upgrade to KDE that my distro provided and may just be a packaging issue. So it is still an open issue.

Let's give information to the developers that resolves the problem please.

[Edney Matias da Silva (edney)]

Hi there.

Writing to let you know that after many hours virtuoso now sit quite on my process list. This already happened before and an update to the system put it crazy again.

Thank you.

[Ernesto Manriquez (alejandronova)]

Basically this issue connects several bugs. Some of the most memorable ones are:

- Virtuoso being locked up with strigi, one core sleeping - killed with KDE 4.6.1.
- Various memory leaks, some with Virtuoso, some with nepomukstorage, and even some with dbus-daemon. The last one of those was fixed with Soprano 2.7.3.
- Virtuoso being locked up with akonadi-nepomuk-email-feeder, one core wasted. That one haunted KDE 4.7.x until ~4.7.4.
- Virtuoso being locked up with akonadi-nepomuk-feeder, one core wasted. That is the latest incarnation of this bug, and is what <email address hidden> was experiencing.

All of these real issues were amplified by more downstream issues.

- Strigi lacks a real release policy. During most of 2010 and part of 2011, Debian shipped Strigi 0.7.2, an extremely buggy and ancient release.
- Akonadi and Soprano are not updated with KDE, in KDE dependent distros.

So, you can't really fix this, unless:

- You require a Virtuoso + Soprano + Akonadi + Strigi + Shared Desktop Ontologies stack, and maintain it with a KDE 4.x release cycle.
- You assign a real maintainer to akonadi-nepomuk-feeder (that package is somewhat orphan, unlike Nepomuk).
- You REQUIRE distros shipping KDE to UPDATE their packages. That can be made easily through CMakeFiles (if you ship KDE 4.8, then you must have certain versions of Soprano, Akonadi, Strigi and shared-desktop-ontologies)
- You MAKE a "Update my Ontologies" app like what Bangarang has. That's a necessity.

Once you have all of this sorted out, you can really begin with bug triaging and reporting. Reporting bugs without all of these requirements unmet will be a waste of time for the reporter and for the developer.

About the Nepomuk database erasing: if I read correctly Sebastian Trueg's blogs and dev history, KDE SC 4.8 is going to be the last release that will require erasing everything to work well. The Akonadi Nepomuk Feeder is fresh code, made with something called DMS (AFAIK, replacing hand tuned SQL queries with queries generated automatically)

Please, be more constructive about how to fix this. I really hate Nepomuk bugs, I really want the thing to work, but we won't get anywhere if we just kill the thing. Remember that Nepomuk is the very thing (WinFS) that Microsoft FAILED to implement, and Sebastian Trueg has made tremendous strides to do what Microsoft failed to do, with no resources, and (as of late) with no money.

[Martin L. (heinrich20)]

Hi Alejandro,

thanks a lot for the information! Some of it I do understand, some I don't. I myself have the project lead in a commercial development project where not all the time I do understand the dependencies. The whole KDE 4.x - project, is there anyone who understands the dependencies and the release plan (if there is one)?

Talking about this issue: What shall we do, reopen it or keep it running in parallel?

Only someone who really understands how the different parts work together could track an issue like that. I myself do not have time and (no longer) patience to do so.

I am a KDE/Linux user who tried to help but stopped writing bug reports because there are too many. I have no idea where to start and where this might end!

Regards, Martin
-------- Original-Nachricht --------
> Datum: Tue, 17 Jan 2012 00:35:10 +0000
> Von: Alejandro Nova <email address hidden>
> An: <email address hidden>
> Betreff: [Bug 246678] virtuoso: Usage of CPU is much too high

> https://bugs.kde.org/show_bug.cgi?id=246678
>
>
>
>
>
> --- Comment #121 from Alejandro Nova <alejandronova gmail com> 2012-01-17
> 00:35:08 ---
> Basically this issue connects several bugs. Some of the most memorable
> ones
> are:
>
> - Virtuoso being locked up with strigi, one core sleeping - killed with
> KDE
> 4.6.1.
> - Various memory leaks, some with Virtuoso, some with nepomukstorage, and
> even
> some with dbus-daemon. The last one of those was fixed with Soprano 2.7.3.
> - Virtuoso being locked up with akonadi-nepomuk-email-feeder, one core
> wasted.
> That one haunted KDE 4.7.x until ~4.7.4.
> - Virtuoso being locked up with akonadi-nepomuk-feeder, one core wasted.
> That
> is the latest incarnation of this bug, and is what <email address hidden> was
> experiencing.
>
> All of these real issues were amplified by more downstream issues.
>
> - Strigi lacks a real release policy. During most of 2010 and part of
> 2011,
> Debian shipped Strigi 0.7.2, an extremely buggy and ancient release.
> - Akonadi and Soprano are not updated with KDE, in KDE dependent distros.
>
> So, you can't really fix this, unless:
>
> - You require a Virtuoso + Soprano + Akonadi + Strigi + Shared Desktop
> Ontologies stack, and maintain it with a KDE 4.x release cycle.
> - You assign a real maintainer to akonadi-nepomuk-feeder (that package is
> somewhat orphan, unlike Nepomuk).
> - You REQUIRE distros shipping KDE to UPDATE their packages. That can be
> made
> easily through CMakeFiles (if you ship KDE 4.8, then you must have certain
> versions of Soprano, Akonadi, Strigi and shared-desktop-ontologies)
> - You MAKE a "Update my Ontologies" app like what Bangarang has. That's a
> necessity.
>
> Once you have all of this sorted out, you can really begin with bug
> triaging
> and reporting. Reporting bugs without all of these requirements unmet will
> be a
> waste of time for the reporter and for the developer.
>
> About the Nepomuk database erasing: if I read correctly Sebastian Trueg's
> blogs
> and dev history, KDE SC 4.8 is going to be the last release that will
> require
> erasing everything to work well. The Akonadi Nepomuk Feeder is fresh code,
> made
> with something called DMS (AFAIK...

[rabauke (sven-burmeister)]

Martin, would you please stop quoting the full comment of somebody else!

Have a look at https://bugs.kde.org/show_bug.cgi?id=246678#c122 if you do not know what I'm talking about!

If virtuoso shows high CPU usage you have to debug it, find the queries that causes the cpu usage etc. A start would be http://kdeatopensuse.wordpress.com/2011/11/09/debugging-nepomukvirtuosos-cpu-usage/ as already pointed out.

[Martin L. (heinrich20)]

Sven,

I can see, what you pointed out. This is, what happened within the GMX client and I did not take care about. Sorry for that! But anyway, when we start to discuss about quoting like that or not, the way we communicate has reached a quality that not at all helps to solve the problem.

Two things where I wonder, how to improve the situation out of the persepctive (=role) of a user:

1. Debugging as described by your link is far too complicated to make that happen by a user.

2. How an where would I start? When would I start debugging? Which application to debug first. This is not cynism, I do think this to be a real problem for users.

/Off-Topic
Even though we are "only" users of KDE/Linux we do a real hard job. We stay on a platform (in case of KDE 4.x) that is shipped with Ubuntu but is really far away from being stable. If e.g. I try to write bug reports I even run into problems with that because I c a n n o t describe the bug itself appearing in a very complex system environment that is far to complecated. See 1. So by the time you wonder what you could do and find out: nothing!
/

Regards, Martin

[Edney Matias da Silva (edney)]

Hi there.

I tried to follow the instructions to debug the problem but couldn't find the isql-vt tool neither the package that contains it. I know it's a distribution problem, but it's just another one that sums up. Running Fedora 16 here.

Regards.

[Rdieter-math (rdieter-math)]

on fedora,
yum install virtuoso-opensource-utils

look in /usr/libexec/virtuoso/
for isql, isqlw, inifile

I'm not sure what isql-tw is supposed to refer to, but I'm guessing just the 'isql' tool (maybe suse renamed it or something).

[Edney Matias da Silva (edney)]

Hi there!

Trying to exec this line isql-vt -H localhost -P 1111 -U dba -P dba from the tutorial available here http://kdeatopensuse.wordpress.com/2011/11/09/debugging-nepomukvirtuosos-cpu-usage/ gives me that on Fedora 16:

[matias@padme ~]$ /usr/libexec/virtuoso/isql -H localhost -P 1114 -U dba -P dba

*** Error IM002: [iODBC][Driver Manager]Data source name not found and no default driver specified. Driver could not be loaded
at line 0 of Top-Level:

Any clues?

My virtuoso is crazy again after a restart on akonadi server from the control painel. :S

top command shows:

top - 14:38:57 up 2 days, 4:46, 3 users, load average: 1.67, 1.59, 1.10
Tasks: 198 total, 2 running, 196 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.4%us, 1.0%sy, 49.6%ni, 46.6%id, 0.0%wa, 0.3%hi, 0.1%si, 0.0%st
Mem: 3943656k total, 3776300k used, 167356k free, 52392k buffers
Swap: 4194300k total, 340968k used, 3853332k free, 990336k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1704 matias 39 19 1514m 513m 3888 S 198.5 13.3 317:24.94 virtuoso-t

Thank you

[Edney Matias da Silva (edney)]

two minutes after posting my last comment virtuoso shut my machine off! :S

[Rdieter-math (rdieter-math)]

comment #127 highlights a problem in fedora's packaging that I'm fixing now (we ship only the iodbc variants of these utilities, which apparently do not work in this context).

[Ernesto Manriquez (alejandronova)]

@Edney Matias: Run the command I posted. It won't do anything about your Virtuoso problems, but it will protect your computer from overheating.

[Kent57 (kent5700)]

Didn't have this problem in Oneiric until the latest update. Now virtuoso-t pegs cpu at 100%.

[wimo (pfuentesor)]

After update my kubuntu 11.10 t0 kde 4.8 i have the same problem.

[Scott Kitterman (kitterman)]

On Thursday, January 26, 2012 07:26:57 PM you wrote:
> After update my kubuntu 11.10 t0 kde 4.8 i have the same problem.

These issues should be filed as a new bug (similar symptoms don't necessarily
mean the same root cause).

[Edney Matias da Silva (edney)]

Hi!

I'm writing to let you know that virtuoso is fine now for a few days. I'm running Fedora 16 and here are my versions:

[matias@padme ~]$ rpm -qa | grep virtuoso
redland-virtuoso-1.0.14-1.fc16.x86_64
virtuoso-opensource-6.1.4-4.fc16.x86_64
virtuoso-opensource-utils-6.1.4-4.fc16.x86_64

[matias@padme ~]$ rpm -qa | grep akonadi
kdepimlibs-akonadi-4.8.0-1.fc16.x86_64
pykde4-akonadi-4.8.0-1.fc16.x86_64
akonadi-1.7.0-1.fc16.x86_64

[matias@padme ~]$ rpm -qa | grep kdelibs
kdelibs-devel-4.8.0-1.fc16.x86_64
kdelibs-common-4.8.0-1.fc16.x86_64
kdelibs-4.8.0-1.fc16.x86_64

[matias@padme ~]$ rpm -qa | grep kde-workspace
kde-workspace-4.8.0-7.fc16.x86_64
kde-workspace-libs-4.8.0-7.fc16.x86_64
kde-workspace-devel-4.8.0-7.fc16.x86_64

Anything else?

Thank you.

[Ernesto Manriquez (alejandronova)]

kde-runtime-4.8.0-3 ;)

Check for it.

[sanette (sanette-linux)]

kde 4.9.3, fresh install on a new laptop
(kubuntu 12.10)

* virtuoso-t is regularly eating 100% of one of my CPUs each time I do something with kmail... had to stop using kmail

* even without kmail, virtuoso-t uses 435 Mo of RAM, even though it seems to do nothing
(and in the systemsettings I have set to 50Mo the max RAM usage for nepomuk

[sanette (sanette-linux)]

ps: see attached graphics for the CPU usage (yellow curve)
this is a typical virtuoso-t burst when kmail is open (and I do nothing)

[bamyasi (iadzhubey)]

Kubuntu 12.10, KDE 4.9.4 fresh install on a 240GB Intel 520 SSD

I have installed a new 4TB data drive in my system today and started transferring files from my old (1TB) drive onto it, using rsync. Note: the new drive is mounted as /disk/; my old drive is mounted under /mnt/. None of the filesystems involved has any relation to my /home partition. All through the file transferring process, virtuosi-t process keeps eating 15-50% of CPU and iotop shows enormous amounts of disk I/O generated by 6-16 copies of virtuosi-t process run in parallel. The desktop is occasionally completely locked up for 1-2 seconds and unresponsive otherwise.

File indexing is configured to include only a handful of subdirectories under my /home directory. Still, virtuoso-t keeps trashing disk I/O subsystem during any file transfers, regardless of whether their source/target is withing the directories configured for indexing or not.

I consider this as a very serious and extremely annoying bug.

[maxis11 (ya-maxis11)]

virtuoso-t eats 50% of cpu in 4.11.2 and in dmesg

[39851.714273] virtuoso-t[12371]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f4b143aad50 error 7 in virtuoso-t[400000+b09000]
[39861.327093] virtuoso-t[12388]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f840fddbd50 error 7 in virtuoso-t[400000+b09000]
[39872.752638] virtuoso-t[12416]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f501f336d50 error 7 in virtuoso-t[400000+b09000]
[39882.341785] virtuoso-t[12452]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f538e8fdd50 error 7 in virtuoso-t[400000+b09000]
[39894.538014] virtuoso-t[12691]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fb4dc1c8d50 error 7 in virtuoso-t[400000+b09000]
[39904.505139] virtuoso-t[12728]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f42807ebd50 error 7 in virtuoso-t[400000+b09000]
[39914.684411] virtuoso-t[12773]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f9af605ad50 error 7 in virtuoso-t[400000+b09000]
[39924.794001] virtuoso-t[12800]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f0df1aaad50 error 7 in virtuoso-t[400000+b09000]
[39934.491591] virtuoso-t[12837]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fc61eabdd50 error 7 in virtuoso-t[400000+b09000]
[39944.314952] virtuoso-t[12853]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f724b51bd50 error 7 in virtuoso-t[400000+b09000]
[39953.429060] virtuoso-t[12881]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fecb1702d50 error 7 in virtuoso-t[400000+b09000]
[39962.463541] virtuoso-t[12910]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f21cf69dd50 error 7 in virtuoso-t[400000+b09000]
[39972.012866] virtuoso-t[12932]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f11b9f3bd50 error 7 in virtuoso-t[400000+b09000]
[39982.649058] virtuoso-t[12971]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fce9a230d50 error 7 in virtuoso-t[400000+b09000]
[39991.769957] virtuoso-t[12992]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fe2aa716d50 error 7 in virtuoso-t[400000+b09000]
[40002.143040] virtuoso-t[13026]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fc1e09b4d50 error 7 in virtuoso-t[400000+b09000]
[40012.900378] virtuoso-t[13057]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007fed11668d50 error 7 in virtuoso-t[400000+b09000]
[40022.103676] virtuoso-t[13091]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f7dbf637d50 error 7 in virtuoso-t[400000+b09000]
[40032.376944] virtuoso-t[13130]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f70b8d63d50 error 7 in virtuoso-t[400000+b09000]
[40042.807790] virtuoso-t[13146]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f9d3a718d50 error 7 in virtuoso-t[400000+b09000]
[40052.312900] virtuoso-t[13173]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f3d6dd58d50 error 7 in virtuoso-t[400000+b09000]
[40062.013635] virtuoso-t[13203]: segfault at ffffffffffffffff ip 0000000000939d75 sp 00007f857cea9d50 error 7 in virtuoso-t[400000+b09000]
[40073.095813] virtuoso-t[13241]: segfault at ffffffffffffffff ip 000...