Extreme memory usage by kio_thumbnail

[what_if]

Binary package hint: dolphin

lsb_release -rd
Description: Ubuntu 8.04.1
Release: 8.04
32-bit

apt-cache policy dolphin
dolphin:
  Installed: 0.9.2-0ubuntu6.1
  Candidate: 0.9.2-0ubuntu6.1

Simple explanation:
When managing large image files in dolphin using the "previews" mode many thumbnail-generating sub-processes are spawned causing excessive memory usage and swap usage (over 2 GB). The computer system essentially "hangs" from the memory usage.

To duplicate the problem:
Place a handful of large image files (I used 10 tiff files of 45.8 megapixels each) in a folder.
Open Dolphin and navigate to the folder containing the image files.
Ensure the "Previews" view mode is selected.
Ensure the "Right Sidebar" is enabled.
Simply hold the cursor over the icon of an image file until it tries to load a preview in the "Right Sidebar"
--A sub-process of kio_thumbnail will be spawned every time the previous step is repeated with a new file to the tune of 202MB each in my case. (2.02GB memory usage in this example from kio_thumbnail processes).

The system will begin to slow as the physical memory is exhausted and all system resources are required to facilitate the swapping.

[Jonathan Thomas (echidnaman)]

Is this an issue with Intrepid or with dolphin-kde4 in Hardy?

[Jonathan Thomas (echidnaman)]

e are closing this bug report because it lacks the information we need to investigate the problem, as described in the previous comments. Please reopen it if you can give us the missing information, and don't hesitate to submit bug reports in the future. To reopen the bug report you can click on the current status, under the Status column, and change the Status back to "New". Thanks again!

[Thomas Weissel (xapient)]

Version: (using KDE 4.2.0)
OS: Linux
Installed from: Ubuntu Packages

for example in gwenview (the thumbnailbar) or when copying a big image file in konqueror that already exists .. (when the window to compare the two versions is started) kiothumnailer is startet and it takes one minute and the whole system is unusable.. this process should NEVER get (or take) that much resources.. even if i had dolphin set up to show 300MB thumnails.. (i have not)

[Loacoon (loacoon)]

I'm having the same problem with latest SVN. kio_thumbnail uses a lot of memory and CPU, and takes a lot of time to free the used memory.

[dwchapin (daren-chapin)]

This is most definitely still a problem, even in a new Jaunty install. It is particularly pronounced with directories of multipage tiff files. kio_thumbnail processes get launched until all available memory is exhausted, swapping out the entire system until nothing works. Before the system freezes up completely I counted 6 instances of kio_thumbnail at ~200MB each on a 1G memory box.

[inspirra (inspirra)]

$ LANG=C date
Mon Jul 6 09:13:30 MSD 2009

$ ps xo pid,%cpu,%mem,rss,comm,start_time | grep -E " kio_thumbnail| dolphin"
24297 0.0 0.0 872 kio_thumbnail 06:11
31624 3.5 0.8 17056 dolphin 08:58
31772 13.5 10.0 209292 kio_thumbnail 09:00
31840 16.4 2.1 43632 kio_thumbnail 09:01
31843 13.4 13.4 278508 kio_thumbnail 09:01
31862 14.4 11.3 236172 kio_thumbnail 09:01
31881 6.2 15.6 324528 kio_thumbnail 09:02
32059 2.3 12.2 254344 kio_thumbnail 09:02
32067 1.7 9.3 194044 kio_thumbnail 09:02
32158 0.1 0.2 5264 kio_thumbnail 09:03

$ kde4-config --version
Qt: 4.5.2
KDE: 4.2.95 (KDE 4.2.95 (KDE 4.3 RC1))
kde4-config: 1.0

$ uname -rms
Linux 2.6.30-gentoo-r1 i686

[scientica (scientica-gmail)]

and here's my 2 {$COIN_DENOMINATOR}
Summary:
 kio_thumbnails really missbehaves by stealing memory for me too (+1.7G DATA as reported by top this morning). Can't say about the CPU usage as it fills RAM and swap causing massive swapping => skyrocketing my load avg to +30.0, top says mostly (+95%) io wait untill I manage to kill the kio_thumnail process (or processes)), this probably due to thrashing caused by the exess memory theft. (I'm puzzled as to why it's not OOM killed!)
Versions:
 OS: Gentoo
 kde-base/kdebase-kioslaves-4.3.0
 (build with USE=debug so it should be possible to get some
  useful debuggging info, not sure how though)

 $ kde4-config --version
     Qt: 4.5.1
     KDE: 4.3.00 (KDE 4.3.0)
     kde4-config: 1.0

Reproduce:
 It usually starts by hovering big movie files or folders with big movies files. Though it shouldn't thumbnail them, max previev file size is set to 5 MB (the movies as much bigger than that), the thumbnails button isn't checked. Further more only "Grafik" (I'm 'guessing' the non localized string is "graphics") and Jpeg is selected. And "Använd miniatyribilder inbäddade i filer" (~= "use embedded thumbnails"/"use thumbnails embedded in files").
 * Clearly kio_thumbnails/dolphin isn't respecting those options. - maybe this is a konqueror/dolphin bug? (otho, imo thumbnailing a file shouldn't consume more than _tops_ 10M RAM, or perhaps I'm living in a fantasy world?)

Work-around:
 I've temporarily renamed kio_thumbnail.so to kio_thumnail.so.disabled, since it's a hazard to the system's stabillity right now.

Wish:
 A kill-switch for kio_thumbnail, i.e. a "Never generate thumbnails" option, if so only accessible via manually editing ~/.kde4/share/config/kio_thumbnailrc. Or a hard memory limit option there, say 100M (which imo is too much, but at least a limit that should prevent grand-theft-ram).
 Or better yet, a real fix that makes kio_thumbnail use a reasonable amount or RAM when generating thumbs.

[E-kde (e-kde)]

I can confirm this. I turned on previews/thumbnails for most file types (as it's a very cool feature) and then went about normal usage with Konqueror. After not long I found several kio_thumbnail processes chewing through resources like it's going out of style, the most flagrant of which goes something like this in top:

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11053 neil 20 0 4396m 1.2g 44m S 25 59.0 6:52.47 kio_thumbnail

...and my 2G RAM machine is suddenly 2.7G into swap (and rising fast).

Since among other things, I opened up my home directory, I checked to see if thumbnails were being generated recursively (which lsof suggested they were not), which is good. However, It's absolutely *INSANE* that kio_thumbnail is now approaching 5G of memory used, it's not like it's thumbnailing files anywhere close to that big!

Since I wrote the first bit, my swap usage has grown to 3.2G and I'm killing kio_thumbnail manually... Much better, close to 4G of RAM + swap freed.

[E-kde (e-kde)]

Oh, and I'm using KDE 4.3.

[Mark (mark-ale8)]

*** This bug has been confirmed by popular vote. ***

[Jakub-januszkiewicz (jakub-januszkiewicz)]

I was also hit by this bug, twice.

In both cases some kio_thumbnail processes suddenly started hogging the memory like crazy, causing massive swap activity.

The first time it happened I couldn't do much at all - the system got so unresponsive that I only managed to run 'top' in konsole which, after refreshing a few times, froze completely (even ATL+CTRL+F1 didn't work). I left the computer on for the night thinking it would settle down, but after a few hours the disk was still swapping like crazy and I couldn't do anything. A hard reboot was needed.

The second time happened today and I was quick enough to see some more details before everything froze (I didn't manage to kill the offending processes though - mem usage skyrocketed and froze the machine in literally several seconds!). There were 3 'kdeinit4: kio_thumbnail' processes, one of which consumed 3900MB VIRT and 2 others consuming over 1500MB VIRT each. I know I should have also looked at RES and/or SHR, but well... I didn't, sorry :/

The only thumbnail-related activity within the last few hours before today's freeze I can recall is copying several large video files from DVDs to hard disk. Although the directory to which I copied the files has thumbnails turned off, Dolphin still shows the thumbnail in tooltip and in the information panel on the right. This might be related, but I have closed all Dolphin windows at least 2 hours before the bug occured, so it might as well be not related at all. I also have a large (22GB) photo collection, but I haven't touched it today at all.

I have 4GB RAM + 4GB swap.
I run KDE 4.3.1 on Gentoo.
I have all thumbnailing plugins enabled (also video via MPlayerThumbs) with max file size set to 5MB.
I use Dolphin as the file manager.
If you need any additional info, please let me know.
If I encounter this bug again, I'll try to provide more useful info.

[E-kde (e-kde)]

So, this morning, I had <100M of swap used. I left and came back in the afternoon to find this:

top - 18:33:11 up 23 days, 19:31, 66 users, load average: 0.40, 0.58, 0.45
Tasks: 185 total, 1 running, 184 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.0%us, 2.7%sy, 0.0%ni, 93.9%id, 0.3%wa, 0.2%hi, 0.0%si, 0.0%st
Mem: 2057212k total, 2040096k used, 17116k free, 8264k buffers
Swap: 7807224k total, 4206896k used, 3600328k free, 221012k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25695 neil 20 0 5037m 1.2g 5824 S 4 61.5 48:39.43 kio_thumbnail
...

Note the swap usage and the memory usage for kio_thumbnail. Though I do have a Konqueror file manager open with a few tabs, I haven't interacted with it whatsoever since yesterday, and I don't use Dolphin.

Killing kio_thumbnail returns a few resources:

top - 18:38:49 up 23 days, 19:37, 66 users, load average: 0.23, 0.38, 0.40
Tasks: 182 total, 1 running, 181 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.8%us, 2.0%sy, 0.0%ni, 94.0%id, 0.5%wa, 0.3%hi, 0.3%si, 0.0%st
Mem: 2057212k total, 765492k used, 1291720k free, 8852k buffers
Swap: 7807224k total, 1013696k used, 6793528k free, 242908k cached

...meaning it really was consuming nearly 5G of RAM, and completely unprovoked at that.

[Dan Reidy (dubkat)]

confirmed over here...
dubkat@stargazer ~ $ kde4-config --version
Qt: 4.5.3
KDE: 4.3.2 (KDE 4.3.2)
kde4-config: 1.0

i frequently have to keep a terminal window open running htop so that i can kill off any runaway kio_thumbnail processes. failure to do so causes the system to freeze, and the disk to thrash. I have 4 gigs of memory, and about 8 gigs of swap space. All easily eaten up by kio_thumbnail.

[Peri Hankey (mpah)]

In Kubuntu 9.10:

Something similar applies to PDF documents. If you have a directory containing several large PDF documents, and you have preview enabled, konqueror (or dolphin) effectively freezes. Even if you turn preview off, there seems to be no way of disabling a preview component for the right-click properties dialog for PDF files. The same applies with tootips - you get a pdf preview, and that can take a long time. So you have to run without preview and without tooltips, and even then you must take care not to attempt to read the properties of any large PDF.

This is an absurd state of affairs. Except for large single image PDF files, the preview image should be the first page, which is unlikely to be very large. It should be possible to get the first page without reading the whole document, which is what seems to be happening.

For certain kinds of usage, where you habitually deal with large PDF documents from other sources, this kind of thing is a road block for karmic + KDE4.

[Peri Hankey (mpah)]

In fact System Monitor shows konqueror/dolphin eating a large percentage of CPU, so perhaps it's not kio_thumbnail. And at some point konqueror spawns pdftotext. Is there any way to stop this behaviour?

[Federico Cuello (fedux)]

It is still happening,

$ kde4-config --version
Qt: 4.6.0
KDE: 4.3.4 (KDE 4.3.4)
kde4-config: 1.0

[ShinobiTeno (lct-mail)]

This is not related to KUbuntu, but its KDE 4.3.3 problem. If there will be solution, please upstream it!

Im currently running x64 gentoo system, based on Calculate linux 10, which is actually prebuilt gentoo.
Machine is dual-core E5300 with 4Gb of ram.

Im experiencing SAME issues with kio_thumbnail, slowly eating memory up, even if im NOT browsing (or browsed) any pictures and did not open any miniatures.
The effect is so insane, that this machine left for 6 hours for a purpose of backing video up(scratched dvd) has experienced kernel panic, because it run out of memory! (4Gb ram + 4Gb swap).

See two small examples in attatchment.

This is a very serious bug, it prevents normal work even if system is running stock and untouched.

[ShinobiTeno (lct-mail)]

This is a picture of htop, displaying kio_thumbnail eating over 4Gb of ram.

[R-clark-01 (r-clark-01)]

Still happening on:

kde4-config --version
Qt: 4.6.2
KDE: 4.3.5 (KDE 4.3.5)
kde4-config: 1.0

[Alexander (roth-a)]

I've the same problem. It eats so much ram thart it blocks my compuer completly.
It seams to occur always with the same pictures. Maybe there is an error in the pictures.

But kio_thumbnail shouldn't have such grave problems with broken images...

gwenview and digikam have also the same problem with the images...

[gene (eugenios)]

Tried to give a non-mono application a try and got upset with it. Not only digikam is strange that it does not find images when an album is specified, it eats up all resources and completely freezes my system. I do have 4 gb of photos, digikam uses all resources to "add" a little fraction of it - a folder worth of 200 mb. If a new user sees f-spot or this even worse digikam, he/she might get a false impression about Linux. I myself use ImageMagick, gthumb, and eog...

[gene (eugenios)]

Well, amarok behaves in a similar way in the cover art manipulation. In this case, how does it manage to hang all the resources with a few tiny pictures? With this KDE is a real mess... However, maybe there is some specifics about our, say video cards? Mine is known to be quite hostile to linux.
lspci -vvv

..........
VGA compatible controller: ATI Technologies Inc RC410 [Radeon Xpress 200M]
...........

Does any of this bug sufferers have a similar chip?
Thanks

[Pavel Malyshev (afunix)]

Lucid-RC still has this bug. Looks like kio_thumbnail does not free memory.
Maybe it creates some cache in memory and does not clean it? If it's true, where can we tune memory limit for cache?

[Vo-zaeb (vo-zaeb)]

http://img72.imageshack.us/img72/3733/42221637.png
http://img99.imageshack.us/img99/3641/73259312.png

[Vo-zaeb (vo-zaeb)]

I can't understand, how "this" could to get in the KDE trunk and stay there for so long time? Moreover, it is still there, without considering the fact, that the question already is about the loss of user data! I just have no words...

http://bugs.kde.org/buglist.cgi?quicksearch=kio_thumbnail

[FiNeX (finex)]

Probably the reason is because developers doesn't use thumbnails or doesn't have big video files.

Anyway, I'm using KDE 4.4.x (now I've 4.4.4) and it is a couple of months that I don't have that problem, maybe it has been partially fixed in the latest release... I hope :-)

[Loacoon (loacoon)]

I don't know about KDE 4.4.4 but I'm sure it still happens in trunk.

[Vo-zaeb (vo-zaeb)]

This isn't the only one problem with kio_thumbnail. For example, you can try to start downloading few video files in ktorrent and open the target folder with preview. Or try to browse a remote folder that contains video files, etc. Seems that this component wasn't tested completely enough.

[kovyrlo (kovyrlo)]

i have same problem

$ kde4-config --version
Qt: 4.6.2
KDE: 4.4.4 (KDE 4.4.4)
kde4-config: 1.0

[Mahasler (mahasler)]

I've completely removed mplayerthumbs from my systems and installed kffmpegthumbnailer instead. No more memory problems for me. Plus, kffmpegthumbnailer is really snappy, even on a netbook.

[Vo-zaeb (vo-zaeb)]

(In reply to comment #18)
> I've completely removed mplayerthumbs from my systems and installed
> kffmpegthumbnailer instead. No more memory problems for me. Plus,
> kffmpegthumbnailer is really snappy, even on a netbook.

+1, but it has its own problems, and from time to time generates poor quality thumbnails. Are you using bittorrent? :)

[Mahasler (mahasler)]

Can't say I do. But at least video clips downloaded "normally" don't cause any problems, the thumbnails are just regenerated regularly during the download.

[Karl-r-ernst (karl-r-ernst)]

I have this problem too with kde 4.4.5, even thought I have disabled video previews and limited previews to files smaller than 5 MB.

It seems to be related to the folder preview, as if I disable folder preview, the problem goes away.

[melendro (melendro)]

To disable video thumbnails run the command "mplayerthumbsconfig" and add all video extensions (avi, mpg, mkv, mp4, etc.) to the "Blacklisted file extensions".

This solved the problem for me.

[Ha-Duong Nguyen (cmpitg)]

Is there any fix yet? The problem persists in KDE 4.4.5. I'm using Gentoo x86_64.

[ShinobiTeno (lct-mail)]

Hello Yang!

Please unmask and update to KDE 4.5.3 or later!
I have tested the case again now in KDE 4.5.3 in Gentoo x86_64 under similar conditions for several hours, this bug was ironed out(resolved).

"Kdeinit4: Kiothumbnail" is started once the thumbnail view is requested via Dolphin "thumbnail" button consuming 20-40 MiB of RAM. Thumbnails are displayed rather fast. Then, it idles for around a minute or two and is killed.

Hope Kubuntu guys can retest this and confirm.

[George (geoaraujo)]

This is still happening on Kubuntu 11.04 with KDE 4.6.4.
Opening a folder with lots of tiff files will make kio_thumbnail to use a lot of cpu and the process doesn't come to an end unless it's stopped manually.

[Nate-b (nate-b)]

It's been a few years... does anybody still see this issue with KDE Frameworks 5.44 or greater?

[Nate-b (nate-b)]

Haven't heard anything from the reporter or anyone else who experienced this at some point in the past. Considering it fixed in KDE Frameworks 5.

[Nate Graham (pointedstick)]

Upstream bug (https://bugs.kde.org/show_bug.cgi?id=185881) has been closed as fixed as of KDE Frameworks 5.