RAM memory used goes very high after advancing pages in a large PDF file!!!

[Sandor Ortegon]

Hello,

This is the first "bug report" in which I mention something about performance and RAM memory consumption. I must say that qpdfview runs quickly in my laptop, so no complains in this aspect. This time I tried to compare qpdfview with other viewers like evince, for example. Clearly qpdfview has a much better interface, works better, is more configurable, remember settings, etc.

But thinking in the browser plugin idea I mentioned earlier, not consumming too much RAM memory is important, so I compared both in this aspect:

I opened a file (in case you want to test with the same file, this is it: http://www.scottaaronson.com/thesis.pdf) that is large (approx. 250 pages) but proceeds from LaTeX compilation, so it is not "too demanding" in graphics and rendering.

In the settings file, I used antialiasing (otherwise, fonts looked really ugly in this file). In evince, the document looked more or less the same than in qpdfview. I didn't used "prefetch" and 32MB as cache (I think it is the default) -results with prefetch were actually very similar-

In my computer, this happened in terms of RAM (using my task manager to see the result):

- Just after opening them, evince took 29.1 MB, qpdfview took 35.5 MB.

- After advancing pages (using the mouse wheel) going to the end of the file, evince took 44 MB as maximum. And I when fast to the end, use the keys to going to the begining, advance and pages, return again to the beginning, etc.

- After advancing pages (using the mouse wheel) until the end of the file, qpdfview took 346 MB of RAM (it was increasing all the time when advancing)!!!! I waited one minute to see if the program was going to "liberate" some memory and it didn't happen. I used the keys to return to the beginning and it stayed between 220 MB and 345 MB

So, I definitely see a memory problem here. It seems to me that after advancing pages, the program (I don't know if this is a problem of qt, popper, or qpdfview) doesn't liberate memory from pages that already were seen, but keeps accumulating. In consequence, RAM consumption could be very high at the end of a really large document.

I think this is an important "bug" to test and check. As a comment, CPU comsumtion never went very high, and the program run fast (never lagged or something like that) during the whole process.

Thanks for your attention :)

[Adam Reichold (adamreichold)]

I agree that this problem should be taken seriously. Especially for a program that aims to be lightweight. (Just thinking about it, I'd say that only prefetching could - under very special usage patterns - consume memory without bounds.)

I tested the file and could not reproduce the problem. Using the default settings for cache and prefetch, qpdfview was never using more than 80 MB of RAM while displaying the linked file. So first things first, what Qt and Poppler versions do you use? (Personally, I use Qt 4.8.2 and Poppler 0.20.2.)

Another user reported a similar problem directly to me in the past where he found that some files make qpdfview and Okular consume a lot of memory which however do not pose a problem for Evince. Hence it would be nice if you could test this with another program like Okular that uses the Poppler-Qt frontend.

The main difference is that the Poppler-Qt relies on the Splash rendering backend whereas the Poppler-GLib frontend relies on the Cairo rendering backend. A "Qt-native" Arthur backend exists but is unfinished. So if Okular has the same memory problems, it is probably something inside the Splash rendering backend which is possibly already fixed in the current Poppler version. If not, we have to try to find the problem in qpdfview in the next two weeks. :-\

Best regards, Adam.

[Benjamin Eltzner (b-eltzner)]

I can roughly confirm Adam's findings: For me the memory consumption of qpdfview ranges between 80MB and 90MB, never exceeding the latter. That is not so bad, while it still loses to the 60-65MB of evince. My version of libpoppler and libpoppler-qt4-3 is 0.18.4-1ubuntu2.

[Adam Reichold (adamreichold)]

I also did some more testing for memory leaks using Valgrind which is not infallible but a good indication of incorrect memory usage patterns and while I did find a small memory leak in the presentation view (a few bytes at most) there seem to be no serious memory leaks in the qpdfview code base.

[Sandor Ortegon (sandortegon)]

Hello,

I can explain why Benjamin and Adam couldn't reproduce the problem and I could reproduce it. Apparently, it depends on the scale factor. My version of libpoppler is 0.18.4-1ubuntu2 (I guess only until Ubuntu Quantal we will have libpoppler 0.20). I use Qt 4.8.1 in Lubuntu 12.04.

Apparently, when programs are forced to use a scale (that minimizes the original size), for example in presentation mode or Fit Page, the problem appears. When I used original size, the results are similar to what Benjamin reported. In Okular, the problem was similar to Qpdfview when used "Fit Page", but actually in Okular the problem appeared everywhere with every size (i.e, unlike Qpdfview, there was no size in which the problem wasn't solved). So clearly Okular was the worst of the three viewers in that aspect. In all test, Evince took the least amount of RAM memory.

1) Fit Page Size:

These are screenshots when I opened the file with Okular (left), Evince (center), Qpdfview (right). I used the mouse wheel to scroll pages and took screenshots at the beginning, middle and end of the file.

Beginning: http://imageshack.us/f/21/201207171247341366x768s.png/

Middle: http://imageshack.us/f/842/201207171249401366x768s.png/

End: http://imageshack.us/f/232/201207171250511366x768s.png/

2) Original size:

These are screenshots when I opened the file with Evince (left), Qpdfview (right). I used them in continuous mode, use the PageDown button to advance pages.

Beginning: http://imageshack.us/f/812/201207171253301366x768s.png/

Middle: http://imageshack.us/f/411/201207171254481366x768s.png/

End: http://imageshack.us/f/821/201207171255231366x768s.png/

I hope by reading the amount of RAM in each case, the problem can be understood better and can be solved.

Thanks for your attention!!

[Benjamin Eltzner (b-eltzner)]

Hi Sandor, I can finally reproduce the bug: If and only if I set the page size such that a page can fully be displayed in the window (without the need for a vertical scroll bar), and I use any kind of single page view, i.e. not continuous mode, the memory consumption grows up to ~590MB with your test file. If the pages are large enough that at least a tiny bit of scrolling is necessary per page, memory consumption stays below 100MB, so the scrolling seems to do something to free memory. I will change the status to "confirmed", although it may still be a bug in the splash backend. Adam, can you reproduce the bug with this additional information?

[Adam Reichold (adamreichold)]

Hello,

Confirmed as well. And hopefully understood. I think the problem is that visibility testing is not performed if the scroll bar does not change, hence a lot of invisible pages get rendered and the rendering results are never discarded or cached. (It get's worse if you scroll faster. Try just holding down space. :-))

I will hopefully be able change that now but I think this will seriously reduce prefetching effectiveness which will hence have to go through the page cache again... Well I never liked prefetching anyway... :-\

Best regards, Adam.

[Adam Reichold (adamreichold)]

Ok, I committed the necessary changes to trunk revision 462. Please test and report your findings.

[Adam Reichold (adamreichold)]

This should be obvious by now but just to clarify: This does not seem to be a problem with Poppler or the Splash backend at all.

[Benjamin Eltzner (b-eltzner)]

The fix works for me. I'll test okular, and if I can confirm the problem there, I'll tell them.

[Benjamin Eltzner (b-eltzner)]

So, I tested okular and it seems that it only has a more generous default concerning cache size. Apparently nothing wrong there. (Would have been quite a surprise, anyway.)