Huge memory leak in calibre-server's heap

[Alan Aversa]

When I first start calibre-server, it uses <100MB of memory in total.

In my particular case, calibre-server was started over 6 hours ago and it has only used about 2 minutes of CPU time. However, the total memory usage has skyrocketed to over 740 MB. I have experienced it get as high as 1.3 GB.

Here is the memory info:

Private 758660 KB (= 3112 KB clean + 755548 KB dirty) ←----- Quite a lot dirty!
Shared 4920 KB (= 4920 KB clean + 0 KB dirty)
Rss 763580 KB (= Private + Shared)
Pss 760022 KB (= Private + Shared/Number of Processes)
Swap 0 KB

Private
724308 KB [heap] ←----- Why is this heap usage so high?
7692 KB [stack:28150]
5440 KB /dev/zero (deleted)
2048 KB [stack:28179]
2048 KB [stack:28172]

Shared
1448 KB /lib64/libc-2.23.so
868 KB /opt/calibre/lib/libpython2.7.so.1.0
860 KB /opt/calibre/lib/python2.7/site-packages/apsw.so
328 KB /opt/calibre/lib/libjpeg.so.8
236 KB /opt/calibre/lib/libpng16.so.16

[Alan Aversa (aversa)]

I am using: calibre-server (calibre 2.56)
on Slackware-current, Linux 4.4.9

[Kovid Goyal (kovid)]

I cannot reproduce this. Starting calibre-server with a 40K book
library, connecting to it via 7 different clients and browsing around
causes total memory consumption to not rise beyond 550MB -- note that
total memory consumption is proportional to library size, since the
server keeps the db in memory. For a library of a few thousand books
typical mem consumption is ~100MB.

I'd guess this is something local to your calibre installation. Look for
third party plugins you have installed or library conflicts.

 status invalid

[Alan Aversa (aversa)]

I have about 2k books.

I disabled all extensions and restarted. I am still having the same issue. It seems the total memory increase is directly proportional to calibre-server's CPU time. Perhaps this is just the nature of the web-server? My Apache webserver uses quite a lot of real memory, too.

[Kovid Goyal (kovid)]

No, it most definitely isn't. But, the entire web server has been
re-written for calibre 3.0 which is around the corner. So I dont want to
spend too much time on investigating an issue in a soon to be obsoleted
codebase. I suggest waiting for calibre 3.0 and if you still experience
this issue, upadte this ticket.

[Alan Aversa (aversa)]

The beta version does the same thing; the amount of real, physical memory used keeps creeping up. Also, I noticed that calibre-server spawns tons of processes when it first loads, all but one of which pretty much immediately quit. Is that normal? I see several calibre-server processes in top initially.

[Kovid Goyal (kovid)]

Umm, what beta version, there is no beta with the new server.