Epub Editor memory leak/improper memory handling with HTML renames

[Anthony Bolden]

I admit this is partially my own fault, but renaming lots of HTML files within an ebook cause memory handling issues.

Version: 5.39.1 but I updated for 5.41.0 and I can tell the same problem exists

OS: Windows 10

I'm doing large-scale reformatting to the Complete Malazan Book of the Fallen (10 books, ~10-12k pages, all in 1 epub file, it's more or less the 10 original ebook releases hastily slapped together by the publisher without consistent formatting). This document contains roughly 400 HTML files and 10 CSS documents before any fiddling on my end. I did a significant amount of changes with the internals of the HTML and CSS and had no problems. I then began renaming the HTML files for consistent formatting, and after about 30 or 40(?) the program hung and never unhung.

Task manager had "calibre worker process" (specifically the subprocess that contains the book's name) at ~8gb of memory usage, which maxed me out with the other things I had open. I closed some other applications to see what would happen and the memory usage shot up to 11gb and the program continued to hang.

After reopening the program, I tested renaming some of the HTML files again and they take a SIGNIFICANT amount of RAM per individual change. You can clear this out by saving, closing, and re-opening the editor, but that's not exactly optimal.

The amount of memory used is more or less linear - it opens at ~410mb +-10 with no changes, the first change increases the memory usage by 180 to ~590, the second change results in ~780, then ~960 and so on. When using the bulk renamer for chapters, it still increments the memory by the same amount.

My guess is this is historical states stored in RAM for the undo/checkpoints feature? Based on the memory usage changes, it seems like it's storing a pretty space-inefficient version of whatever is needed for the checkpoint (the source epub is ~13mb, the file browser adds the open files up to ~30mb) But there's no way for it to offload into a temp file or anything, and I don't think there's an option in the editor to clear, cap or offload stored checkpoints, so it just builds up forever and/or eventually leaks.

[Kovid Goyal (kovid)]

checkpoints are not stored in ram they are stored on disk, if there's a
leak it's elsewhere. This is not a high priority for me since its very
much a corner case, but someday when I have nothing better to do I will
look into it, until then patches welcome.

[Kovid Goyal (kovid)]

Fixed in branch master. The fix will be in the next release. calibre is usually released every alternate Friday.

 status fixreleased