Huge Memory consumption by extension and pipe not closed

[suresh]

Hi,

I was just playing around the different extension and I come to know a strange result.

As per my understanding about Inkscape extension is as below.

When any extesnion apply it creates temp. file in /tmp folder (in Linux) with extension effect and copy the temp. file using copy_doc function (in script.cpp_ file.

Now to just play around I just comment the line
copy_doc(doc->doc()->rroot, mydoc->rroot);
from script.cpp file, now this will not display extension result on the screen but open pipes(fifo) to run the script, now every time pipes open to run the script as child process, but this pipes are not closed hence after some time it will give error as
Can't Spawn!!! spawn returns: Failed to create pipe for communicating with child process (Too many open files
which is why max. file descriptor limit reached.

So this is one thing about pipes.

Now second thing is huge memory consumption, to generate the scenario please do the steps as below.
Open Inkscape
Go to extension->Render->Gear
Now Just click Apply

Now if you just applying the same plugin again and again inkscape memory will keep increasing, and If we delete the resulting effect by extension then also it is not getting free.

You can also apply different extension to generate the same memory issue. I did it on inkscape version 0.84.4 on Ubuntu 10.04 and Ubuntu 12.04 and results are same.

Now I have debug the things and I found that when extension create new file and it is copied using copy_doc into original doc at that time inkscape also create Root, Metadata, namedview element, which is I think inkscape default, that creates when inkscape start and they will remain in memory until inkscape exit.

So I think when new svg copied to the original at that time it start eating memory.

I was behind this from last 2 days and while searching for memory leak issue when applying extension, I found this.

I don't know how to contact developer, But If you give me more hints I can give you better testing feedback. In this bug If you require further details, please let me know. Also please let me know If there is any quick work around are possible to reduce memory usage.

[suresh (suresh-meetsme)]

Can somebody please conform the behaviour? Please let me know if you require further information.

[suresh (suresh-meetsme)]

Hi

Some more information I found by the time.

I found that copy-doc function eats the memory when actually final svg copied to the original document. So I think we need to find some other work around to copy the document into original document.

Also while searching for the GSList I found this http://gtk.10911.n7.nabble.com/really-freeing-the-memory-allocated-by-g-slist-td60323.html

I have explore the code and I found that inkscape use g_slist_free function to free the GSList, however the above link stats that is require to use g_slist_foreach to free the data element.

Please somebody conform this bug so that we can move ahead.

[jazzynico (jazzynico)]

> I found that inkscape use g_slist_free function to free the GSList, however the above link stats that is require to use g_slist_foreach to free the data element.

Note that as of glib-2.28, you can also use g_slist_free_full (https://developer.gnome.org/glib/2.36/glib-Singly-Linked-Lists.html#g-slist-free-full).

> Please somebody conform this bug so that we can move ahead.

With the following steps:
1. Open Inkscape trunk revision 12414 with an empty document.
2. Verify that the RAM used is relatively stable (5 minutes).
3. Apply Extensions>Render>Gear>Gear... (default settings).
4. Undo
5. Loop 50 times in 2 and 3.
6. Verify that the RAM used is relatively stable (5 minutes).

I noticed an increase of 10Mio in memory used by the inkscape.exe process (mainly in private data committed) between the end of step 2 and the end of step 6.

[suresh (suresh-meetsme)]

@Jazzy,

Thanks for the triggering the bug.

I will give 'g-slist-free-full' a try today and will check the result. Also I wanted to know Is there any way to remove the pipe which create everytime when we apply extension. If you keep applying the 'gear' extension, pipe limit will reached and extension will not work. So is there any work-around to fix the issue?

Thanks.

[suresh (suresh-meetsme)]

Ok, I found where is the problem.

When I use any extension inkscape increase memory more than any other operation. i.e. Move object, delete object.

Now what happen when we apply extension, inkscape create 2 new svg in /tmp folder, Now once the extension complete its task inkscape load the extension applied svg file. I debug the code and found in script.cpp does the things inside 'copy_doc' function.

In this function it delete the old root node and from the modified svg file it create a new node which has exntesion applied. Now at this stage inkscape eats more memory.

I guess at the time of new node deletion inkscape can't delete/release the memory hence memory keep increasing.

I have tried with many extension and observed the same behaviour. Please let me know is there any work around to fix the memory leak issue by modifying 'copy_doc' function or with any patches.

Let me know if you require further information, I have debug the script.cpp code and analyze the complete extension behaviour.

Thanks.

[suresh (suresh-meetsme)]

Also, come to know that in script.cpp file in copy_doc function,

oldroot->appendChild(child->duplicate(oldroot->document()));

this line add the new document child to the oldroot, at this point memory increase, If i comment the line, effect not take place, but memory stop increasing.

Now if someone please find the solution then I think Inkscape will be more reliable than before.

Inshort when we add/remove any child node to existing root element, at that time memory keep increasing,

Doing further R&D and going forward towards solution, :)

Thanks.

[Shlomi Fish (shlomif-gmail)]

I can reproduce this problem with inkscape-0.48.4-14.mga4 on Mageia Linux Cauldron (what will become Mageia Linux 4). Whenever, I press Apply in the gear render extension dialogue, then the memory use increases significantly (by about 0.6-0.8% of my 8 GB of RAM). Thanks for noticing and reporting it.

[su_v (suv-lp)]

Earlier reports about mem leaks when running extensions:
- Bug #887103 Memory leak when using JessyInk
- Bug #676393 "Measure path" doesn't deallocate memory
- Bug #644257 some way to flush ram

[suresh (suresh-meetsme)]

@suv,

Got it, is there any work around to fix the issue? in script.cpp inside 'copy_doc' function memory increase, when new node append in oldroot document.

[suresh (suresh-meetsme)]

Also just wanted to conform that is this a GC bug or Inkscape bug? because Inkscape uses GC to collect garbage. May be there could be a bug in GC.

Please give more information where to look for the solution?

[suresh (suresh-meetsme)]

updates..

is there any way to copy document directly into memory,

        doc->doc()->emitReconstructionStart();

        //copy_doc(doc->doc()->rroot, mydoc->rroot);
        doc = reinterpret_cast<Inkscape::UI::View::View *>(mydoc);
        doc->doc()->emitReconstructionFinish();

I tried above code, but it is not applying any effect, so just wanted to know is there any way to assign new document directly into memory?

[Jozef Vesely (jove)]

Hi,

just a note that I'am also affected.

Inkscape 0.48.3.1 r9886 on Ubuntu

I've a 400kb .svg document with about 20 2048x2048px (about 4MB .png each) images linked.
On load this uses about 500MB of memory (About memory).

Each Render->Gear invocation causes memory usage to rise by staggering 400MB.
At 2GB memory usage (half of my RAM) this leads to:

Can't Spawn!!! spawn returns: Failed to fork (Cannot allocate memory)

on console (message box would be nice). The fork call tries to commit copy (on write) of
all the inkscape's memory, but it is hardly used since it execve's the script interpreter immediately.
Using vfork call would be better. Anyway with:

echo 1 > /proc/sys/vm/overcommit_memory

I can continue calling the extension few more times until I actually run out of memory and inkscape gets killed.

I understand that some memory can be used for undo. However I think it is clear that
inkscape is leaking loaded bitmaps which is bad. I am willing to help with debugging ...

[Jozef Vesely (jove)]

I found some time to debug:

extension/implementation/script.cpp line: 722
        mydoc->release();
here mydoc is no longer needed (it has been merged into doc),
reference is dropped (debug log says to zero) but the destructor is not called (ever).
I don't really know how the garbage collector works but it seems it still sees some references to mydoc.

Setting mydoc = NULL; does not help.

Calling delete mydoc; solves the problem, however GC probably would not be too happy about it once it decides
delete it itself (again I've no idea how your GC works).

I also noticed that pump_events(); at line 731 leads to call of sp_document_idle_handler and eventually
to the update of mydoc, that unnecessarily loads the pixmaps (they are loaded for doc afterwards and mydoc is deleted anyway).
Nothing seems to break by commenting it out.

[suresh (suresh-meetsme)]

Hello Jozef,

Thanks for the patch. I tried the patch and its working fine. Now since we apply extension it generated a new svg file and copy_doc function copy the doc to the main svg file and changes reflects on the screen, Is there any way to do it differently? I.e. inside copy_doc it first delete the node and item and add new doc and item from the generated svg file. Because I have found whenever we there is call to appnedChild is made it increase memory.

Most of the memory leak from script.cpp file inside copy_doc function.

Thanks.

[jazzynico (jazzynico)]

Patch from comment #14 tested on Windows XP, Inkscape trunk revision 13445, with the Gears extension, live preview mode enabled.
I confirm that the memory consumption is very low compared to the unpatched version.

But unfortunately Inkscape crashes after 3 or 4 changes in the extension values and the following message shows in the console:

**
ERROR:src/xml/simple-document.cpp:49:virtual Inkscape::XML::Event* Inkscape::XML::SimpleDocument::commitUndoable(): assertion failed: (_in_transaction)

[jazzynico (jazzynico)]

The memory leak issue may be related to Bug #235371 "Memory is not cleared after file is closed".

I'm not a GC expert and my C++ skills are somewhat limited, but I feel there's something wrong in the way the garbage collector interacts with the SPDocument's destructor. But maybe it's just the destructor itself that fails.

[jazzynico (jazzynico)]

Replacing mydoc->release(); with delete mydoc; seems to work well (at least Inkscape doesn't crash like in comment #16).
How would such a change affect the way the garbage collector works? Is there a risk to crash elsewhere?

[Liam P. White (liampwhite)]

I believe that with our GC, calling Inkscape::GC::release will unref the document and it will be collected on the next cleanup pass.

[bernhard.altaner (bernhard-altaner)]

This bug also affects me. Both in version 0.48.5 (stable Ubuntu), 0.48.5 (stable arch), trunk version as well as 0.91.
I can reproduce it like stated in comment #3. However, it seems to be not dependent on the extension, which seems to be in accordance with the bug reports in comment #8.
I tried the fixes from comment #14 and #16, however they do not seem to work (I will try to compile it again with the exact same version stated in comment #16).
Unfortunately, this bug renders dealing with large files completely useless. In particular, it prevents productive use of inkscape + sozi or inkscape + jessyink as a presentation software.
I strongly recommend to increase the importance of that bug and assign it to a developer.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "script.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[bernhard.altaner (bernhard-altaner)]

Regarding comment #21:
I am not sure how to do it. Maybe someone more experienced in launchpad/programming/bucktracking can help me out.

Maybe I should file a new bugreport regarding the newest version. The problem I am experiencing is essentially the same reported here (second part) and at Bug #235371.
However, there are no crashes. It is basically what is described in the second part of the OP:

Applying any extension irreversibly fills memory, until _all_ open inkscape documents are closed.
The larger the file or the more rastered images are embedded/linked the more drastic the problem ist.

The easiest way to reproduce the bug ist:

[bernhard.altaner (bernhard-altaner)]

(pressed [Shift+Enter] or something and triggered a post and unfortunately do not know how to edit a comment)
The easiest way to reproduce the bug ist:
- Load an empty document
- Link a couple of bitmaps
- Regard memory consumption upon application of any extension (the easiest one being Render->Gear, as you can just apply it multiple times)

My question is now: the OP basically contains two things: The "Can't spawn" error message which I have never encountered and the memory consumption error. Should these issues be reported in different bugreports?

How can I be of further help?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jonathan Hofinger (jhofinger)]

Hi - thanks for reporting this bug, I've manually migrated it to Inkscape's new
bug tracker on GitLab, and closed it here.

Please feel free to file new bugs about the issues you're seeing at
http://inkscape.org/report.

Moved to: https://gitlab.com/inkscape/inbox/-/issues/6489
Closed by: https://gitlab.com/jhofinger