Phatch eats-up nearly all the RAM and swap available

[Paillomams]

Ubuntu 9.10
Phatch 0.2.5 from PPA
1go RAM
2go swap

I am trying to rename about 200 photos taken with a 10MP camera, the photos are between 4 and 5Mo each. I am renaming them using the exif data (<Exif_Photo_DateTimeOriginal.year>-<##Exif_Photo_DateTimeOriginal.month>-<##Exif_Photo_DateTimeOriginal.day>). This action takes less than 1min with phatch 0.1.6 but it takes more than an hour with phatch 0.2.5. Phatch uses 80% of the RAM and swap available. See picture attached of the freed memory when Phatch is stopped.

[Stani (stani)]

Please always attach your actionlist and an example image, which we can copy 200 times. Otherwise we can't investigate it.

[Paillomams (aymeric-pallottini)]

Action list and photo attached.

[Stani (stani)]

Hi Matthieu, could you have a look at this with your profiling experience? To reproduce the bug copy the image 200 times. Mark the bug as confirmed if you can reproduce it.

[Nadia Alramli (nadiana)]

Today I was trying to use phatch to resize 58 images each one of them is between 3-5 MB. The machine is has 2 GB of RAM. Phatch memory consumption started to increase quickly with each image until it was killed by the system for exhausting the memory. By the time it was Killed 27 images were resized. The only output in the console was:Killed
Looks like a memory leak or the garbage collector is unable to free memory in time.

Nadia

[Nadia Alramli (nadiana)]

Actually after checking more closely phatch was only able to finish 13 images before being killed. At the time no other application was running. This happens consistently when I apply the library resize actionlist on the images.

[Stani (stani)]

This is pretty serious. I hope we can fix this soon. After a quick look, I did the following, but doubt it will solve the problem:
http://bazaar.launchpad.net/~stani/phatch/trunk/revision/1640

[Nadia Alramli (nadiana)]

Hi Stani,
You change has introduced a bug. This line:
785 image = photo.get_layer().image
Was removed without replacing it with something else. Now when I try to run the actionlist I get:

Traceback (most recent call last):
  File "/home/nadia/projects/phatch/phatch/lib/pyWx/droplet.py", line 44, in OnDropFiles
    self.method(fix_paths(filenames), x, y)
  File "/home/nadia/projects/phatch/phatch/pyWx/gui.py", line 933, in on_drop
    wx.GetApp().settings, paths=filenames, drop=True)
  File "/home/nadia/projects/phatch/phatch/core/api.py", line 732, in apply_actions_to_photos
    is_done, image_index, repeat)
  File "/home/nadia/projects/phatch/phatch/core/api.py", line 797, in apply_actions_to_photo
    photo.get_layer().image = image.copy()
NameError: global name 'image' is not defined

[Nadia Alramli (nadiana)]

I can still reproduce the issue after fixing the bug above. When I start phatch it will take 20.2 MB of memory. During the processing of the first and second images it will jump to 163 MB even though each one of them is 3.5 MB. Then the memory usage will increase quickly with each image to take all available free memory. I canceled the image processing when phatch was using about 255 MB, some memory was freed but phatch is still using 194 MB. I closed phatch without saving changes and interestingly it took about 10-20 seconds to close during that time the memory usage increased as well to about 220 MB.
This laptop only have 500 MB of memory not 2GB as I reported before. But this increase of memory makes it impossible to process big images even though the sizes combined should fit into memory.

[Nadia Alramli (nadiana)]

When used a tool to monitor the memory usage (heapy). I couldn't reproduce the issue. It turned out the tool was forcing the garbage collector run and collect all collectible objects. Then I noticed if I add gc.collect() to line 817 in phatch/core/api.py (after the del statement) the memory will be freed after each image as it should. The problem with this is that it slows down processing for small images, while speeding it for large images. I don't think this is a solution at all. Could be a temp fix. But obviously there is a bigger issue behind it. I tried to set the gc threshold to something small like 50, 2, 2 but that didn't make a difference. I also enabled debugging mode in gc and I still have no clear idea on what is going on. I don't think it is a circular reference issue otherwise gc.collect won't be able to collect the objects.
I'm still working on debugging the issue. This is just an update.

[Stani (stani)]

This bug was caused by circular references in lib/metadata.py and core/pil.py Thanks Nadia for your hard work on this!

http://bazaar.launchpad.net/~stani/phatch/trunk/revision/1674

It should be fixed now, but feel free to do more extensive testing.

[Stani (stani)]

@Paillomams, julez, Jean-François Fortin Tam and others affected by this bug

I have uploaded a test release to my PPA:
https://launchpad.net/~stani/+archive/ppa

To install it do exactly this (eg not dist-upgrade) for Karmic/Lucid:
sudo add-apt-repository ppa:stani
sudo apt-get update
sudo apt-get install phatch

Let me know if it solved your problems as soon as possible! This will be uploaded today to Lucid.

[Jeff Fortin Tam (kiddo)]

@stani, tested with your PPA, and it now seems to behave fine. Great job!

[Stani (stani)]

@Jean-François Fortin Tam

Great, I am going to release today!

[Paillomams (aymeric-pallottini)]

@stani

Tested the PPA version and it is fine for me as well. Well done. Thanks.