garbage collection problem
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
IPython |
Fix Released
|
High
|
Fernando Perez |
Bug Description
If a variable is created within a script that was executed using the %run command, there is a reference to it created somewhere
that prevents it from being garbage collected after deleting of the variable.
I am not sure if this is really a bug or just a misunderstanding on my side. However, it would be good if it were possible to delete
a variable created inside a script without deleting the whole name space using %reset.
This example is from an email thread in ipython-user: http://
example:
create the following script named test_destructor.py and execute it using the ipython %run command:
kilian@chebang:~$ cat test_destructor.py
class C(object):
def __del__(self):
print 'deleting object...'
c = C()
kilian@chebang:~$ python test_destructor.py
deleting object...
now, let's try in ipython:
In [1]: run test_destructor.py
In [2]: del c
In [3]: import gc
In [4]: gc.collect()
Out[4]: 47
(object still not deleted)
In [5]: %reset
Once deleted, variables cannot be recovered. Proceed (y/[n])? y
deleting object...
Finally!
Related branches
- Brian Granger: Needs Fixing (overview)
-
Diff: 12304 lines (+6035/-2557)88 files modifiedIPython/__init__.py (+12/-9)
IPython/config/loader.py (+66/-25)
IPython/config/tests/test_loader.py (+9/-10)
IPython/core/application.py (+206/-81)
IPython/core/completer.py (+89/-73)
IPython/core/crashhandler.py (+41/-47)
IPython/core/debugger.py (+33/-0)
IPython/core/history.py (+47/-25)
IPython/core/hooks.py (+1/-2)
IPython/core/ipapp.py (+295/-184)
IPython/core/iplib.py (+240/-179)
IPython/core/magic.py (+96/-60)
IPython/core/prefilter.py (+72/-15)
IPython/core/prompts.py (+20/-8)
IPython/core/pylabtools.py (+145/-0)
IPython/core/quitter.py (+12/-7)
IPython/core/release.py (+1/-1)
IPython/core/tests/obj_del.py (+0/-35)
IPython/core/tests/simpleerr.py (+32/-0)
IPython/core/tests/tclass.py (+14/-10)
IPython/core/tests/test_completer.py (+35/-0)
IPython/core/tests/test_iplib.py (+216/-28)
IPython/core/tests/test_magic.py (+137/-190)
IPython/core/tests/test_prefilter.py (+34/-0)
IPython/core/tests/test_run.py (+174/-0)
IPython/core/ultratb.py (+60/-21)
IPython/core/usage.py (+34/-325)
IPython/extensions/parallelmagic.py (+4/-0)
IPython/extensions/pretty.py (+3/-58)
IPython/extensions/tests/test_pretty.py (+51/-6)
IPython/external/_numpy_testing_utils.py (+120/-0)
IPython/external/argparse.py (+89/-38)
IPython/external/decorators.py (+245/-51)
IPython/frontend/prefilterfrontend.py (+17/-37)
IPython/frontend/tests/test_prefilterfrontend.py (+9/-6)
IPython/gui/wx/ipshell_nonblocking.py (+15/-14)
IPython/kernel/__init__.py (+1/-1)
IPython/kernel/clusterdir.py (+53/-65)
IPython/kernel/core/interpreter.py (+3/-4)
IPython/kernel/core/tests/test_redirectors.py (+6/-6)
IPython/kernel/engineservice.py (+3/-3)
IPython/kernel/error.py (+2/-2)
IPython/kernel/ipclusterapp.py (+8/-19)
IPython/kernel/ipcontrollerapp.py (+14/-34)
IPython/kernel/ipengineapp.py (+8/-27)
IPython/kernel/tests/test_multienginefc.py (+9/-1)
IPython/kernel/tests/test_taskfc.py (+9/-1)
IPython/lib/inputhook.py (+49/-3)
IPython/quarantine/InterpreterPasteInput.py (+0/-124)
IPython/scripts/iptest (+19/-3)
IPython/testing/__init__.py (+18/-0)
IPython/testing/_doctest26.py (+110/-0)
IPython/testing/_paramtestpy2.py (+89/-0)
IPython/testing/_paramtestpy3.py (+62/-0)
IPython/testing/decorators.py (+84/-14)
IPython/testing/decorators_trial.py (+0/-132)
IPython/testing/globalipapp.py (+168/-0)
IPython/testing/iptest.py (+349/-185)
IPython/testing/ipunittest.py (+189/-0)
IPython/testing/nosepatch.py (+53/-0)
IPython/testing/parametric.py (+3/-0)
IPython/testing/plugin/ipdoctest.py (+27/-180)
IPython/testing/plugin/test_ipdoctest.py (+0/-19)
IPython/testing/tests/test_decorators.py (+71/-13)
IPython/testing/tests/test_decorators_trial.py (+9/-4)
IPython/testing/tests/test_ipunittest.py (+122/-0)
IPython/testing/tests/test_tools.py (+31/-9)
IPython/testing/tools.py (+255/-7)
IPython/utils/baseutils.py (+51/-0)
IPython/utils/genutils.py (+129/-41)
IPython/utils/platutils.py (+1/-1)
IPython/utils/platutils_posix.py (+14/-1)
IPython/utils/tests/test_genutils.py (+71/-51)
IPython/utils/tests/test_imports.py (+0/-1)
IPython/utils/tests/test_platutils.py (+19/-6)
MANIFEST.in (+1/-0)
README.txt (+22/-4)
docs/emacs/ipython.el (+1/-1)
docs/source/development/coding_guide.txt (+34/-0)
docs/source/development/index.txt (+1/-0)
docs/source/development/magic_blueprint.txt (+103/-0)
docs/source/development/testing.txt (+366/-32)
docs/sphinxext/ipython_directive.py (+641/-0)
iptest.py (+26/-0)
ipython.py (+6/-2)
setup.py (+44/-3)
setupegg.py (+1/-7)
tools/build_release (+6/-6)
- Brian Granger: Approve
- Diff: None lines
Changed in ipython: | |
status: | Fix Committed → Fix Released |
I discovered the same issue. I think it is a serious memory leak.
For example, if the script you are running with "run" creates large arrays or images. In a typical situation, I am running the same script over and over as I improve and test it. Because of this bug, each script run leaks whatever memory is attached to the objects created by the script. This can add up to a large amount of memory in the sort of lengthy interactive session that IPython is so good at encouraging and supporting.
I tracked it down to IPython/ Magic.py: 1570 (in Ipython release 0.8.4) in the magic_run function. Here is an extract:
if opts.has_key('i'):
__ name__save = self.shell. user_ns[ '__name_ _']
prog_ ns['__name_ _'] = '__main__'
main_ mod = FakeModule(prog_ns)
name = os.path. splitext( os.path. basename( filename) )[0]
name = '__main__'
main_ mod = FakeModule()
prog_ ns['__name_ _'] = name
self. shell._ user_main_ modules. append( main_mod)
# Run in user's interactive namespace
prog_ns = self.shell.user_ns
else:
# Run in a fresh, empty namespace
if opts.has_key('n'):
else:
prog_ns = main_mod.__dict__
# The shell MUST hold a reference to main_mod so after %run exits,
# the python deletion mechanism doesn't zero it out (leaving
# dangling references)
The last line appends the namespace of the script to a list. The list is only cleaned up if the user runs the %reset command. But that is overkill -- generally I want to keep the prompt's namespace but I am happy to overwrite the namespace when I re-run a script.
I found that commenting out the last line fixes the problem. I disagree with the comment above the last line.