Lazy expressions appear to cause memory leaks

[geoffd]

In Zope 2.8+ there is a little known but very useful TALES feature: you
can have expressions be lazily evaluated. For example,

<span tal:define="foo lazy:python:someExpensiveMethod()" />

The "lazy:" prefix causes the python expression to not be evaluated until
foo is used somewhere.

There appears to be a fairly big problem with this setup. The lazy:
prefix wraps the expression in a LazyExpr which stores the expression and
its context. The LazyExpr in turn generates a LazyWrapper (that
holds similar information) which ends up getting held in the
TALInterpreter's global/local variable list.

The expression context for TAL expressions in a page template includes
things like the template itself, the context object (a fully wrapped
object), the request (chock full of complicated stuff), and so on. It
appears that storing the expression context in the lazy wrapper creates
some kind of circular reference or something similar that is preventing
garbage collection of these lazy wrappers. The result is a nasty memory
leak.

The problem appears to be fixable via some cleaning up in
PageTemplate.pt_render after the interpreter does its thing. The code
snippet below is probably overkill, but something like this appears to be
what is needed:

> context = getEngine().getContext(c)
> TALInterpreter(self._v_program, self._v_macros,
> context,
> output,
> tal=not source, strictinsert=0)()
>
> # clean up - try to eliminate circular references - this may be overkill
> context._compiler = None
> context.contexts = None
> context.repeat_vars = None
> from Products.PageTemplates.DeferExpr import LazyWrapper
> for k,v in context.global_vars.items():
> if isinstance(v, LazyWrapper):
> v._expr = None
> v._econtext = None
> v._result = None
> if context.vars:
> while len(context.vars):
> context.vars._pop()
> context.global_vars.clear()
> context.global_vars = None
> context.local_vars.clear()
> context.local_vars = None
> context.vars = None
> context._scope_stack = None

[Florent Guillaume (efge)]

Please keep in mind that a datastructure with circular references is not a "leak". It will get garbage collected. Python's gc doesn't only rely on reference counting these days.

[Hanno Schlichting (hannosch)]

Python has garbage collection that can deal with cycles these days.

[Colin Watson (cjwatson)]

The zope.tales project on Launchpad has been archived at the request of the Zope developers (see https://answers.launchpad.net/launchpad/+question/683589 and https://answers.launchpad.net/launchpad/+question/685285). If this bug is still relevant, please refile it at https://github.com/zopefoundation/zope.tales.