Memory leak in Python Expression interface

[Garth Wells]

The below code leads to a memory leak (I see a leak over about 2MB over 90s). The leak is related to the Expression. If I comment out the call to

    eval(Array<double>&, const Array<double>&);

from

  dolfin::GenericFunction::evaluate(double*, const double*, const ufc::cell) const

I don't detect a leak.

------

from dolfin import *
parameters["form_compiler"]["cpp_optimize"] = True
parameters["form_compiler"]["optimize"] = True

class MyExpression(Expression):
    def eval(self, values, x):
        if x[0] < DOLFIN_EPS:
            values[0] = 1.0

mesh = UnitSquare(32, 32)
for t in xrange(20000):
    for level in xrange(100):
        V = FunctionSpace(mesh, "CG", 1)
        v = TestFunction(V)
        u = Function(V)
        u0 = Function(V)
        f = MyExpression(degree=1)
        L = v*(u-u0)*dx + v*f*ds
        b = assemble(L)
    print "----"

[Garth Wells (garth-wells)]

Here's another solver that leaks. It doesn't use Expressions.

from dolfin import *

class MyProblem(NonlinearProblem):
    def __init__(self, a, L):
        NonlinearProblem.__init__(self)
        self.L = L
        self.a = a
    def F(self, b, x):
        assemble(self.L, tensor=b)
    def J(self, A, x):
        assemble(self.a, tensor=A)

mesh = UnitSquare(8, 8, "crossed")
BDM = FunctionSpace(mesh, "Brezzi-Douglas-Marini", 1)
DG = FunctionSpace(mesh, "Discontinuous Lagrange", 0)
mixed_space = MixedFunctionSpace([BDM, DG, DG])

V = TestFunction(mixed_space)
dU = TrialFunction(mixed_space)
U = Function(mixed_space)
U0 = Function(mixed_space)
v, q, r = split(V)
u, p, s = split(U)
u0, p0, s0 = split(U0)

L = inner(v, u)*dx - div(v)*p*dx + v[0]*dx + q*div(u)*dx + r*(s - s0)*dx
a = derivative(L, U, dU)

for level0 in xrange(1000):
    for level in xrange(10):

        A = Matrix();
        b = Vector();

        # This does not appear to leak
        problem = MyProblem(a, L)
        problem.F(b, U.vector())
        problem.J(A, U.vector())

        # This does leak
        #solver = NewtonSolver()
        #solver.solve(problem, U.vector())

        # This does leak (goes through call-back to Python)
        lu = LUSolver()
        lu.solve(A, U.vector(), b)

        U.vector().zero()
    print "End inner loop"

[Johan Hake (johan-hake)]

On Saturday 13 February 2010 13:08:05 Garth Wells wrote:
> Public bug reported:
>
> The below code leads to a memory leak (I see a leak over about 2MB over
> 90s). The leak is related to the Expression. If I comment out the call
> to
>
> eval(Array<double>&, const Array<double>&);
>
> from
>
> dolfin::GenericFunction::evaluate(double*, const double*, const
> ufc::cell) const
>
> I don't detect a leak.

I can confirm this bug. Not sure it is in the typemap, as it should also have
shown up if we moved f = MyExpression(degree=1) outside of the loop.

Johan

> ------
>
> from dolfin import *
> parameters["form_compiler"]["cpp_optimize"] = True
> parameters["form_compiler"]["optimize"] = True
>
> class MyExpression(Expression):
> def eval(self, values, x):
> if x[0] < DOLFIN_EPS:
> values[0] = 1.0
>
> mesh = UnitSquare(32, 32)
> for t in xrange(20000):
> for level in xrange(100):
> V = FunctionSpace(mesh, "CG", 1)
> v = TestFunction(V)
> u = Function(V)
> u0 = Function(V)
> f = MyExpression(degree=1)
> L = v*(u-u0)*dx + v*f*ds
> b = assemble(L)
> print "----"
>
> ** Affects: dolfin
> Importance: Undecided
> Status: New
>

[Garth Wells (garth-wells)]

Johan Hake wrote:
> On Saturday 13 February 2010 13:08:05 Garth Wells wrote:
>> Public bug reported:
>>
>> The below code leads to a memory leak (I see a leak over about 2MB over
>> 90s). The leak is related to the Expression. If I comment out the call
>> to
>>
>> eval(Array<double>&, const Array<double>&);
>>
>> from
>>
>> dolfin::GenericFunction::evaluate(double*, const double*, const
>> ufc::cell) const
>>
>> I don't detect a leak.
>
> I can confirm this bug. Not sure it is in the typemap, as it should also have
> shown up if we moved f = MyExpression(degree=1) outside of the loop.
>

I wonder if it could be in FFC and related to caching.

Garth

> Johan
>
>> ------
>>
>> from dolfin import *
>> parameters["form_compiler"]["cpp_optimize"] = True
>> parameters["form_compiler"]["optimize"] = True
>>
>> class MyExpression(Expression):
>> def eval(self, values, x):
>> if x[0] < DOLFIN_EPS:
>> values[0] = 1.0
>>
>> mesh = UnitSquare(32, 32)
>> for t in xrange(20000):
>> for level in xrange(100):
>> V = FunctionSpace(mesh, "CG", 1)
>> v = TestFunction(V)
>> u = Function(V)
>> u0 = Function(V)
>> f = MyExpression(degree=1)
>> L = v*(u-u0)*dx + v*f*ds
>> b = assemble(L)
>> print "----"
>>
>> ** Affects: dolfin
>> Importance: Undecided
>> Status: New
>>
>

[Garth Wells (garth-wells)]

Johan Hake wrote:
> On Saturday 13 February 2010 13:08:05 Garth Wells wrote:
>> Public bug reported:
>>
>> The below code leads to a memory leak (I see a leak over about 2MB over
>> 90s). The leak is related to the Expression. If I comment out the call
>> to
>>
>> eval(Array<double>&, const Array<double>&);
>>
>> from
>>
>> dolfin::GenericFunction::evaluate(double*, const double*, const
>> ufc::cell) const
>>
>> I don't detect a leak.
>
> I can confirm this bug. Not sure it is in the typemap, as it should also have
> shown up if we moved f = MyExpression(degree=1) outside of the loop.
>

I've add some scripts for testing memory to

  lp:~dolfin-core/dolfin/sandbox

under 'memtest'.

I'm seeing a case where a call-back from C++ to a Python sublass of
NonlinearProblem is causing a big leak.

Garth

> Johan
>

[Garth Wells (garth-wells)]

If you place

  return std::make_pair(0, true);

just after

  // Compute F(u)
  nonlinear_problem.form(*A, *b, x);
  nonlinear_problem.F(*b, x);

in NewtonSolver.cpp, the below script leaks so badly that your computer will crash if you don't kill it pretty quick.

from dolfin import *

parameters["linear_algebra_backend"] = "uBLAS"

class MyProblem(NonlinearProblem):
    def __init__(self, a, L):
        NonlinearProblem.__init__(self)
        self.L = L
        self.a = a
        #self.L = Form(L)
        #self.a = Form(a)
    def F(self, b, x):
        return
    def J(self, A, x):
        return

mesh = UnitSquare(16, 16, "crossed")
V = FunctionSpace(mesh, "Discontinuous Lagrange", 0)
v = TestFunction(V)
du = TrialFunction(V)
u = Function(V)
u0 = Function(V)

for level0 in xrange(10000):
    for level in xrange(10):

        L = inner(v, u-u0)*dx
        a = derivative(L, u, du)

        # This leaks like crazy
        problem = MyProblem(a, L)
        solver = NewtonSolver()
        solver.solve(problem, u.vector())

        # This does not leak
        #problem = VariationalProblem(a, L, [], nonlinear=True)
        #problem.solve(u)
    print "End inner loop"

[Garth Wells (garth-wells)]

Here's an even simpler leaky example. It doesn't involve any forms.

from dolfin import *
parameters["linear_algebra_backend"] = "uBLAS"
class MyProblem(NonlinearProblem):
    def F(self, b, x):
        return
    def J(self, A, x):
        return

for level0 in xrange(1000000):
    problem = MyProblem()
    solver = NewtonSolver()
    x = Vector()
    solver.solve(problem, x)

[Garth Wells (garth-wells)]

Looks like it is a SWIG bug, which can be worked around by removing the SWIG flag '-dirvtable', see the change

http://bazaar.launchpad.net/~dolfin-core/dolfin/main/revision/4511

[Garth Wells (garth-wells)]

I'll close this because the SWIG options have been changed, and this seems to help. There are still tests in

  lp:~dolfin-core/dolfin/sandbox

that exhibit leaks. I'll register a new report for these.

[Johan Hake (johan-hake)]

On Sunday 14 February 2010 11:27:11 Garth Wells wrote:
> Looks like it is a SWIG bug, which can be worked around by removing the
> SWIG flag '-dirvtable', see the change

Ok, this actually make sense, as the -dirvtable creates a lookup table for
faster dynamic dispatches during runtime. This table is probably not deleted.
It is created the fist time a director method is called, and explains why the
code did not leak when the call to eval was out commented, as it was never
created. It also explains why the code did not leak when the expression were
created only once.

I can try to see if I can produce a minimal example that I can attach to a bug
report upstream to SWIG.

Johan

> http://bazaar.launchpad.net/~dolfin-core/dolfin/main/revision/4511
>
> ** Changed in: dolfin
> Importance: Undecided => High
>
> ** Changed in: dolfin
> Milestone: None => 0.9.7
>