Excessive memory usage in python-subversion

[Jelmer Vernooij]

Branching from a large Subversion branch (15000+ revisions) results in excessive memory usage by Bazaar (about 1.5 Gigabytes). I'm not sure yet whether this is a Bazaar or a bzr-svn problem. bzr-svn doesn't ever keep references to more data than the current revision when fetching, but there might be a memory leak in the python svn bindings.

[Jelmer Vernooij (jelmer)]

Also reproduced by Erik Bågfors.

[John A Meinel (jameinel)]

Can you pull the latest bzr.dev and see what happens?

We fixed one of our big memory consumers. But that was for _KnitData, so it may have no bearing on what you are doing.

[Erik Bågfors (zindar)]

I used latest bzr.dev from this morning, revno 1885, so I guess not.
(1886 which is the latest now, doesn't seem to have anything to do with
memory stuff :) )

/Erik

tor 2006-07-27 klockan 15:28 +0000 skrev John A Meinel:

> Can you pull the latest bzr.dev and see what happens?
>
> We fixed one of our big memory consumers. But that was for _KnitData, so
> it may have no bearing on what you are doing.
>

[Michael Ellerman (michael-ellerman)]

Does the branch complete? If so you should be able to branch from the resulting bzr branch, and see how that performs, which should help eliminate whether it's a bzr-svn problem?

[Erik Bågfors (zindar)]

fre 2006-07-28 klockan 00:31 +0000 skrev Michael Ellerman:
> Does the branch complete? If so you should be able to branch from the
> resulting bzr branch, and see how that performs, which should help
> eliminate whether it's a bzr-svn problem?

Mine didn't complete.
I will try a smaller svn repo and see. A really small works just fine.

/Erik

[Erik Bågfors (zindar)]

fre 2006-07-28 klockan 00:31 +0000 skrev Michael Ellerman:
> Does the branch complete? If so you should be able to branch from the
> resulting bzr branch, and see how that performs, which should help
> eliminate whether it's a bzr-svn problem?

So I tried this with a smaller branch. This one only has 1598 revisions.
During the branching from svn it took 215MB of RAM. When branching the
bzr branch it took about 20MB. So this seams to be in bzr-svn.

Regards,
Erik

[Jelmer Vernooij (jelmer)]

It looks like this has to do with the APR memory management routines that Subversion uses. Somewhere a reference isn't being dropped.

[Jelmer Vernooij (jelmer)]

in particular the revision information fetch code seems leaky. no idea why.

[John A Meinel (jameinel)]

For now I'm rejecting this as part of bzr, if Jelmer thinks it is our fault, I'm happy to re-open it.

(I know we had a bug in 'bzr log' that consumed too much memory, but that has also been fixed)

[Jelmer Vernooij (jelmer)]

The Mercurial folks found a memory leak in svn.ra.get_log() that is at
least partially responsible for the memory usage:

http://hg.intevation.org/mercurial/crew/file/e8f4e40f285a/hgext/convert/subversion.py

  affects /products/subversion

[Jelmer Vernooij (jelmer)]

Newer versions of Subversion don't appear to leak as much memory.
Especially 1.3.x seems to use a lot of memory.
--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

[Lukáš Lalinský (luks)]

I was importing one project from KDE SVN, which has currently 733k revisions, most of them obviously unrelated to the project. ^C and restart every few minutes was no fun, so I wrote this ugly workaround - http://users.musicbrainz.org/~luks/tmp/bzr-svn-log.diff - it has no error checking and is definitely not meant for inclusion in bzr-svn, but it did the job for me.

[Nicholas Allen (nick-allen)]

Thanks for the workaround! Will try it out. Would be nice if something like this could get into the official bzr-svn.

[Jelmer Vernooij (jelmer)]

After some digging, I found out where the leak is (bug 3052 filed in Subversions Issue Tracker). I've added a workaround to bzr-svn that frees most of the memory python-subversion leaks.

[Jelmer Vernooij (jelmer)]

python-svn specific. I'll attach a proposed fix.

[Jelmer Vernooij (jelmer)]

Actually a leak in the python-subversion bindings.

[Daniel Holbach (dholbach)]

Kees: can you take a look at it?

[Kees Cook (kees)]

This debdiff looks good. Thanks for getting it tested! I'll upload it shortly.

[Launchpad Janitor (janitor)]

This bug was fixed in the package subversion - 1.4.4dfsg1-1ubuntu5

---------------
subversion (1.4.4dfsg1-1ubuntu5) hardy; urgency=medium

  * Add debian/patches/python-memleak: Fix major memory leak in the
    Python bindings. Taken from Debian SVN head. (LP: #54253)

 -- Jelmer Vernooij <email address hidden> Fri, 18 Jan 2008 01:28:28 +0100