Excessive memory usage in python-subversion

Bug #54253 reported by Jelmer Vernooij on 2006-07-27
54
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Bazaar
Undecided
Unassigned
Bazaar Subversion Plugin
High
Jelmer Vernooij
subversion
Fix Released
Medium
bzr-svn (Debian)
Fix Released
Unknown
bzr-svn (Ubuntu)
Undecided
Jelmer Vernooij
subversion (Ubuntu)
High
Kees Cook

Bug Description

Branching from a large Subversion branch (15000+ revisions) results in excessive memory usage by Bazaar (about 1.5 Gigabytes). I'm not sure yet whether this is a Bazaar or a bzr-svn problem. bzr-svn doesn't ever keep references to more data than the current revision when fetching, but there might be a memory leak in the python svn bindings.

Related branches

Jelmer Vernooij (jelmer) wrote :

Also reproduced by Erik Bågfors.

Changed in bzr-svn:
importance: Untriaged → High
status: Unconfirmed → Confirmed
John A Meinel (jameinel) wrote :

Can you pull the latest bzr.dev and see what happens?

We fixed one of our big memory consumers. But that was for _KnitData, so it may have no bearing on what you are doing.

I used latest bzr.dev from this morning, revno 1885, so I guess not.
(1886 which is the latest now, doesn't seem to have anything to do with
memory stuff :) )

/Erik

tor 2006-07-27 klockan 15:28 +0000 skrev John A Meinel:

> Can you pull the latest bzr.dev and see what happens?
>
> We fixed one of our big memory consumers. But that was for _KnitData, so
> it may have no bearing on what you are doing.
>

Does the branch complete? If so you should be able to branch from the resulting bzr branch, and see how that performs, which should help eliminate whether it's a bzr-svn problem?

fre 2006-07-28 klockan 00:31 +0000 skrev Michael Ellerman:
> Does the branch complete? If so you should be able to branch from the
> resulting bzr branch, and see how that performs, which should help
> eliminate whether it's a bzr-svn problem?

Mine didn't complete.
I will try a smaller svn repo and see. A really small works just fine.

/Erik

Erik Bågfors (zindar) wrote :

fre 2006-07-28 klockan 00:31 +0000 skrev Michael Ellerman:
> Does the branch complete? If so you should be able to branch from the
> resulting bzr branch, and see how that performs, which should help
> eliminate whether it's a bzr-svn problem?

So I tried this with a smaller branch. This one only has 1598 revisions.
During the branching from svn it took 215MB of RAM. When branching the
bzr branch it took about 20MB. So this seams to be in bzr-svn.

Regards,
Erik

Jelmer Vernooij (jelmer) on 2006-10-08
Changed in bzr-svn:
assignee: nobody → jelmer

It looks like this has to do with the APR memory management routines that Subversion uses. Somewhere a reference isn't being dropped.

Jelmer Vernooij (jelmer) wrote :

in particular the revision information fetch code seems leaky. no idea why.

For now I'm rejecting this as part of bzr, if Jelmer thinks it is our fault, I'm happy to re-open it.

(I know we had a bug in 'bzr log' that consumed too much memory, but that has also been fixed)

Changed in bzr:
status: Unconfirmed → Rejected
Changed in bzr-svn:
status: Unknown → Confirmed
Jelmer Vernooij (jelmer) on 2007-07-20
Changed in bzr-svn:
assignee: nobody → jelmer
status: New → Confirmed
Jelmer Vernooij (jelmer) on 2007-07-20
Changed in bzr-svn:
status: Confirmed → Triaged

The Mercurial folks found a memory leak in svn.ra.get_log() that is at
least partially responsible for the memory usage:

http://hg.intevation.org/mercurial/crew/file/e8f4e40f285a/hgext/convert/subversion.py

  affects /products/subversion

Jelmer Vernooij (jelmer) wrote :

Newer versions of Subversion don't appear to leak as much memory.
Especially 1.3.x seems to use a lot of memory.
--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

I was importing one project from KDE SVN, which has currently 733k revisions, most of them obviously unrelated to the project. ^C and restart every few minutes was no fun, so I wrote this ugly workaround - http://users.musicbrainz.org/~luks/tmp/bzr-svn-log.diff - it has no error checking and is definitely not meant for inclusion in bzr-svn, but it did the job for me.

Nicholas Allen (nick-allen) wrote :

Thanks for the workaround! Will try it out. Would be nice if something like this could get into the official bzr-svn.

Jelmer Vernooij (jelmer) on 2007-12-14
Changed in subversion:
status: New → Unknown
Jelmer Vernooij (jelmer) wrote :

After some digging, I found out where the leak is (bug 3052 filed in Subversions Issue Tracker). I've added a workaround to bzr-svn that frees most of the memory python-subversion leaks.

Changed in subversion:
status: Unknown → Confirmed
Jelmer Vernooij (jelmer) wrote :

python-svn specific. I'll attach a proposed fix.

Changed in bzr-svn:
status: Triaged → Invalid
Jelmer Vernooij (jelmer) wrote :

Actually a leak in the python-subversion bindings.

Changed in bzr-svn:
status: Confirmed → Invalid
Jelmer Vernooij (jelmer) wrote :
Jelmer Vernooij (jelmer) wrote :
Changed in subversion:
status: Confirmed → Fix Released
Jelmer Vernooij (jelmer) on 2007-12-20
Changed in subversion:
importance: Undecided → High
status: New → Triaged
Jelmer Vernooij (jelmer) wrote :
Daniel Holbach (dholbach) wrote :

Kees: can you take a look at it?

Kees Cook (kees) wrote :

This debdiff looks good. Thanks for getting it tested! I'll upload it shortly.

Changed in subversion:
assignee: nobody → keescook
status: Triaged → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package subversion - 1.4.4dfsg1-1ubuntu5

---------------
subversion (1.4.4dfsg1-1ubuntu5) hardy; urgency=medium

  * Add debian/patches/python-memleak: Fix major memory leak in the
    Python bindings. Taken from Debian SVN head. (LP: #54253)

 -- Jelmer Vernooij <email address hidden> Fri, 18 Jan 2008 01:28:28 +0100

Changed in subversion:
status: Fix Committed → Fix Released
Changed in bzr-svn:
status: Confirmed → Fix Released
Changed in subversion:
importance: Unknown → Medium
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.