Too much memory used when branching

[Jean-Paul Calderone]

After upgrading bzr-svn and python-subvertpy, checking out a branch from svn using a file:/// URL uses at least 2.9GB memory. The process is eventually OOM killed before it completes, so real peak usage is unknown, but presumably higher.

The old versions in use were:

python-subvertpy 0.7.5-1~bazaar1~lucid1
bzr-svn 1.0.4+bzr3475-1~bazaar1~lucid1

The new versions are:

python-subvertpy 0.8.4-1~lucid1
bzr-svn 1.1.0-1~bazaar1~lucid1

The particular case for which I encountered this behavior was branching from the Twisted svn repository.

[Jean-Paul Calderone (exarkun)]

There doesn't appear to be anything interesting in bzr.log:

Thu 2011-10-27 07:25:38 -0600
0.034 bazaar version: 2.3.3
0.034 bzr arguments: [u'branch', u'file:///svn/Twisted/branches/halfclose-tls-5341', u'/var/www/bzr/Twisted/branches/halfclose-tls-5341']
0.051 looking for plugins in /var/www/.bazaar/plugins
0.051 looking for plugins in /usr/lib/pymodules/python2.6/bzrlib/plugins
0.059 looking for plugins in /usr/lib/python2.6/dist-packages/bzrlib/plugins
0.097 bzr-svn: using Subversion 1.6.6 (), subvertpy 0.8.4
0.116 encoding stdout as sys.stdout encoding 'UTF-8'

[Jelmer Vernooij (jelmer)]

Sorry this is such a pain. I'd really like to get to the bottom of this.

Just doing a clone of this branch into a shared repository seems reasonably fast here:

localhost:/tmp/twisted% time bzr branch svn://svn.twistedmatrix.com/svn/Twisted/branches/halfclose-tls-5341
Branched 16035 revisions.
bzr branch svn://svn.twistedmatrix.com/svn/Twisted/branches/halfclose-tls-534 5.50s user 0.22s system 53% cpu 10.641 total

Is this a clone from scratch or into a shared repository?

Can you reproduce this against a remote repository, or just with file:// ?

[Jean-Paul Calderone (exarkun)]

> Sorry this is such a pain.

Me too :( I appreciate your help so far.

It's a clone into a shared repository. I just tried using svn:// and it completed in 1m50s with peak memory usage around 170MB. So the problem does seem limited to use of the file:// url.

> 5.50s user 0.22s system 53% cpu 10.641 total

Am I reading this right? You branched in 10 seconds? I wonder why it still takes 1m50s when the branch is checked out from the host on which the repository resides.

[Jean-Paul Calderone (exarkun)]

Hmmm. Checking out the branch again over svn:// only takes 10 seconds. Is that because the first time had to initialize some more svn cache stuff? Or is it just because it's the second time that branch has been checked out in that shared repository?

[Jelmer Vernooij (jelmer)]

Can you perhaps try importing from file:/// again?

[Jean-Paul Calderone (exarkun)]

Without changing anything else? Sure - it seems to behave the same way as it did the first time.

[Jelmer Vernooij (jelmer)]

Ok, so that means that the problem is specific to file:// I guess.

The only thing we're doing different for file:// URLs is that by default the cache is disabled. Not having to build the cache can save a lot of time, especially if you're fetching from a repository that has a lot of other projects in it.

Can you set "use-cache = True" in the relevant entry in ~/.bazaar/subversion.conf and try again?

I can try mirrorring the twisted repository here locally, but that's going to take quite a bit of time and bandwidth.

[Jean-Paul Calderone (exarkun)]

Adding the `use-cache = True´ line to the configuration causes branching from the file:// URL to complete in about ten seconds.

[Jelmer Vernooij (jelmer)]

Ok, so still not great but a lot better.

Thanks for verifying, I'll have a closer look at what might be causing this.