Comment 22 for bug 407834

Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 407834] Re: fetching from pack to 2a format is slow

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

...

> Once we convert, it might also make sense to make tarballs of a pre-
> initialised shared repo available for machines with too little memory to
> do the initial branching themselves.
>

bzr 2.1.0b2 cuts memory consumption in ~1/2 for my test cases.
(Launchpad's code base). For LP's codebase it was ~1GB => 500MB for
branching outside the shared repository. I haven't specifically tweaked
any of the conversion code paths. I would expect them to be "better" but
probably not 2:1 better.

Converting from 1.9 => 2a is not going to get a lot better. More than
have the time is generally spent extracting the data from the 1.9 format
(I believe the times I saw was 2/3rds extraction, 1/3rd insertion into
the new format.)

There is one patch I know that can make the conversion significantly faster:

=== modified file 'bzrlib/xml8.py'
- --- bzrlib/xml8.py 2009-07-07 04:32:13 +0000
+++ bzrlib/xml8.py 2009-11-16 16:49:46 +0000
@@ -433,9 +433,9 @@
                 pass
             else:
                 # Only copying directory entries drops us 2.85s => 2.35s
- - # if cached_ie.kind == 'directory':
- - # return cached_ie.copy()
- - # return cached_ie
+ if cached_ie.kind == 'directory':
+ return cached_ie.copy()
+ return cached_ie
                 return cached_ie.copy()

         kind = elt.tag

Note that the code is already there, just commented out. The reason for
this is that it is "unsafe" for some code paths, but it works great for
the conversion case. (If someone mutates the object that you have stored
in the cache... bad things happen. The conversion code *doesn't* though.)

That also might help memory consumption a little bit, but I wouldn't
expect too much there.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksBg4UACgkQJdeBCYSNAANwLQCfY8id1CFonwMlz7Nkg8K7Mec2
V7YAoJWdT3IuBwfG4IzUVo/6or7mTpkt
=qrhR
-----END PGP SIGNATURE-----