Comment 12 for bug 461992

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew Bennetts wrote:
> John: how much does the repo format matter to memory consumption? Given
> that the report started with bzr 1.14 I'm guessing this repo isn't using
> the latest format. (Although 1.1G is a pretty big repo even in 1.9 or
> pack-0.92 formats.)
>
> Also, does this bug need to be Critical? I don't think we'd block a
> release on fixing this (although obviously it would be great to fix or
> at least improve this).
>

I didn't mark it Critical, Patrick Regan did. I agree with the High
setting. I think Patrick was responding to Jim's comment that it
prevents them from using bzr.

Given the location of the failure, I'm guessing this is an issue that
they are extracting a very large file from a knitpack format repository.
Which has not been as memory tuned as the groupcompress code. Note that
both of them require at least 2x the size of the file. So if you have a
100MB file, we need at least 200MB of ram to extract it.

(A copy of the original, + a copy to update.)

2a format repos will generally be significantly lower than knitpack
(pack-0.92, 1.6, 1.9, 1.14) format repositories. Because they will work
on the raw bytes, rather than splitting things into lines (and
encounting a PyString 24 bytes per line + 4 bytes in a PyList.)

However, if the problem is during *fetch* that is a different issue.

Oh, and *merge* requires at least 3x the memory. Because we have to
unpack BASE + THIS + OTHER. And I'm guessing that means at least 4x,
since when unpacking one of those, we need the 2x memory to apply the
deltas.

So... I would expect this to get better with 2a...

Note that in knitpack, I think we are using iter_files_bytes, which
potentially unpacks many files at once. So it could be even higher. It
is a bit tricky to give specifics without having the actual content
available.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksf4tEACgkQJdeBCYSNAAPSHwCeOjCXDqlOqGNbw0WtNTMZ4lZV
Qc0AoIi8/ZILVaGA+2ijki4tVPJD87uT
=4v4q
-----END PGP SIGNATURE-----