bzr-hg consumes very large amounts of memory during initial download

[Peter Bex]

While downloading the initial set of revisions, hg-bzr blows up in memory causing Launchpad's bzr integration to fail on importing these two branches:

https://code.launchpad.net/~peter-bex/chamilo/chamilo-hg-mirror

and

https://code.launchpad.net/~peter-bex/chamilo/chamilo-classic-hg-mirror

I got 1.6G of memory used during the process of fetching manifests early on: at 500/4219

See also https://answers.edge.launchpad.net/launchpad-code/+question/118906

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter Bex wrote:
> Public bug reported:
>
> While downloading the initial set of revisions, hg-bzr blows up in
> memory causing Launchpad's bzr integration to fail on importing these
> two branches:
>
> https://code.launchpad.net/~peter-bex/chamilo/chamilo-hg-mirror
>
> and
>
> https://code.launchpad.net/~peter-bex/chamilo/chamilo-classic-hg-mirror
>
> I got 1.6G of memory used during the process of fetching manifests early
> on: at 500/4219
>
> See also https://answers.edge.launchpad.net/launchpad-
> code/+question/118906
>
> ** Affects: bzr-hg
> Importance: Undecided
> Status: New
>

Would you be able to install Meliae (lp:meliae), and then when memory
consumption gets big use either SIGQUIT (Ctrl+\) or SIGBREAK
(Ctrl+Pause/Break) on Windows to get into a debugger. And then run:

 from meliae import scanner
 scanner.dump_all_objects('memory.dump')

And then bzip2 the result and attach it to this bug?

That would make it easier to determine where the memory is actually
being consumed.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxNx+MACgkQJdeBCYSNAAMHiACgmALF4yx8dI3q+o2g/NwHSO3B
CmQAn2dGrIYr+QGwoixf+4QsbXQhzUWl
=umdh
-----END PGP SIGNATURE-----

[Jelmer Vernooij (jelmer)]

  status triaged
  importance medium

This is a known issue - because of the way the mercurial smart server
protocol works (it sends revisions first, manifests second and then
contents) the memory usage of is linear with the number of revisions agt
the moment.

This particular repository is pretty large.