fast-export out of memory error

[Greg]

I'm trying to export my branch, however it fails with 'out of memory'.
The bzr.log file contains...

Thu 2011-07-28 15:53:33 +1000
0.046 bazaar version: 2.4b5
0.046 bzr arguments: [u'fast-export', u'C:/bzrtest2/main/.bzr']
0.062 looking for plugins in C:/Users/myname/AppData/Roaming/bazaar/2.0/plugins
0.062 looking for plugins in C:/Program Files (x86)/Bazaar/plugins
0.109 encoding stdout as sys.stdin encoding 'cp850'
[ 5940] 2011-07-28 15:53:33.618 INFO: 15:53:33 Calculating the revisions to include ...
[ 5940] 2011-07-28 15:53:33.634 INFO: 15:53:33 Starting export of 827 revisions ...
1.076 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x02DDD650>, 304220324, 53656873) to an LRUSizeCache failed. value 120102299 is too big to fit in a the cache with size 41943040 52428800
...SNIP lots of similar lines...
82.926 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x02DDD650>, 271957253, 9880575) to an LRUSizeCache failed. value 76989455 is too big to fit in a the cache with size 41943040 52428800
84.268 Transferred: 0kB (0.0kB/s r:0kB w:0kB)
84.268 Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 946, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1150, in run_bzr
  File "bzrlib\commands.pyo", line 699, in run_argv_aliases
  File "bzrlib\commands.pyo", line 721, in run
  File "bzrlib\cleanup.pyo", line 135, in run_simple
  File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
  File "C:/Program Files (x86)/Bazaar/plugins\fastimport\cmds.py", line 644, in run
  File "C:/Program Files (x86)/Bazaar/plugins\fastimport\exporter.py", line 145, in run
  File "C:/Program Files (x86)/Bazaar/plugins\fastimport\exporter.py", line 257, in emit_commit
  File "C:/Program Files (x86)/Bazaar/plugins\fastimport\exporter.py", line 192, in print_cmd
  File "fastimport\commands.pyo", line 141, in __repr__
  File "fastimport\commands.pyo", line 189, in to_string
MemoryError
84.268 return code 3

What can I do to work around this?
I'm running Windows 7 with 12GB RAM. Perhaps I can run 64bit, increase cache limits? Split it up? I've tried repacking texts and pulling a fresh branch. This is fairly urgent if anyone can help.

[Jelmer Vernooij (jelmer)]

On Thu, 2011-07-28 at 05:58 +0000, Greg wrote:
> Public bug reported:
>
> I'm trying to export my branch, however it fails with 'out of memory'.
> The bzr.log file contains...
[...]
> What can I do to work around this? I'm running Windows 7 with 12GB RAM.
> Perhaps I can run 64bit, increase cache limits? Split it up? I've
> tried repacking texts and pulling a fresh branch. This is fairly
> urgent if anyone can help.
How big are the files in the repository?

Bazaar doesn't deal with big files very well at the moment, and may keep
several copies of a file it processes in memory. This is fine for
smaller files, but it quickly becomes an issue for anything that is a
couple of hundred megabyte large.

Cheers,

Jelmer

[Greg (gregspecialsource)]

The entire repository is 450mb in size. The largest single file would be <100mb. It's mostly source code and some binaries, no large data. I can't access the files at the moment but I think there are about 15K files and the export shows <850 revisions recorded.

[Martin Pool (mbp)]

It looks like the specific problem here is in python-fastimport where
it's trying to produce one string for the whole commit which is pretty
silly. That could be changed to probably return an iterator with a
bit of work.

The best short term workaround is probably to install and run under a
64-bit python.

Martin

[Greg (gregspecialsource)]

Thanks Martin. I would like to try running under 64-bit python but have not been able to find any information on how to do this. Please point me in the right direction if you can. I'm running the Windows Standalone package for Bazaar and have little knowledge of Python.

[Jelmer Vernooij (jelmer)]

On Fri, 2011-07-29 at 10:39 +0000, Greg wrote:
> The entire repository is 450mb in size. The largest single file would
> be <100mb. It's mostly source code and some binaries, no large data. I
> can't access the files at the moment but I think there are about 15K
> files and the export shows <850 revisions recorded.
With that number of files, some of the recent fixes in bzr 2.3.4 and bzr
2.4 might have an effect on the memory usage.

Cheers,

Jelme

[John A Meinel (jameinel)]

Sounds like a bug in fastimport if it needs to buffer the entire repository history. That would never scale well.

For example, bzr's repository was only 20MB when compressed, but was 2.4GB when expanded across all file texts.

[Jelmer Vernooij (jelmer)]

It doesn't buffer the entire repository history. For bzr.dev here its memory usage doesn't exceed 2% memory usage on my system.

[ced (alexanderypema)]

One workaround is doing the conversion on linux (in a virtual machine for example), there's no 64 bit client for windows, and given you have enough ram (it took 6GB RAM here for a 800MB repo), it'll work just fine on 64 bit linux.