bzr mv --auto crash: out of memory

[Greg]

I have moved 50 files and renamed a few of them in the process.
Running the command 'bzr mv --auto' or simulated versions results in the following error message:
bzr: out of memory
Use -Dmem_dump to dump memory to a file.

The .bzr.log file shows:
0.047 bazaar version: 2.4b5
0.047 bzr arguments: [u'mv', u'--auto', u'--dry-run']
0.078 looking for plugins in C:/Users/username/AppData/Roaming/bazaar/2.0/plugins
0.078 looking for plugins in C:/Program Files (x86)/Bazaar/plugins
0.109 encoding stdout as sys.stdout encoding 'cp850'
0.140 opening working tree 'C:/src/Proj/myProj'
1.201 Transferred: 0kB (0.0kB/s r:0kB w:0kB)
1.201 Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 946, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1150, in run_bzr
  File "bzrlib\commands.pyo", line 699, in run_argv_aliases
  File "bzrlib\commands.pyo", line 721, in run
  File "bzrlib\cleanup.pyo", line 135, in run_simple
  File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
  File "bzrlib\builtins.pyo", line 855, in run
  File "bzrlib\builtins.pyo", line 876, in run_auto
  File "bzrlib\rename_map.pyo", line 230, in guess_renames
  File "bzrlib\rename_map.pyo", line 116, in file_match
  File "bzrlib\rename_map.pyo", line 108, in get_all_hits
  File "bzrlib\workingtree.pyo", line 495, in get_file_lines
MemoryError

1.217 return code 3

[John A Meinel (jameinel)]

I'm not sure what a fix for this would be. mv --auto reads all the unknown files into memory and computes some line-wise statistics about what files would be similar.

Can you give a bit more information about the size of the files in question, etc.?

The best I can think of would be to write out summary information into a file rather than holding it in memory for the whole time.

[Greg (gregspecialsource)]

More info as requested...
I moved 50 files totaling 650KB

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 8/31/2011 1:06 AM, Greg wrote:
> More info as requested... I moved 50 files totaling 650KB
>

What about the size of the files that were not moved? It is possible
that we had to read some of them to decide whether or not they were
part of the moved set.

Certainly 50files and 650kB doesn't look like something that should
run out of memory. Is there a ulimit, or low maximum memory on your
machine?

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk5d4NQACgkQJdeBCYSNAAPTGgCgvx/5+guz08hGAihU66f0B9+5
KPIAoLeDW/maotOYkZOhgAKr3P6xz/At
=bE1y
-----END PGP SIGNATURE-----

[Greg (gregspecialsource)]

> What about the size of the files that were not moved?
The repository has 10,000 files totaling 523MB with the largest single file 60MB

[Aaron Bentley (abentley)]

It's not supposed to hold all the files in memory. It's supposed to generate a map of hashes to file ids. There is a hard limit of 10 Mi hashes. If multiple copies of the file-id were stored in memory, perhaps that could cause this memory exhaustion, but the file-ids should be interned.