MemoryError in bytes_to_gzip during checkout

[trallallero]

during cloning of a very large project (550 MB) copying a file text exceeded available memory:
  File "bzrlib\repository.pyc", line 2312, in fetch
  File "bzrlib\fetch.pyc", line 103, in __init__
  File "bzrlib\fetch.pyc", line 132, in __fetch
  File "bzrlib\fetch.pyc", line 158, in _fetch_everything_for_revisions
  File "bzrlib\fetch.pyc", line 210, in _fetch_weave_text
  File "bzrlib\versionedfile.pyc", line 421, in join
  File "bzrlib\knit.pyc", line 2570, in join
  File "bzrlib\knit.pyc", line 2591, in _anno_to_plain_converter
  File "bzrlib\knit.pyc", line 2281, in _record_to_data
  File "bzrlib\tuned_gzip.pyc", line 57, in bytes_to_gzip
MemoryError

[Alexander Belchenko (bialix)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

trallallero пишет:
| Public bug reported:
|
| during a checkout operation of a very large project (550 MB)
| got a MemoryError
|
| informations:
|
| bzr 1.0.0 on python 2.5.1.final.0 (win32)

Try to increase swap file on your system.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHlkLGzYr338mxwCURAqkdAJ99JGLr0l0tgXiNZCTHyy0Nn8MX3ACdGwoC
hFf+RG7c7y8KN6BR0lSJoSc=
=emlf
-----END PGP SIGNATURE-----

[Robert Collins (lifeless)]

Memory use is important for us to correct; in this case a Try:Except MemoryError block around the calling code, with a fallback to streaming the content out of the compressed data is probably all that is needed.

[bgrupe (bgrupe)]

I've got a similar problem checking out a large project on windows:

D:\bzr>bzr co sftp://bzr/repos/main
Connected (version 2.0, client OpenSSH_4.6p1)
SSH bgrupe@bzr password:
Authentication (password) successful!
Secsh channel 1 opened.
Opened sftp connection (server version 3)
bzr: ERROR: exceptions.MemoryError:

Traceback (most recent call last):
  File "bzrlib\commands.pyc", line 834, in run_bzr_catch_errors
  File "bzrlib\commands.pyc", line 790, in run_bzr
  File "bzrlib\commands.pyc", line 492, in run_argv_aliases
  File "bzrlib\builtins.pyc", line 990, in run
  File "bzrlib\branch.pyc", line 783, in create_checkout
  File "bzrlib\decorators.pyc", line 165, in write_locked
  File "bzrlib\branch.pyc", line 1727, in pull
  File "bzrlib\decorators.pyc", line 165, in write_locked
  File "bzrlib\branch.pyc", line 1526, in pull
  File "bzrlib\decorators.pyc", line 165, in write_locked
  File "bzrlib\branch.pyc", line 1478, in update_revisions
  File "bzrlib\decorators.pyc", line 165, in write_locked
  File "bzrlib\branch.pyc", line 286, in fetch
  File "bzrlib\repository.pyc", line 949, in fetch
  File "bzrlib\decorators.pyc", line 165, in write_locked
  File "bzrlib\repository.pyc", line 2645, in fetch
  File "bzrlib\fetch.pyc", line 109, in __init__
  File "bzrlib\fetch.pyc", line 139, in __fetch
  File "bzrlib\fetch.pyc", line 169, in _fetch_everything_for_search
  File "bzrlib\fetch.pyc", line 219, in _fetch_weave_text
  File "bzrlib\versionedfile.pyc", line 421, in join
  File "bzrlib\knit.pyc", line 2633, in join
  File "bzrlib\knit.pyc", line 2446, in read_records_iter_raw
  File "bzrlib\knit.pyc", line 2009, in get_raw_records
  File "bzrlib\transport\sftp.pyc", line 344, in _sftp_readv
MemoryError

I've got plenty of memory available (3 gb physical, 1gb swap) and it crashes when it uses approx. 750mb ram.

I've got no such error on Linux.

[Zygmunt Krynicki (zyga)]

This affects:

Bazaar (bzr) 2.1.0
  Python interpreter: C:\Program Files\Bazaar\python25.dll 2.5.4
  Python standard library: C:\Program Files\Bazaar\lib\library.zip
  Platform: Windows-XP-5.1.2600-SP3
  bzrlib: C:\Program Files\Bazaar\lib\library.zip\bzrlib
  Bazaar configuration: C:\Documents and Settings\z.krynicki\Application Data\ba
zaar\2.0
  Bazaar log file: C:\Documents and Settings\z.krynicki\My Documents\.bzr.log

Copyright 2005, 2006, 2007, 2008, 2009 Canonical Ltd.
http://bazaar-vcs.org/

bzr comes with ABSOLUTELY NO WARRANTY. bzr is free software, and
you may use, modify and redistribute it under the terms of the GNU
General Public License version 2 or later.

Bazaar is part of the GNU Project to produce a free operating system.

[Zygmunt Krynicki (zyga)]

A backtrace from my case:

156.625 Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 853, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1055, in run_bzr
  File "bzrlib\commands.pyo", line 661, in run_argv_aliases
  File "bzrlib\commands.pyo", line 665, in run_direct
  File "bzrlib\cleanup.pyo", line 122, in run_simple
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "bzrlib\builtins.pyo", line 3138, in run
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\workingtree_4.pyo", line 197, in commit
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\mutabletree.pyo", line 225, in commit
  File "bzrlib\commit.pyo", line 257, in commit
  File "bzrlib\cleanup.pyo", line 118, in run
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "bzrlib\commit.pyo", line 387, in _commit
  File "bzrlib\commit.pyo", line 622, in _update_builder_with_changes
  File "bzrlib\repository.pyo", line 767, in record_iter_changes
  File "bzrlib\repository.pyo", line 833, in _add_text_to_weave
  File "bzrlib\groupcompress.pyo", line 1242, in _add_text
  File "bzrlib\groupcompress.pyo", line 1758, in _insert_record_stream
  File "bzrlib\groupcompress.pyo", line 1633, in flush
  File "bzrlib\groupcompress.pyo", line 310, in to_bytes
  File "bzrlib\groupcompress.pyo", line 303, in _create_z_content
  File "bzrlib\groupcompress.pyo", line 291, in _create_z_content_from_chunks
MemoryError

[Zygmunt Krynicki (zyga)]

I was attempting to add a directory with 1.5GB of data inside (text and binary)

Largest file size was a 919MB .a file

[Zygmunt Krynicki (zyga)]

I think I understand the problem

Basically bzr assumes it can store the whole file in memory N times. This assumption is wrong.
I think that for binary files or basically for files > 32MB or so bzr should switch to streaming store method, without any compression.

This is much better than current situation because it will allow to store arbitrary stuff in .bzr which is what many people/companies need. Imagine a single tree for "project" that has ~10GB that is currently using p4. You cannot switch away from p4 because you cannot even store the _snapshot_ of that project into bzr without hitting this issue. Ideological statements such as "don't store binary stuff" don't help as the reality of existing projects cannot be altered easily.

I think there should be one big bug about this that can simply track all the other bugs that we have here (about memory consumption). I suspect that at least some of them are about storing large stuff in bzr. A separate issue should track events where bzr is not using streaming api and requires lots of memory to process/store/whatever regular source files with history.

[Zygmunt Krynicki (zyga)]

This was using the same version of bzr+windows as reported by me above.
I have cygwin to make the junk file but python and bzr is from windows 2.1 release.

D:\bzr-memory-issue-test>dd if=/dev/urandom of=big-junk bs=1024 count=1024K
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 74.526 seconds, 14.4 MB/s

D:\bzr-memory-issue-test>bzr init .
Created a standalone tree (format: 2a)

D:\bzr-memory-issue-test>bzr add .
adding big-junk

D:\bzr-memory-issue-test>bzr ci -m "test"
Committing to: D:/bzr-memory-issue-test/
added big-junk
aborting commit write group: MemoryError()
bzr: out of memory

This fails with the following:
Wt 2010-03-02 15:53:26 +0100
0.093 bazaar version: 2.1.0
0.093 bzr arguments: [u'ci', u'-m', u'test']
0.109 looking for plugins in C:/Documents and Settings/z.krynicki/Application Data/bazaar/2.0/plugins
0.109 looking for plugins in C:/Program Files/Bazaar/plugins
0.218 encoding stdout as sys.stdout encoding 'cp852'
0.265 opening working tree 'D:/bzr-memory-issue-test'
0.406 preparing to commit
[ 1644] 2010-03-02 15:53:26.957 INFO: Committing to: D:/bzr-memory-issue-test/
0.437 Selecting files for commit with filter None
[ 1644] 2010-03-02 15:53:26.973 INFO: added big-junk
0.453 aborting commit write group because of exception:
0.453 Traceback (most recent call last):
  File "bzrlib\commit.pyo", line 387, in _commit
  File "bzrlib\commit.pyo", line 622, in _update_builder_with_changes
  File "bzrlib\repository.pyo", line 762, in record_iter_changes
MemoryError

[ 1644] 2010-03-02 15:53:26.973 INFO: aborting commit write group: MemoryError()
0.515 Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 853, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1055, in run_bzr
  File "bzrlib\commands.pyo", line 661, in run_argv_aliases
  File "bzrlib\commands.pyo", line 665, in run_direct
  File "bzrlib\cleanup.pyo", line 122, in run_simple
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "bzrlib\builtins.pyo", line 3138, in run
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\workingtree_4.pyo", line 197, in commit
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\mutabletree.pyo", line 225, in commit
  File "bzrlib\commit.pyo", line 257, in commit
  File "bzrlib\cleanup.pyo", line 118, in run
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "bzrlib\commit.pyo", line 387, in _commit
  File "bzrlib\commit.pyo", line 622, in _update_builder_with_changes
  File "bzrlib\repository.pyo", line 762, in record_iter_changes
MemoryError

0.515 Transferred: 0KiB (0.0K/s r:0K w:0K)
0.515 return code 3

[Matt Nordhoff (mnordhoff)]

There _is_ one big bug about memory consumption -- bug #109114. Please don't add random other MemoryErrors to this bug; it was about one in particular.

[Martin Pool (mbp)]

I want to keep the different failure modes that end up in "out of memory" in different bugs since they need to be fixed separately.

Zygmunt, your crash in comment #5 is bug 890085. #8 is possibly a different bug; I'll see if I can reproduce that too.

The originally reported failure was in knit.py which is an obsolete format. So, although the general thing of memory usage in commit is important, I'm going to close this bug.