Comment 2 for bug 531612

Including the traceback:
PythonVersion: 2.6.4
Traceback:
 Traceback (most recent call last):
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in exception_to_return_code
     return the_callable(*args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1037, in run_bzr
     ret = run(*run_argv)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 654, in run_argv_aliases
     return self.run(**all_cmd_args)
   File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 3058, in run
     exclude=safe_relpath_files(tree, exclude))
   File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/workingtree_4.py", line 197, in commit
     result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/mutabletree.py", line 229, in commit
     *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commit.py", line 393, in commit
     self.branch.repository, new_revno, self.rev_id)
   File "/usr/lib/python2.6/dist-packages/bzrlib/branch.py", line 909, in import_last_revision_info
     self.repository.fetch(source_repo, revision_id=revid)
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 1695, in fetch
     find_ghosts=find_ghosts, fetch_spec=fetch_spec)
   File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 3413, in fetch
     pb=pb, find_ghosts=find_ghosts)
   File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 81, in __init__
     self.__fetch()
   File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 107, in __fetch
     self._fetch_everything_for_search(search)
   File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 135, in _fetch_everything_for_search
     stream, from_format, [])
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 4238, in insert_stream
     return self._locked_insert_stream(stream, src_format, is_resume)
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 4271, in _locked_insert_stream
     self.target_repo.texts.insert_record_stream(substream)
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 1608, in insert_record_stream
     for _ in self._insert_record_stream(stream, random_id=False):
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 1714, in _insert_record_stream
     bytes = record.get_bytes_as('fulltext')
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 454, in get_bytes_as
     self._manager._prepare_for_extract()
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 554, in _prepare_for_extract
     self._block._ensure_content(self._last_byte)
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 158, in _ensure_content
     self._z_content, num_bytes + _ZLIB_DECOMP_WINDOW)
 OverflowError: signed integer is greater than maximum

Basically, it seems that we are asking zlib to do the decompression, and it is telling us that the request is invalid. I'm a bit surprised that the compression succeeded, to have the decompression fail.

The immediate fix is that we could probably change the single 'decompress()' call into looping over a fixed size decompression (1GB should be more than reasonable, 1-10MB may be just as good.)

However, I'll note that bzr has a general policy that we may allocate at least 3x the size of a single file (so in your case, of a 3.8GB file, we might allocate 3.8*3=11.4GB.) We've tuned commit to not, (it should be close to 1x), but we also probably have some code paths that still need to be tuned (some probably allocate >5x the size of a single text).

Anyway, just to say that bzr isn't going to handle a 4GB file particularly well regardless. But yes, this specific failure could probably be worked around (I just expect you'll run into quite a few more problems).