MemoryError in bytes_to_gzip during commit to knit repository

Bug #193685 reported by Kalle Kipinä
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Bazaar
Confirmed
Medium
Unassigned

Bug Description

DESCRIPTION
When committing a file whose size is 1M or larger, bzr crashes with "MemoryError". Files of approximately 500k do not cause this behaviour.

OUTPUT
~/saku2/raakakuvat >/foo/bin/bzr/bzr ci -m "test" new2
Committing to: /foo/saku2/
bzr: ERROR: exceptions.MemoryError:

Traceback (most recent call last):
  File "/foo/bin/bzr/bzrlib/commands.py", line 834, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/foo/bin/bzr/bzrlib/commands.py", line 790, in run_bzr
    ret = run(*run_argv)
  File "/foo/bin/bzr/bzrlib/commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/foo/bin/bzr/bzrlib/builtins.py", line 2329, in run
    author=author)
  File "/foo/bin/bzr/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/foo/bin/bzr/bzrlib/workingtree_4.py", line 246, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/foo/bin/bzr/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/foo/bin/bzr/bzrlib/mutabletree.py", line 187, in commit
    revprops=revprops, *args, **kwargs)
  File "/foo/bin/bzr/bzrlib/commit.py", line 355, in commit
    self._update_builder_with_changes()
  File "/foo/bin/bzr/bzrlib/commit.py", line 656, in _update_builder_with_changes
    self._populate_from_inventory(specific_files)
  File "/foo/bin/bzr/bzrlib/commit.py", line 784, in _populate_from_inventory
    content_summary)
  File "/foo/bin/bzr/bzrlib/commit.py", line 826, in _record_entry
    self.parent_invs, path, self.work_tree, content_summary)
  File "/foo/bin/bzr/bzrlib/repository.py", line 362, in record_entry_contents
    ie.file_id, lines, heads, nostore_sha)
  File "/foo/bin/bzr/bzrlib/repofmt/pack_repo.py", line 91, in _add_text_to_weave
    self.random_revid)
  File "/foo/bin/bzr/bzrlib/repofmt/pack_repo.py", line 1169, in _add_text_to_weave
    random_id=random_revid, check_content=False)[0:2]
  File "/foo/bin/bzr/bzrlib/versionedfile.py", line 138, in add_lines_with_ghosts
    parent_texts, nostore_sha, random_id, check_content)
  File "/foo/bin/bzr/bzrlib/knit.py", line 917, in _add_lines_with_ghosts
    parent_texts, None, nostore_sha, random_id)
  File "/foo/bin/bzr/bzrlib/knit.py", line 1007, in _add
    lines, [line_bytes])
  File "/foo/bin/bzr/bzrlib/knit.py", line 2350, in _record_to_data
    compressed_bytes = bytes_to_gzip(bytes)
  File "/foo/bin/bzr/bzrlib/tuned_gzip.py", line 62, in bytes_to_gzip
    return ''.join(result)
MemoryError

bzr 1.2.0 on python 2.5.1.final.0 (sunos5)
arguments: ['/foo/bin/bzr/bzr', 'ci', '-m', 'test', 'new2']
encoding: 'ISO8859-1', fsenc: 'ISO8859-1', lang: 'en_US'
plugins:
  launchpad /foo/bin/bzr/bzrlib/plugins/launchpad [unknown]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

~/saku2/raakakuvat >~/bin/bzr/bzr info
Standalone tree (format: pack-0.92)
Location:
  branch root: /foo/saku2

Tags: knits memory
Revision history for this message
mrflip (mrflip) wrote :

I'm also seeing this error.

James Westby (james-w)
Changed in bzr:
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Dirk Räder (db6edr) wrote : Re: MemoryError in bytes_to_gzip during commit on linux

I can confirm this error on WinXP Professional SP3. The system has 4 GiB RAM and a swap file of the same size.

Actions taken:
bzr init .
bzr init-repo --no-trees f:\some\path
bzr branch . f:\some\path\branch
bzr bind f:\some\path\branch
bzr add
(adding 7051 files, 310 folders, 2.17 GiB on disk)
bzr commit
(working some time ...)

Error Message:
bzr: ERROR: exceptions.MemoryError:

Traceback (most recent call last):
  File "bzrlib\commands.pyc", line 846, in run_bzr_catch_errors
  File "bzrlib\commands.pyc", line 797, in run_bzr
  File "bzrlib\commands.pyc", line 499, in run_argv_aliases
  File "bzrlib\builtins.pyc", line 2364, in run
  File "bzrlib\decorators.pyc", line 165, in write_locked
  File "bzrlib\workingtree_4.pyc", line 240, in commit
  File "bzrlib\decorators.pyc", line 165, in write_locked
  File "bzrlib\mutabletree.pyc", line 197, in commit
  File "bzrlib\commit.pyc", line 355, in commit
  File "bzrlib\commit.pyc", line 655, in _update_builder_with_changes
  File "bzrlib\commit.pyc", line 783, in _populate_from_inventory
  File "bzrlib\commit.pyc", line 825, in _record_entry
  File "bzrlib\repository.pyc", line 361, in record_entry_contents
  File "bzrlib\repofmt\pack_repo.pyc", line 97, in _add_text_to_weave
  File "bzrlib\repofmt\pack_repo.pyc", line 1180, in _add_text_to_weave
  File "bzrlib\versionedfile.pyc", line 137, in add_lines_with_ghosts
  File "bzrlib\knit.pyc", line 999, in _add_lines_with_ghosts
  File "bzrlib\knit.pyc", line 1088, in _add
  File "bzrlib\knit.pyc", line 2415, in _record_to_data
  File "bzrlib\tuned_gzip.pyc", line 62, in bytes_to_gzip
MemoryError

bzr 1.5 on python 2.5.2 (win32)
arguments: ['C:\\Programme\\Bazaar\\bzr.exe', 'ci']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  launchpad C:\Programme\Bazaar\plugins\launchpad [unknown]

Revision history for this message
Dirk Räder (db6edr) wrote : Re: MemoryError in bytes_to_gzip during commit

Hi folks,

I just stumpled upon this bug again. On my current Ubuntu 09.04 64 bit machine this bug does NOT occur.

As I also have a 64 bit Windows installation to check against: Is there any way to get a real 64bit installation of bazaar? I'd prefer an installer if that's available.

For testing purposes, only the command line interface of bazaar has to work.

Revision history for this message
Martin Pool (mbp) wrote :

Hi,

All the tracebacks here seem to relate to commits to the old 'knit' repository format. If you upgrade bzr to 2.1 and upgrade the repository to format 2a, memory usage should be much less.

Dirk, you could try running bzr from source under a 64-bit Python. https://bugs.edge.launchpad.net/bzr/+bug/331342 asks for a proper 64-bit build.

summary: - MemoryError in bytes_to_gzip during commit
+ MemoryError in bytes_to_gzip during commit to knit repository
Revision history for this message
Dirk Räder (db6edr) wrote :

Hi Martin,

yes, the tracebacks were made in 2008 against the old 'knit' format. I have not yet created a traceback for the current 2a format, but can do so if you think it helpful.

The result running the most current Bazaar release on Windows 7 x64 is definitely the same: Out of memory error. I suppose it has to do something with the packs file growing beyond the 'magical' size of 2 GiB.

I subscribed to the bug you pointed at and will have a look into it.

Regards, Dirk

Jelmer Vernooij (jelmer)
tags: added: knits
Jelmer Vernooij (jelmer)
tags: added: check-for-breezy
Jelmer Vernooij (jelmer)
tags: removed: check-for-breezy
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.