MemoryError in groupcompress to_bytes during commit of large tree

Bug #408526 reported by Jan Danielsson on 2009-08-03
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Bazaar
Medium
Unassigned

Bug Description

$ export CVS_RSH=ssh
$ export <email address hidden>:/cvsroot
$ cd ~/bazaar
$ cvs export -r netbsd-5-0-RELEASE -d netbsd-5.0 src
(Now go drink 10 cups of tea)
$ cd netbsd-5.0
$ bzr init --2a
Created a standalone tree (format: 2a)
$ bzr add
(Wait. There was one ignored file on my system)
$ bzr ignored
usr.bin/nvi/docs/USD.doc/vi.ref/ref.so *.so
$ bzr add usr.bin/nvi/docs/USD.doc/vi.ref/ref.so
adding usr.bin/nvi/docs/USD.doc/vi.ref/ref.so
$ ulimit -d
262144
$ bzr commit -m "Initial commit."
[---]
added x11/tools/xkbcomp/xkbfile-xkbmisc.c
aborting commit write group: MemoryError("Can't allocate memory to compress data",)
bzr: ERROR: exceptions.MemoryError: Can't allocate memory to compress data

Traceback (most recent call last):
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/commands.py", line 729, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/commands.py", line 924, in run_bzr
    ret = run(*run_argv)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/commands.py", line 560, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/builtins.py", line 2937, in run
    exclude=safe_relpath_files(tree, exclude))
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/workingtree_4.py", line 226, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/mutabletree.py", line 228, in commit
    *args, **kwargs)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/commit.py", line 363, in commit
    self._update_builder_with_changes()
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/commit.py", line 644, in _update_builder_with_changes
    self.work_tree, self.basis_revid, iter_changes):
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/repository.py", line 762, in record_iter_changes
    file_id, lines, heads, nostore_sha)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/repository.py", line 831, in _add_text_to_weave
    check_content=False)[0:2]
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/groupcompress.py", line 1008, in add_lines
    nostore_sha=nostore_sha))[0]
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/groupcompress.py", line 1508, in _insert_record_stream
    flush()
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/groupcompress.py", line 1404, in flush
    bytes = self._compressor.flush().to_bytes()
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/groupcompress.py", line 290, in to_bytes
    self._z_content = compress(self._content)
MemoryError: Can't allocate memory to compress data

bzr 1.16.1 on python 2.5.4 (netbsd4)
arguments: ['/usr/pkg/bin/bzr', 'commit', '-m', 'Initial commit.']
encoding: '646', fsenc: '646', lang: None
plugins:
  bzrtools /usr/pkg/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.16]
  launchpad /usr/pkg/lib/python2.5/site-packages/bzrlib/plugins/launchpad [1.16.1]
  netrc_credential_store /usr/pkg/lib/python2.5/site-packages/bzrlib/plugins/netrc_credential_store [1.16.1]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.
$ rm -Rf .bzr/
$ bzr init --2a
Created a standalone tree (format: 2a)
$ bzr add
(Wait. There was one ignored file on my system)
$ bzr ignored
usr.bin/nvi/docs/USD.doc/vi.ref/ref.so *.so
$ bzr add usr.bin/nvi/docs/USD.doc/vi.ref/ref.so
adding usr.bin/nvi/docs/USD.doc/vi.ref/ref.so
$ ulimit -d 524288
$ ulimit -d
524288
$ bzr commit -m "Initial commit."
[---]
Committed revision 1.
(Runs ok!)

So, bzr uses a very large amount of memory during the commit phase. I ran top during the commit, and the python process appears to be using something just over 256MB, so 512 is a little bit excessive. But, IMHO, 256MB should have been more than enough.

Running NetBSD/amd64 4.0 with:
$ bzr --version
Bazaar (bzr) 1.16.1
  Python interpreter: /usr/pkg/bin/python2.5 2.5.4
  Python standard library: /usr/pkg/lib/python2.5
  bzrlib: /usr/pkg/lib/python2.5/site-packages/bzrlib
  Bazaar configuration: /home/jan/.bazaar
  Bazaar log file: /home/jan/.bzr.log

Andrew Bennetts (spiv) wrote :

I'd be curious to know if this is any better for you in 2.1? We did land some memory use improvements that may have helped this case.

Changed in bzr:
importance: Undecided → Medium
status: New → Confirmed
Andrew Bennetts (spiv) wrote :

p.s. Thanks for pointing out that we'd wrongly marked this as a dupe of bug 109114.

Martin Pool (mbp) wrote :

So the complaint here is that the tree has many files, but no large individual files?

tags: added: 2a
summary: - bzr commit on large projects require large amounts of memory
+ MemoryError in groupcompress to_bytes during commit of large tree

Replying to Martin Pool,

Yes, that's it exactly. However, as one can see from the log I posted, I fixed it by bumping up the memory limit for the process. But I wrote another bug report, which I think also was marked as a dupe. After I managed to do the initial commit, I tried branching. That failed with a memory error too. But in that case I didn't get it to finish, even with the memory limit bumped up. So it would seem that it consumes even more memory when branching.

Replying to Andrew Bennetts,

I will, and I'll get back with the results.

Jelmer Vernooij (jelmer) on 2017-11-09
tags: added: check-for-breezy
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers