Bazaar Version Control System

Out of memory error in _ensure_content on auto repacking

Reported by Greg on 2010-07-07
54
This bug affects 9 people
Affects Status Importance Assigned to Milestone
Bazaar
High
John A Meinel

Bug Description

Running command
bzr commit -m "my comment"
On 2 or 3 modified small text files produces the output:

committing to: .....
modified .....
modified .....
aborting commit write group: Memory Error()
bzr: out of memory

System:
bzrlib v2.1.1
QBzr v0.1.8.4
Python v2.5.4
Windows 7 Pro 64bit, 6gb RAM.

Repository is ~30mb, ~5000 files version controlled.
Commit is bound to network accessible folder / branch.

Here is the entire .bzr.log file generated from the command
bzr commit -m 'my comment'

The critical bit is...

0.597 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x02F04BF0>, 42, 7045986) to an LRUSizeCache failed. value 611025734 is too big to fit in a the cache with size 41943040 52428800
2.220 aborting commit write group because of exception:

-----begin file-----
this is a debug log for diagnosing/reporting problems in bzr
you can delete or truncate this file, or include sections in
bug reports to https://bugs.launchpad.net/bzr/+filebug

Wed 2010-07-07 13:13:11 +1000
0.048 bazaar version: 2.1.1
0.050 bzr arguments: [u'commit', u'-m', u'my comment']
0.059 looking for plugins in C:/Users/userA/AppData/Roaming/bazaar/2.0/plugins
0.060 looking for plugins in C:/Program Files (x86)/Bazaar/plugins
0.150 encoding stdout as sys.stdout encoding 'cp850'
0.176 opening working tree 'C:/Dev/Proj1/userAAPlay'
0.261 preparing to commit
[ 2044] 2010-07-07 13:13:12.283 INFO: Committing to: V:/Proj1/userAA1/
0.368 Selecting files for commit with filter None
[ 2044] 2010-07-07 13:13:12.358 INFO: modified src/Tests/TestBB1.cpp
[ 2044] 2010-07-07 13:13:12.361 INFO: modified src/Tests/bbEngine.cpp
[ 2044] 2010-07-07 13:13:12.361 INFO: modified src/Tests/bbEngine.h
0.492 Auto-packing repository GCRepositoryPackCollection(CHKInventoryRepository('file:///C:/Dev/Proj1/.bzr/repository/')), which has 11 pack files, containing 20 revisions. Packing 10 files into 1 affecting 10 revisions
0.495 repacking 10 revisions
0.502 repacking 10 inventories
0.507 repacking chk: 10 id_to_entry roots, 5 p_id_map roots, 77 total keys
0.562 repacking 60 texts
0.597 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x02F04BF0>, 42, 7045986) to an LRUSizeCache failed. value 611025734 is too big to fit in a the cache with size 41943040 52428800
2.220 aborting commit write group because of exception:
2.222 Traceback (most recent call last):
  File "bzrlib\commit.pyo", line 402, in _commit
  File "bzrlib\repository.pyo", line 179, in commit
  File "bzrlib\repository.pyo", line 1563, in commit_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 2314, in _commit_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 2174, in _commit_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 1476, in autopack
  File "bzrlib\repofmt\pack_repo.pyo", line 1516, in _do_autopack
  File "bzrlib\repofmt\groupcompress_repo.pyo", line 693, in _execute_pack_operations
  File "bzrlib\repofmt\pack_repo.pyo", line 761, in pack
  File "bzrlib\repofmt\groupcompress_repo.pyo", line 478, in _create_pack_from_packs
  File "bzrlib\repofmt\groupcompress_repo.pyo", line 461, in _copy_text_texts
  File "bzrlib\repofmt\groupcompress_repo.pyo", line 402, in _copy_stream
  File "bzrlib\groupcompress.pyo", line 1704, in _insert_record_stream
  File "bzrlib\groupcompress.pyo", line 438, in get_bytes_as
  File "bzrlib\groupcompress.pyo", line 538, in _prepare_for_extract
  File "bzrlib\groupcompress.pyo", line 151, in _ensure_content
MemoryError

[ 2044] 2010-07-07 13:13:14.140 INFO: aborting commit write group: MemoryError()
2.295 Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 853, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1055, in run_bzr
  File "bzrlib\commands.pyo", line 661, in run_argv_aliases
  File "bzrlib\commands.pyo", line 665, in run_direct
  File "bzrlib\cleanup.pyo", line 122, in run_simple
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "bzrlib\builtins.pyo", line 3138, in run
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\workingtree_4.pyo", line 197, in commit
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\mutabletree.pyo", line 225, in commit
  File "bzrlib\commit.pyo", line 257, in commit
  File "bzrlib\cleanup.pyo", line 118, in run
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "bzrlib\commit.pyo", line 402, in _commit
  File "bzrlib\repository.pyo", line 179, in commit
  File "bzrlib\repository.pyo", line 1563, in commit_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 2314, in _commit_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 2174, in _commit_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 1476, in autopack
  File "bzrlib\repofmt\pack_repo.pyo", line 1516, in _do_autopack
  File "bzrlib\repofmt\groupcompress_repo.pyo", line 693, in _execute_pack_operations
  File "bzrlib\repofmt\pack_repo.pyo", line 761, in pack
  File "bzrlib\repofmt\groupcompress_repo.pyo", line 478, in _create_pack_from_packs
  File "bzrlib\repofmt\groupcompress_repo.pyo", line 461, in _copy_text_texts
  File "bzrlib\repofmt\groupcompress_repo.pyo", line 402, in _copy_stream
  File "bzrlib\groupcompress.pyo", line 1704, in _insert_record_stream
  File "bzrlib\groupcompress.pyo", line 438, in get_bytes_as
  File "bzrlib\groupcompress.pyo", line 538, in _prepare_for_extract
  File "bzrlib\groupcompress.pyo", line 151, in _ensure_content
MemoryError

2.297 Transferred: 0KiB (0.0K/s r:0K w:0K)
2.298 return code 3
-----end file-----

Related branches

lp:~jameinel/bzr/2.4-max-entries-gc-602614
Merged into lp:bzr at revision 5870
Jelmer Vernooij (community): Approve on 2011-05-16
Martin Packman: Pending requested 2011-05-16
Greg (gregspecialsource) on 2010-07-08
description: updated

The commit is triggering an automatic repack, you can separate the two operations by doing an explicit 'bzr pack'.
Can you try that and report whether it's still failing ?

summary: - Out of memory error on tiny commit
+ Out of memory error on auto repacking
Changed in bzr:
importance: Undecided → High
status: New → Confirmed
Greg (gregspecialsource) wrote :

Thanks Vincent. I eventually grew so frustrated I deleted the repository and started afresh. I will see if I can restore it and try what you say, just to report the result. I'll report back in a few days.

Vincent Ladeuil (vila) on 2010-07-16
Changed in bzr:
status: Confirmed → Incomplete
Greg (gregspecialsource) wrote :

Vincent, I have recovered enough files to recreate the issue.
The command:
  bzr pack
Produces the output:
  bzr: out of memory

Vincent Ladeuil (vila) wrote :

Ok, so it's indeed a repack problem. I thought John commented on the issue and even had a patch to propose but his comment is not there and I can't find it back in my mail either...

@Greg: Do you have a reproducing recipe ?

Changed in bzr:
status: Incomplete → Confirmed
John A Meinel (jameinel) wrote :

@vila you're probably thinking about bug #605480

Greg (gregspecialsource) wrote :

I believe I can reproduce the issue at will.
I don't have any unique steps to recreate the issue on a stable repository. I am also unable to share the repository as it is part of a proprietary commercial code base.

Karl Bielefeldt (kbielefe) wrote :

I believe I'm seeing this same bug by a different route. I'm getting an out of memory error on the exact same line when it tries to decompress a text larger than 1 GB. Seems to be directly related to this python bug: http://bugs.python.org/issue8571

My repo at work is the one that exhibits the bug. I'm planning to figure out which python versions contain the patch this weekend and test on Monday.

Karl Bielefeldt (kbielefe) wrote :

I was wrong about it being related to that python bug. My python is patched for that bug. I have managed to track it down further that might aid someone else to reproduce it.

A little while ago, I accidentally added my "tags" file. It is over 1 GB in size, but is highly compressible, down to around 54 MB. Now whenever there is a need to decompress that file, such as switching to a branch that contains that file, or doing a bzr check --repo, it fails with out of memory. Doing an auto repack would also hit it, which would explain why Greg is seeing it.

Changed in bzr:
status: Confirmed → In Progress
assignee: nobody → Karl Bielefeldt (kbielefe)
Changed in bzr:
status: In Progress → Confirmed
assignee: Karl Bielefeldt (kbielefe) → nobody
John A Meinel (jameinel) on 2010-11-09
Changed in bzr:
milestone: none → 2.3b4
assignee: nobody → John A Meinel (jameinel)
Vincent Ladeuil (vila) on 2010-12-02
Changed in bzr:
milestone: 2.3b4 → 2.3.0
Jelmer Vernooij (jelmer) on 2011-02-01
tags: added: autopack memory packs
Vincent Ladeuil (vila) wrote :

@jam: ping, this is targeted at 2.3.0 freezing tomorrow, should we retarget ?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/2/2011 4:56 AM, Vincent Ladeuil wrote:
> @jam: ping, this is targeted at 2.3.0 freezing tomorrow, should we
> retarget ?
>

Just remove the target.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1J9GsACgkQJdeBCYSNAANvIACdEzXbaYtQRGbtQPYL36wCZsA1
VcIAoNXSKYjmit+rTUGPUd1LIGRLYV33
=uL5r
-----END PGP SIGNATURE-----

Vincent Ladeuil (vila) on 2011-02-03
Changed in bzr:
milestone: 2.3.0 → none
Martin Pool (mbp) on 2011-03-02
summary: - Out of memory error on auto repacking
+ Out of memory error in _ensure_content on auto repacking

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 3/2/2011 1:52 AM, Martin Pool wrote:
> ** Summary changed:
>
> - Out of memory error on auto repacking
> + Out of memory error in _ensure_content on auto repacking
>

_ensure_content is the extraction side of things. (Decompress the zlib
data into raw groupcompress instructions that can be used to generate
the fulltexts.)

Just adding some context.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1uWXQACgkQJdeBCYSNAAPcgQCgkGmv6o2pDpPc0RoeWni44QfA
GQ8AnR+MW/t/F8YhajrnxbCr+9HaGxVZ
=IIz/
-----END PGP SIGNATURE-----

Jay Wilson (jwilson-rvue) wrote :

Does anyone have a workaround on this? Is there a process to cleanse my tree without starting from scratch?

treaves (treaves) wrote :

As a workaround, you can do the following:
bzr pack nosmart+<url to your repository>

Jay Wilson (jwilson-rvue) wrote :

Thanks treaves! Unfortunately I get the response:
bzr: out of memory

I watch the process in top in a separate console and see that bzr crashes when I reach about 500M of resident memory usage.

treaves (treaves) wrote :

Jay: sorry, that worked for me. Are you running the command from a local machine? Maybe that is the difference. It is my server that has this defect, and the command I pasted actually performs the pack on the client.

Jay Wilson (jwilson-rvue) wrote :

Its all remote... I finally got past the issue by using "push" to send the tree into the server instead of "pull" or "update". I have 3 remote servers, 2 servers that produce a bzr: out of memory and one server that will not (each has 2Gb), but the server that works is our dev server, running a much lighter load. I guess I could try to get more memory, but this really shouldn't be happening. If it gets the point where we have to rebuild the tree from scratch I might jump ship over to svn if no one has fixed this yet. Kinda frustrating when your trying to deploy new software.... Thanks treaves...

Jay Wilson (jwilson-rvue) wrote :

Problem resurfaced. Prevented deployment of code release. I've switched myself and my team of 5 developers to svn.

I'm having this problem, too.

Bzr is running on a Windows 7 (32 bit) with service pack 1.

There is one commit (pack) with ~ 2.5 GB which is probably causing the memory error. Both 'bzr pack' and 'bzr pack nosmart' fail with out of memory:

C:\nnn\mmm>bzr pack
bzr: out of memory
**** entering debugger
> e:\nnn\mmm\bzrlib\groupcompress.pyo(293)_create_z_content_from_chunks()

A normal commit (bzr commit) fails at: bzrlib\groupcompress.pyo(151)_ensure_content()

Is there any workaround I can try?

John A Meinel (jameinel) wrote :

I just associated a branch which might help. The idea is that if it gets a text which is large, rather than creating a delta index which overlays every byte, it 'skips' some. This should reduce the best-possible compression it can compute, but should lower peak memory as well. I tested it with MAX_NUM_ENTRIES = 1000 (which would mean that texts >16kB would be under-sampled). And the resulting bzr repository was 80,488kB rather than 79,892kB.

I don't have a great feeling for where the best cutoff is, etc.

If you want to test it, I suggest you copy your repository to a temp location, build this bzr, and then run:
 test/bzr pack temp/repo -Dmemory

That should give some information about whether this is worthwhile or not.

Note, the change is not well tested, but I did do some sanity tests. As such, I wouldn't run it on your live repository, but it certainly would help give information about whether it is worth pursuing this further.

The other workaround is to just switch to a 64-bit machine and run "bzr pack" there. It should only need to be run on a single machine one time, and from then the clients should be ok with the one-large-pack for a while.

John, thanks for your reply. I've tried it with a 64-bit machine (Win 7 SP1), but it failed with "out of memory" too. This machine has only 2 GB RAM, though.

Is there any way to prevent bzr from repackaging? Removing the pack-file causes it to crash immediately.

Greg (gregspecialsource) wrote :

Encountering this same problem again right now I believe. Can't check in changes.
Note this error occurs attempting to check in even a single small file, so something must be corrupted. I am not trying to check in large files and the workstation is 64bit Windows with 24GB ram.

Bazaar Explorer commit status window looks like:

Run command: bzr commit -m "blah blah"
Committing to: V:/Proj/Bob/
modified src/BobLocal/file1.h
aborting commit write group: MemoryError()
bzr: out of memory
Use -Dmem_dump to dump memory to a file.

Running command 'bzr check -v' results in out of memory error.

Greg (gregspecialsource) wrote :

I'm afraid I will have to abandon Bazaar if this issue can't be fixed real soon. I can't accept this level of instability and my coworkers are angry. Another friendly developer is about to abandon for the same reason. This is very unfortunate as I love Bazaar and admire design goals. I realize free software means waiting for volunteer activity. I'd pay for improvement if it were possible.

My work around for this issue once again was to backup local tree, delete the local repository, pull another version, then copy back the local tree to preserve code and versioning.

John A Meinel (jameinel) on 2011-04-27
Changed in bzr:
status: Confirmed → In Progress

Now, with a new repository and after 10 commits, bzr tries to repack and fails again with an out of memory error. Starting anew I would lose all history.

Is it possible to increase the number of commits after a repack happens?

Vincent Ladeuil (vila) on 2011-05-26
Changed in bzr:
milestone: none → 2.4b3
status: In Progress → Fix Released
Martin Pool (mbp) wrote :

For those following this bug, <https://code.launchpad.net/~jameinel/bzr/2.4-max-entries-gc-602614/+merge/61072> adds a fix in to bzr 2.4 that should reduce memory consumption when repacking. If you would like to test 2.4b3 or the nightly ppa and let us know if it fixes it for you or not, that would be good.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.