out of memory in bzrlib._groupcompress_pyx.DeltaIndex._populate_first_index

[treaves]

I can no longer push to my repository. When I try, I get the following in the error log:

Sat 2011-02-19 19:46:10 +0000
0.059 bazaar version: 2.3.0
0.059 bzr arguments: [u'serve', u'--inet', u'--directory=/', u'--allow-writes']
0.088 looking for plugins in /home/treaves/.bazaar/plugins
0.088 looking for plugins in /usr/lib/pymodules/python2.6/bzrlib/plugins
0.098 looking for plugins in /usr/lib/python2.6/dist-packages/bzrlib/plugins
0.103 encoding stdout as osutils.get_user_encoding() 'ANSI_X3.4-1968'
3.607 Auto-packing repository GCRepositoryPackCollection(CHKInventoryRepository('filtered-139787365415440:///opt/bzr/DartsTrainer/.bzr/repository/')), which has 19 pack files, containing 100 revisions. Packing 19 files into 1 affecting 100 revisions
3.608 repacking 100 revisions
3.750 repacking 100 inventories
3.786 repacking chk: 99 id_to_entry roots, 39 p_id_map roots, 138 total keys
3.895 repacking 872 texts
4.048 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x21ed210>, 10159, 4021575) to an LRUSizeCache failed. value 75541536 is too big to fit in a the cache with size 41943040 52428800
4.099 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x21ed210>, 10159, 4021575) to an LRUSizeCache failed. value 75541536 is too big to fit in a the cache with size 41943040 52428800
4.186 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x21ea7d0>, 9042, 6832747) to an LRUSizeCache failed. value 90155494 is too big to fit in a the cache with size 41943040 52428800
4.298 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x21ea7d0>, 9042, 6832747) to an LRUSizeCache failed. value 90155494 is too big to fit in a the cache with size 41943040 52428800
4.329 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x21ed210>, 10159, 4021575) to an LRUSizeCache failed. value 75541536 is too big to fit in a the cache with size 41943040 52428800
4.342 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x21ed210>, 10159, 4021575) to an LRUSizeCache failed. value 75541536 is too big to fit in a the cache with size 41943040 52428800
4.622 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x21ea7d0>, 9042, 6832747) to an LRUSizeCache failed. value 90155494 is too big to fit in a the cache with size 41943040 52428800
5.990 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x21ed210>, 10159, 4021575) to an LRUSizeCache failed. value 75541536 is too big to fit in a the cache with size 41943040 52428800
23.172 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x21ea7d0>, 9042, 6832747) to an LRUSizeCache failed. value 90155494 is too big to fit in a the cache with size 41943040 52428800
66.876 Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/request.py", line 355, in _call_converting_errors
    return callable(*args, **kwargs)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/repository.py", line 760, in _inserter_thread
    stream, src_format, self.tokens)
  File "/usr/lib/pymodules/python2.6/bzrlib/repository.py", line 4105, in insert_stream
    hint = self.target_repo.commit_write_group()
  File "/usr/lib/pymodules/python2.6/bzrlib/repository.py", line 1640, in commit_write_group
    result = self._commit_write_group()
  File "/usr/lib/pymodules/python2.6/bzrlib/repofmt/pack_repo.py", line 2333, in _commit_write_group
    hint = self._pack_collection._commit_write_group()
  File "/usr/lib/pymodules/python2.6/bzrlib/repofmt/pack_repo.py", line 2193, in _commit_write_group
    result = self.autopack()
  File "/usr/lib/pymodules/python2.6/bzrlib/repofmt/pack_repo.py", line 1469, in autopack
    return self._do_autopack()
  File "/usr/lib/pymodules/python2.6/bzrlib/repofmt/pack_repo.py", line 1509, in _do_autopack
    reload_func=self._restart_autopack)
  File "/usr/lib/pymodules/python2.6/bzrlib/repofmt/groupcompress_repo.py", line 800, in _execute_pack_operations
    result = packer.pack()
  File "/usr/lib/pymodules/python2.6/bzrlib/repofmt/pack_repo.py", line 745, in pack
    return self._create_pack_from_packs()
  File "/usr/lib/pymodules/python2.6/bzrlib/repofmt/groupcompress_repo.py", line 480, in _create_pack_from_packs
    self._copy_text_texts()
  File "/usr/lib/pymodules/python2.6/bzrlib/repofmt/groupcompress_repo.py", line 463, in _copy_text_texts
    'texts', self._get_progress_stream, 4)
  File "/usr/lib/pymodules/python2.6/bzrlib/repofmt/groupcompress_repo.py", line 401, in _copy_stream
    reuse_blocks=False):
  File "/usr/lib/pymodules/python2.6/bzrlib/groupcompress.py", line 1751, in _insert_record_stream
    nostore_sha=nostore_sha)
  File "/usr/lib/pymodules/python2.6/bzrlib/groupcompress.py", line 824, in compress
    start, end, type = self._compress(key, bytes, len(bytes) / 2, soft)
  File "/usr/lib/pymodules/python2.6/bzrlib/groupcompress.py", line 986, in _compress
    delta = self._delta_index.make_delta(bytes, max_delta_size)
  File "_groupcompress_pyx.pyx", line 246, in bzrlib._groupcompress_pyx.DeltaIndex.make_delta
  File "_groupcompress_pyx.pyx", line 224, in bzrlib._groupcompress_pyx.DeltaIndex._populate_first_index
AssertionError

69.312 return code 0

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

...

> delta = self._delta_index.make_delta(bytes, max_delta_size)
> File "_groupcompress_pyx.pyx", line 246, in bzrlib._groupcompress_pyx.DeltaIndex.make_delta
> File "_groupcompress_pyx.pyx", line 224, in bzrlib._groupcompress_pyx.DeltaIndex._populate_first_index
> AssertionError
>

This is the code in question:

        with nogil:
            self._index = create_delta_index(&self._source_infos[0], NULL)
        assert self._index != NULL

The assertion triggering means that we are unable to create the delta
index. Given the earlier warnings, I have a feeling this is actually a
memory error (malloc() returning NULL).

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1gOAsACgkQJdeBCYSNAAOp0wCbBz6vQNZPQRNLuTh6xHjVQUxM
nHoAoIz8xT6jeyTBZNZl8nDT9DuendM1
=AUuU
-----END PGP SIGNATURE-----

[Martin Packman (gz)]

Consensus here and in bug 721598 is that this is an OOM issue.

If so, the bug is that hitting it in _groupcompress_pyx.DeltaIndex._populate_first_index reports an unhelpful assertion rather than a MemoryError that Bazaar could usefully report.

As create_delta_index is from diff-delta.c which doesn't use the Python API so can't report a malloc failure as a python exception, instead perhaps pyrex callers of create_delta_index assert the preconditions then treat NULL return as OOM.

[treaves (treaves)]

I do not dispute that it is an OOM error; the issue is my entire repository is only 100 meg, and the pack files are - on average - about 50k. So, yes, this is generating the OOM condition. But it should not be.

[Martin Pool (mbp)]

On 22 February 2011 05:38, treaves <email address hidden> wrote:
> I do not dispute that it is an OOM error; the issue is my entire
> repository is only 100 meg, and the pack files are - on average - about
> 50k.  So, yes, this is generating the OOM condition.  But it should not
> be.

And do you know how much memory that server has?

[treaves (treaves)]

Yes, but I'm not sure why that's particularly relevant. Are you seriously
implying that a 100 meg repo with 50 k pack files may be too much for some
servers? Really?!

On Mon, Feb 21, 2011 at 9:41 PM, Martin Pool <email address hidden> wrote:

> On 22 February 2011 05:38, treaves <email address hidden> wrote:
> > I do not dispute that it is an OOM error; the issue is my entire
> > repository is only 100 meg, and the pack files are - on average - about
> > 50k. So, yes, this is generating the OOM condition. But it should not
> > be.
>
> And do you know how much memory that server has?
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/721897
>
> Title:
> compress issue makes bzr unusable
>
> Status in Bazaar Version Control System:
> Confirmed
>
> Bug description:
> I can no longer push to my repository. When I try, I get the
> following in the error log:
>
> Sat 2011-02-19 19:46:10 +0000
> 0.059 bazaar version: 2.3.0
> 0.059 bzr arguments: [u'serve', u'--inet', u'--directory=/',
> u'--allow-writes']
> 0.088 looking for plugins in /home/treaves/.bazaar/plugins
> 0.088 looking for plugins in /usr/lib/pymodules/python2.6/bzrlib/plugins
> 0.098 looking for plugins in
> /usr/lib/python2.6/dist-packages/bzrlib/plugins
> 0.103 encoding stdout as osutils.get_user_encoding() 'ANSI_X3.4-1968'
> 3.607 Auto-packing repository
> GCRepositoryPackCollection(CHKInventoryRepository('filtered-139787365415440:///opt/bzr/DartsTrainer/.bzr/repository/')),
> which has 19 pack files, containing 100 revisions. Packing 19 files into 1
> affecting 100 revisions
> 3.608 repacking 100 revisions
> 3.750 repacking 100 inventories
> 3.786 repacking chk: 99 id_to_entry roots, 39 p_id_map roots, 138 total
> keys
> 3.895 repacking 872 texts
> 4.048 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at
> 0x21ed210>, 10159, 4021575) to an LRUSizeCache failed. value 75541536 is too
> big to fit in a the cache with size 41943040 52428800
> 4.099 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at
> 0x21ed210>, 10159, 4021575) to an LRUSizeCache failed. value 75541536 is too
> big to fit in a the cache with size 41943040 52428800
> 4.186 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at
> 0x21ea7d0>, 9042, 6832747) to an LRUSizeCache failed. value 90155494 is too
> big to fit in a the cache with size 41943040 52428800
> 4.298 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at
> 0x21ea7d0>, 9042, 6832747) to an LRUSizeCache failed. value 90155494 is too
> big to fit in a the cache with size 41943040 52428800
> 4.329 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at
> 0x21ed210>, 10159, 4021575) to an LRUSizeCache failed. value 75541536 is too
> big to fit in a the cache with size 41943040 52428800
> 4.342 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at
> 0x21ed210>, 10159, 4021575) to an LRUSizeCache failed. value 75541536 is too
> big to fit in a the cache with size 41943040 52428800
> 4.622 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at
> 0x21ea7d0>, 9042, 6832747) to an LRUS...

[Vincent Ladeuil (vila)]

> Are you seriously implying that a 100 meg repo with 50 k pack files may be too much for some servers?

Nobody implies anything so far, we are trying to understand what is happening so we're asking questions. We'll draw conclusions later when enough information will be available.

> why that's particularly relevant.

Same here, until we understand what is happening, nobody knows what is relevant :-)

We are all pretty surprised that an OOM is raised (if that's indeed the issue even that hasn't been proved yet, though jam's intuitions are generally good) with a repository that size.

Now if the server has a really small memory footprint it *would* be an explanation (we had reports where tiny servers with less than 512MB were used with bzr). 512M may still be far enough in your case but may be not, so we'll be interested to know your server config.

Thanks in advance.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/22/2011 12:39 AM, Vincent Ladeuil wrote:
>> Are you seriously implying that a 100 meg repo with 50 k pack files
> may be too much for some servers?
>
> Nobody implies anything so far, we are trying to understand what is
> happening so we're asking questions. We'll draw conclusions later when
> enough information will be available.
>
>> why that's particularly relevant.
>
> Same here, until we understand what is happening, nobody knows what is
> relevant :-)
>
> We are all pretty surprised that an OOM is raised (if that's indeed the
> issue even that hasn't been proved yet, though jam's intuitions are
> generally good) with a repository that size.
>
> Now if the server has a really small memory footprint it *would* be an
> explanation (we had reports where tiny servers with less than 512MB were
> used with bzr). 512M may still be far enough in your case but may be
> not, so we'll be interested to know your server config.
>
> Thanks in advance.
>

We know that he has content that is something like 90MB in raw content
size. If that is in a single groupcompress block, that would hint that
the size of the original text is about 45MB. (The rule is to cap at 4MB,
or 2x the size of the largest text.)

We've seen pack operations consume roughly 4-6x the size of the largest
content. Which puts a lower bound on needed memory about 5x50=250MB.
Which could still be well under a 512MB server. Though if the 90MB was a
single text, and it was 6x, that gets you 540MB, etc.

Just to say, it may not be OOM. But it is pretty darn likely from the
other hints.

But yes, knowing the server config is useful info.

So would having the actual repository be available, if only for
supervised inspection.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1jXMwACgkQJdeBCYSNAAP9YACffvwhuKRuoYQ2/fqR9ynwN3ES
nmwAnA+vXibShLzCff+XSiPuu3FJKcai
=W3rI
-----END PGP SIGNATURE-----

[treaves (treaves)]

The server is a base VM from RackSpace: so it is configured with 256M memory and twice that for swap. In a quiescent state memory consumption runs 72 meg and 31 meg for swap. The machine currently hosts Trac (with Apache), and has hosted multiple Plone instances, although those are now typically shut down. It has mulit-gigabyte Subversion repositories as well, although they are accessed infrequently.

When the bazaar push is attempted, that 72 meg goes up to the full 256M, and the swap jumps to the full 512M.

The largest single file in the repo is a C header file that is 68meg (which is no longer used in the code base).

I'm more than willing to provide access to the system for someone that wants to debug.

[Martin Pool (mbp)]

So that probably explains it; 256MB is an unusually small server by modern standards. As a rule of thumb at the moment bzr needs 2-3x as much free memory as the largest file it's dealing with. We'd like to reduce that, but as a workaround at the moment maybe you can allocate more memory?

This is a distinct traceback from bug 602614 so I'll leave it separate.

[treaves (treaves)]

Have you ever tried to run a Plone instance? They actually state that performance is not something they have paid any attention to. This server can run three instances: 3 zodb with 6 clients, along with several mutli-gigabyte svn repositories, and about 10 Trac instances. But when those are all shut down, it can not handle a single 100 mb bazaars instance.

If this is the case, Bazaar is so intrinsically flawed, with such a fucked up design, that no one should ever consider it for anything other than academic interest. I mean seriously folks, look at this. Are you serious? Three zope instance, one with thousands of pages, a 26 GB svn repository, and Trac, and that's fine. But 20 files in Bazaar renders the entire server unusable? Good Gods!

Scrap the damn thing, and start over with people that know what they are doing!

[Robert Collins (lifeless)]

On Wed, Mar 2, 2011 at 4:18 PM, treaves <email address hidden> wrote:
> If this is the case, Bazaar is so intrinsically flawed, with such a
> fucked up design, that no one should ever consider it for anything other
> than academic interest.  I mean seriously folks, look at this.  Are you
> serious?  Three zope instance, one with thousands of pages, a 26 GB svn
> repository, and Trac, and that's fine.  But 20 files in Bazaar renders
> the entire server unusable?  Good Gods!

Its most likely that your 68MB file is the problem: bzr will be using
24MB on its own to repack (68*3) - and even though we do logarithmic
backoff on packing, it will be packed from time to time. bzr *happily*
handles repositories with hundreds of thousands of commits and
hundreds of thousands of files when the hardware matches the resources
its working with.

You've found a corner case - congrats. And I understand that its
frustrating, but lashing out really isn't going to help things.

As a workaround, you can do the following:
bzr pack nosmart+<url to your repository>

This will do the pack on your client, and once you have a larger pack
it won't try to repack it until the next power of 10 commits have
occured.

-Rob

[treaves (treaves)]

This did allow me to get past this issue. Thanks.