SystemError('../Objects/stringobject.c:4271: bad argument to internal function') inside groupcompress during commit of large tree

[climpinho]

aborting commit write group: SystemError('../Objects/stringobject.c:4271: bad argument to internal function',)
bzr: ERROR: exceptions.SystemError: ../Objects/stringobject.c:4271: bad argument to internal function

 Traceback (most recent call last):
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in exception_to_return_code
     return the_callable(*args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1037, in run_bzr
     ret = run(*run_argv)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 654, in run_argv_aliases
     return self.run(**all_cmd_args)
   File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 3058, in run
     exclude=safe_relpath_files(tree, exclude))
   File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/workingtree_4.py", line 197, in commit
     result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/mutabletree.py", line 229, in commit
     *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commit.py", line 360, in commit
     self._update_builder_with_changes()
   File "/usr/lib/python2.6/dist-packages/bzrlib/commit.py", line 633, in _update_builder_with_changes
     self.work_tree, self.basis_revid, iter_changes):
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 760, in record_iter_changes
     file_id, text, heads, nostore_sha)
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 826, in _add_text_to_weave
     nostore_sha=nostore_sha, random_id=self.random_revid)[0:2]
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 1258, in _add_text
     nostore_sha=nostore_sha))[0]
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 1763, in _insert_record_stream
     flush()
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 1643, in flush
     bytes = self._compressor.flush().to_bytes()
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 326, in to_bytes
     self._create_z_content()
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 319, in _create_z_content
     self._create_z_content_from_chunks()
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 307, in _create_z_content_from_chunks
     compressed_chunks = map(compressor.compress, self._content_chunks)
 SystemError: ../Objects/stringobject.c:4271: bad argument to internal function

[Martin Pool (mbp)]

Hi,

Can you tell us anything more about this failure? Is it reproducible? Can you give a script or a tarball of a repository that reproduces it? Are you trying to commit a particularly large file or something?

This is a bit like, but not the same as, some other recently-reported bugs in calling zlib inside groupcompress.

[climpinho (djsme69)]

Hi,

it's a directory with a lot of files!

The files are small and medium size, but it seems that the error occurs because the number of files to commit!

i only run bzr commit -m "MSG"!! the init and add command run normaly.

[Martin Pool (mbp)]

If you can reproduce this, can you run with BZR_PDB=1 then do

  pp locals()

when the error occurs.

[Martin Pool (mbp)]

How many is a lot?

[climpinho (djsme69)]

Ok, the values are 2592 directories, 28455 files.

the result off BZR_PDB=1 is :

aborting commit write group: SystemError('../Objects/stringobject.c:4271: bad argument to internal function',)
bzr: ERROR: exceptions.SystemError: ../Objects/stringobject.c:4271: bad argument to internal function

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    attaching the crash file
        /.cache/crash/bzr-20100331184222-2913.crash
    and including a description of the problem.

    The crash file is plain text and you can inspect or edit it to remove
    private information.
**** entering debugger
> /usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py(291)_create_z_content_from_chunks()
-> compressed_chunks = map(compressor.compress, self._content_chunks)
(Pdb) pp locals()
{'compressor': <zlib.Compress object at 0x5e1ce70>,
 'self': <bzrlib.groupcompress.GroupCompressBlock object at 0x714fc10>}
(Pdb)

[climpinho (djsme69)]

I forgot to tell that before the crash the memory is all consumed by the bzr process and when there is no more memory the crash occurs

[Chadarius (csutton-chadarius)]

I started having this issue when I added two iso images to a bzr repo that were 4.7GB in total size.
1.7G 2010-05-24 16:30 KB3AIK_EN.iso
3.0G 2010-05-03 16:24 SW_DVD5_SA_Win_Ent_7_64BIT_English_Full_MLF_X15-70749.ISO

As soon as I move them out of the repo everything works just fine.

[Martin Pool (mbp)]

I wonder if this bug is to do with overflow and confusion between 32 and 64 bit int types.

For now I would not recommend adding ISOs to the bzr repository.

[Martin Packman (gz)]

This is a bug in the Python zlib module, which Python 2.6.6 should fix. However duplicate bug 723234 appears to be from on 2.6.6 so the upstream fix may be incomplete.