SystemError('../Objects/stringobject.c:4271: bad argument to internal function') inside groupcompress during commit of large tree

Bug #542691 reported by climpinho on 2010-03-20
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Bazaar
Low
Unassigned
Python
Fix Released
Unknown

Bug Description

aborting commit write group: SystemError('../Objects/stringobject.c:4271: bad argument to internal function',)
bzr: ERROR: exceptions.SystemError: ../Objects/stringobject.c:4271: bad argument to internal function

 Traceback (most recent call last):
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in exception_to_return_code
     return the_callable(*args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1037, in run_bzr
     ret = run(*run_argv)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 654, in run_argv_aliases
     return self.run(**all_cmd_args)
   File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 3058, in run
     exclude=safe_relpath_files(tree, exclude))
   File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/workingtree_4.py", line 197, in commit
     result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/mutabletree.py", line 229, in commit
     *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commit.py", line 360, in commit
     self._update_builder_with_changes()
   File "/usr/lib/python2.6/dist-packages/bzrlib/commit.py", line 633, in _update_builder_with_changes
     self.work_tree, self.basis_revid, iter_changes):
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 760, in record_iter_changes
     file_id, text, heads, nostore_sha)
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 826, in _add_text_to_weave
     nostore_sha=nostore_sha, random_id=self.random_revid)[0:2]
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 1258, in _add_text
     nostore_sha=nostore_sha))[0]
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 1763, in _insert_record_stream
     flush()
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 1643, in flush
     bytes = self._compressor.flush().to_bytes()
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 326, in to_bytes
     self._create_z_content()
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 319, in _create_z_content
     self._create_z_content_from_chunks()
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 307, in _create_z_content_from_chunks
     compressed_chunks = map(compressor.compress, self._content_chunks)
 SystemError: ../Objects/stringobject.c:4271: bad argument to internal function

climpinho (djsme69) wrote :

Hi,

Can you tell us anything more about this failure? Is it reproducible? Can you give a script or a tarball of a repository that reproduces it? Are you trying to commit a particularly large file or something?

This is a bit like, but not the same as, some other recently-reported bugs in calling zlib inside groupcompress.

summary: - crash during commit
+ SystemError('../Objects/stringobject.c:4271: bad argument to internal
+ function') during commit
description: updated
summary: SystemError('../Objects/stringobject.c:4271: bad argument to internal
- function') during commit
+ function') inside groupcompress during commit
Martin Pool (mbp) on 2010-03-21
Changed in bzr:
status: New → Incomplete
climpinho (djsme69) wrote :

Hi,

it's a directory with a lot of files!

The files are small and medium size, but it seems that the error occurs because the number of files to commit!

i only run bzr commit -m "MSG"!! the init and add command run normaly.

Martin Pool (mbp) wrote :

If you can reproduce this, can you run with BZR_PDB=1 then do

  pp locals()

when the error occurs.

tags: added: 2a memory
summary: SystemError('../Objects/stringobject.c:4271: bad argument to internal
- function') inside groupcompress during commit
+ function') inside groupcompress during commit of large tree
Martin Pool (mbp) wrote :

How many is a lot?

Changed in bzr:
importance: Undecided → Low
status: Incomplete → Confirmed
climpinho (djsme69) wrote :

Ok, the values are 2592 directories, 28455 files.

the result off BZR_PDB=1 is :

aborting commit write group: SystemError('../Objects/stringobject.c:4271: bad argument to internal function',)
bzr: ERROR: exceptions.SystemError: ../Objects/stringobject.c:4271: bad argument to internal function

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    attaching the crash file
        /.cache/crash/bzr-20100331184222-2913.crash
    and including a description of the problem.

    The crash file is plain text and you can inspect or edit it to remove
    private information.
**** entering debugger
> /usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py(291)_create_z_content_from_chunks()
-> compressed_chunks = map(compressor.compress, self._content_chunks)
(Pdb) pp locals()
{'compressor': <zlib.Compress object at 0x5e1ce70>,
 'self': <bzrlib.groupcompress.GroupCompressBlock object at 0x714fc10>}
(Pdb)

climpinho (djsme69) wrote :

I forgot to tell that before the crash the memory is all consumed by the bzr process and when there is no more memory the crash occurs

Chadarius (csutton-chadarius) wrote :

I started having this issue when I added two iso images to a bzr repo that were 4.7GB in total size.
1.7G 2010-05-24 16:30 KB3AIK_EN.iso
3.0G 2010-05-03 16:24 SW_DVD5_SA_Win_Ent_7_64BIT_English_Full_MLF_X15-70749.ISO

As soon as I move them out of the repo everything works just fine.

Martin Pool (mbp) wrote :

I wonder if this bug is to do with overflow and confusion between 32 and 64 bit int types.

For now I would not recommend adding ISOs to the bzr repository.

Martin Packman (gz) wrote :

This is a bug in the Python zlib module, which Python 2.6.6 should fix. However duplicate bug 723234 appears to be from on 2.6.6 so the upstream fix may be incomplete.

Changed in python:
status: Unknown → Fix Released
Jelmer Vernooij (jelmer) on 2017-11-09
tags: added: check-for-breezy
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.