[master] zlib "incorrect data check" error while decompressing should mention hardware error and the associated filename

[displayname]

I was pushing a nontrivial branch to a new branch on a USB disk... the source branch is a couple years old, has a few hundred revisions, and takes up about 800mb, so making a simple set of repro steps could be quite a challenge.

G:\>mkdir Development
G:\>cd Development
G:\Development>bzr init
Created a standalone tree (format: 2a)

G:\Development>c:

C:\Users\Public\Documents\Development>bzr push g:\Development

bzr: ERROR: zlib.error: Error -3 while decompressing: incorrect data check

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 842, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1037, in run_bzr
  File "bzrlib\commands.pyo", line 654, in run_argv_aliases
  File "bzrlib\builtins.pyo", line 1156, in run
  File "bzrlib\push.pyo", line 128, in _show_push_branch
  File "bzrlib\bzrdir.pyo", line 1293, in push_branch
  File "bzrlib\workingtree.pyo", line 2216, in update
  File "bzrlib\mutabletree.pyo", line 53, in tree_write_locked
  File "bzrlib\workingtree.pyo", line 2256, in _update_tree
  File "bzrlib\merge.pyo", line 1538, in merge_inner
  File "bzrlib\merge.pyo", line 508, in do_merge
  File "bzrlib\merge.pyo", line 480, in _do_merge_to
  File "bzrlib\merge.pyo", line 620, in do_merge
  File "bzrlib\merge.pyo", line 664, in _compute_transform
  File "bzrlib\merge.pyo", line 1166, in merge_contents
  File "bzrlib\transform.pyo", line 2430, in create_from_tree
  File "bzrlib\revisiontree.pyo", line 72, in get_file
  File "bzrlib\revisiontree.pyo", line 68, in get_file_text
  File "bzrlib\revisiontree.pyo", line 81, in iter_files_bytes
  File "bzrlib\repository.pyo", line 2192, in iter_files_bytes
  File "bzrlib\groupcompress.pyo", line 454, in get_bytes_as
  File "bzrlib\groupcompress.pyo", line 554, in _prepare_for_extract
  File "bzrlib\groupcompress.pyo", line 158, in _ensure_content
error: Error -3 while decompressing: incorrect data check

bzr 2.0.2 on python 2.5.4 (Windows-Vista-6.1.7600)
arguments: ['bzr', 'push', 'g:\\Development']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files (x86)\Bazaar\plugins\bzrtools [2.0.1]
  explorer C:\Program Files (x86)\Bazaar\plugins\explorer [0.8.3]
  launchpad C:\Program Files (x86)\Bazaar\plugins\launchpad [2.0.2]
  netrc_credential_store C:\Program Files (x86)\Bazaar\plugins\netrc_credential_store [2.0.2]
  qbzr C:\Program Files (x86)\Bazaar\plugins\qbzr [0.14.5]
  rebase C:\Program Files (x86)\Bazaar\plugins\rebase [0.5.4]
  svn C:\Program Files (x86)\Bazaar\plugins\svn [1.0.0]
  upload C:\Program Files (x86)\Bazaar\plugins\upload [1.0.0dev]
  xmloutput C:\Program Files (x86)\Bazaar\plugins\xmloutput [0.8.5]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.

[displayname (displayname)]

I tried copying all files from the source branch to the destination, then ran "bzr check" at the destination, and got a slightly different error. It might be helpful to name the file that was being processed when this happened...

Checking working tree at 'H:/Development'.
Checking branch at 'file:///H:/Development/'.
Checking repository at 'file:///H:/Development/'.
bzr: ERROR: zlib.error: Error -3 while decompressing data: invalid distance too far back

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 853, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1055, in run_bzr
  File "bzrlib\commands.pyo", line 661, in run_argv_aliases
  File "bzrlib\commands.pyo", line 665, in run_direct
  File "bzrlib\cleanup.pyo", line 122, in run_simple
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "bzrlib\builtins.pyo", line 3225, in run
  File "bzrlib\check.pyo", line 456, in check_dwim
  File "bzrlib\decorators.pyo", line 140, in read_locked
  File "bzrlib\repository.pyo", line 2777, in check
  File "bzrlib\repository.pyo", line 2781, in _check
  File "bzrlib\check.pyo", line 102, in check
  File "bzrlib\repository.pyo", line 1178, in _check_inventories
  File "bzrlib\repository.pyo", line 1229, in _do_check_inventories
  File "bzrlib\groupcompress.pyo", line 1413, in get_record_stream
  File "bzrlib\groupcompress.pyo", line 1518, in _get_remaining_record_stream
  File "bzrlib\groupcompress.pyo", line 2006, in get_build_details
  File "bzrlib\groupcompress.pyo", line 1941, in _get_entries
  File "bzrlib\index.pyo", line 1321, in iter_entries
  File "bzrlib\btree_index.pyo", line 1162, in iter_entries
  File "bzrlib\btree_index.pyo", line 1116, in _walk_through_internal_nodes
  File "bzrlib\btree_index.pyo", line 957, in _get_leaf_nodes
  File "bzrlib\btree_index.pyo", line 934, in _get_nodes
  File "bzrlib\btree_index.pyo", line 711, in _get_and_cache_nodes
  File "bzrlib\btree_index.pyo", line 1538, in _read_nodes
error: Error -3 while decompressing data: invalid distance too far back

bzr 2.1.1 on python 2.5.4 (Windows-Vista-6.1.7600)
arguments: ['bzr', 'check']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files (x86)\Bazaar\plugins\bzrtools [2.1.0]
  explorer C:\Program Files (x86)\Bazaar\plugins\explorer [1.0.1]
  launchpad C:\Program Files (x86)\Bazaar\plugins\launchpad [2.1.1]
  netrc_credential_store C:\Program Files (x86)\Bazaar\plugins\netrc_credential_store [2.1.1]
  news_merge C:\Program Files (x86)\Bazaar\plugins\news_merge [2.1.1]
  qbzr C:\Program Files (x86)\Bazaar\plugins\qbzr [0.18.4]
  rebase C:\Program Files (x86)\Bazaar\plugins\rebase [0.5.5]
  svn C:\Program Files (x86)\Bazaar\plugins\svn [1.0.2]
  upload C:\Program Files (x86)\Bazaar\plugins\upload [1.0.0dev]
  xmloutput C:\Program Files (x86)\Bazaar\plugins\xmloutput [0.8.7.dev]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including t...

[Robert Collins (lifeless)]

This almost certainly means you've had either a disk corruption error, or a machine crash right after doing a bzr command.

For the former, checkdisk sometimes finds and fixes things - we had a case of that recently on the bzr list.

For the latter problem, you may have the old pack files on disk still in obsolete_packs - if you do we might be able to manually move them back into place (but don't just do it - its going to need some python etc).

-Rob

[displayname (displayname)]

Thanks, Robert. This root cause was a defective USB drive. The corrupt file was in the destination branch, and the source branch is still completely intact (a huge relief).

If I could, I'd rename this bug to "zlib errors should include file name" as that would have helped me understand the problem much more quickly. I incorrectly assumed that the decompression error indicated a corrupt file in the source branch, and it wasn't until I tried to reformat the destination drive that I really figured it out.

[Kristjan (kristjan-kosic)]

Hi, i have the same problem and would like to identify the corrupt file. In what file should I look for and what python code can I change - just temporary so that filename would be included when error with decompressing happens...

[Martin Pool (mbp)]

On 27 July 2010 02:26, Kristjan <email address hidden> wrote:
> Hi, i have the same problem and would like to identify the corrupt file.
> In what file should I look for and what python code can I change - just
> temporary so that filename would be included when error with
> decompressing happens...
>

Kristjan, run 'md5sum *' in the packs directory - the one that does
not match its name is probably corrupt.

--
Martin

[Martin Pool (mbp)]

As Robert said, this is almost always a hardware error.

[Kristjan (kristjan-kosic)]

I agree with you Martin. I turned out that my RAM was corrupter. After changing RAM everything worked without problems.

[Vincent Ladeuil (vila)]

I think we should keep this bug open to provide a better error message.

Martin said: "As Robert said, this is almost always a hardware error."

AFAIK, we still haven't encountered a case where it *wasn't* an hardware error, so we may as well point that in the error message, if it turns out that we *can* produce such bogus data, then that will be another bug to fix.

[Martin Pool (mbp)]

high priority to give a better message because this is so commonly hit.