IOError: CRC check failed

Bug #676767 reported by Lukas on 2010-11-18
46
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Duplicity
Undecided
Unassigned
Déjà Dup
Critical
Unassigned

Bug Description

NOTE FOR PEOPLE EXPERIENCING THIS PROBLEM:

The instructions at http://live.gnome.org/DejaDup/Help/Restore/WorstCase seem to help people recover from this. Duplicity works, but Deja Dup doesn't.

Original report (edited for more brevity) below.

-----------------

Hi, I can't restore my backup, deja dup shows an error:

-----------------

DUPLICITY: ERROR 30 IOError
DUPLICITY: . Traceback (most recent call last):
DUPLICITY: . File "/usr/bin/duplicity", line 1257, in <module>
DUPLICITY: . with_tempdir(main)
DUPLICITY: . File "/usr/bin/duplicity", line 1250, in with_tempdir
DUPLICITY: . fn()
DUPLICITY: . File "/usr/bin/duplicity", line 1204, in main
DUPLICITY: . restore(col_stats)
DUPLICITY: . File "/usr/bin/duplicity", line 539, in restore
DUPLICITY: . restore_get_patched_rop_iter(col_stats)):
DUPLICITY: . File "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 522, in Write_ROPaths
DUPLICITY: . ITR( ropath.index, ropath )
DUPLICITY: . File "/usr/lib/python2.6/dist-packages/duplicity/lazy.py", line 335, in __call__
DUPLICITY: . last_branch.fast_process, args)
DUPLICITY: . File "/usr/lib/python2.6/dist-packages/duplicity/robust.py", line 37, in check_common_error
DUPLICITY: . return function(*args)
DUPLICITY: . File "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 575, in fast_process
DUPLICITY: . ropath.copy( self.base_path.new_index( index ) )
DUPLICITY: . File "/usr/lib/python2.6/dist-packages/duplicity/path.py", line 416, in copy
DUPLICITY: . other.writefileobj(self.open("rb"))
DUPLICITY: . File "/usr/lib/python2.6/dist-packages/duplicity/path.py", line 591, in writefileobj
DUPLICITY: . buf = fin.read(_copy_blocksize)
DUPLICITY: . File "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 200, in read
DUPLICITY: . if not self.addtobuffer():
DUPLICITY: . File "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 221, in addtobuffer
DUPLICITY: . self.buffer += fp.read()
DUPLICITY: . File "/usr/lib/python2.6/dist-packages/duplicity/tarfile.py", line 1338, in _readnormal
DUPLICITY: . return self.fileobj.read(bytestoread)
DUPLICITY: . File "/usr/lib/python2.6/dist-packages/duplicity/dup_temp.py", line 204, in read
DUPLICITY: . return self.fileobj.read(length)
DUPLICITY: . File "/usr/lib/python2.6/gzip.py", line 219, in read
DUPLICITY: . self._read(readsize)
DUPLICITY: . File "/usr/lib/python2.6/gzip.py", line 284, in _read
DUPLICITY: . self._read_eof()
DUPLICITY: . File "/usr/lib/python2.6/gzip.py", line 304, in _read_eof
DUPLICITY: . hex(self.crc)))
DUPLICITY: . IOError: CRC check failed 0xbfacd5c8L != 0xca281a1cL
DUPLICITY: .

** (deja-dup:4599): DEBUG: DuplicityInstance.vala:553: duplicity (4613) exited with value 30

---------------------

Version of deja-dup and duplicity

deja-dup 16.0-0ubuntu1
duplicity 0.6.10-0ubuntu1

---------------

System: Ubuntu 10.04.1 LTS

I backed up the Data using Ubuntu 10.04 as well. Created the backup on an external harddrive. Formated my harddisk today and reinstalled Ubuntu 10.04. Then I tried to restore the data.

Hope you can help me.

Lukas (l-niemeyer) wrote :
Michael Terry (mterry) wrote :

That looks like a corrupted backup file. Can you restore from a previous backup date?

Changed in deja-dup:
status: New → Incomplete
Lukas (l-niemeyer) wrote :

no, it's the only backup i have. did it minutes before formatting the complete internal HD. but i saved the most important stuff in the cloud as well - only my adresses and firefox bookmarks are gone...

is there any chance to repair the corrupted file?

Michael Terry (mterry) wrote :

I'm not sure how. Ken, do you know?

Download full text (9.2 KiB)

The gzip portion of the file is corrupted somehow and I'm not sure how to
recover the contents. It may be possible to manually go into the gzip file
after decryption and run some sort of gzip-fix on it, but I don't see
anything in the repositories to suggest that such a critter exists.

Before you do anything, copy the backup itself to a work area and don't
touch the original.

You'll need to run with verbosity high to find out which file is the
culprit.

On Mon, Nov 29, 2010 at 7:23 AM, Michael Terry
<email address hidden>wrote:

> I'm not sure how. Ken, do you know?
>
> --
> IOError: CRC check failed
> https://bugs.launchpad.net/bugs/676767
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Déjà Dup: Incomplete
>
> Bug description:
> Hi,
> I can't restore my backup, deja dup shows an error:
>
> -----------------
>
> /tmp/deja-dup.log
>
> DUPLICITY: INFO 1
> DUPLICITY: . Making directory /home/lukas/.jdownloader/libs
>
> DUPLICITY: INFO 7 'home/lukas/.jdownloader/libs/BrowserLauncher2.jar' reg
> DUPLICITY: . Writing home/lukas/.jdownloader/libs/BrowserLauncher2.jar of
> type reg
>
> DUPLICITY: INFO 7 'home/lukas/.jdownloader/libs/Filters.jar' reg
> DUPLICITY: . Writing home/lukas/.jdownloader/libs/Filters.jar of type reg
>
> DUPLICITY: INFO 7 'home/lukas/.jdownloader/libs/JAntiCaptcha.jar' reg
> DUPLICITY: . Writing home/lukas/.jdownloader/libs/JAntiCaptcha.jar of type
> reg
>
> DUPLICITY: INFO 7 'home/lukas/.jdownloader/libs/JDGUI.jar' reg
> DUPLICITY: . Writing home/lukas/.jdownloader/libs/JDGUI.jar of type reg
>
> DUPLICITY: INFO 7 'home/lukas/.jdownloader/libs/JDHttp.jar' reg
> DUPLICITY: . Writing home/lukas/.jdownloader/libs/JDHttp.jar of type reg
>
> DUPLICITY: INFO 7 'home/lukas/.jdownloader/libs/JDUtils.jar' reg
> DUPLICITY: . Writing home/lukas/.jdownloader/libs/JDUtils.jar of type reg
>
> DUPLICITY: INFO 7 'home/lukas/.jdownloader/libs/JUnique.jar' reg
> DUPLICITY: . Writing home/lukas/.jdownloader/libs/JUnique.jar of type reg
>
> DUPLICITY: INFO 7 'home/lukas/.jdownloader/libs/balloontip.jar' reg
> DUPLICITY: . Writing home/lukas/.jdownloader/libs/balloontip.jar of type
> reg
>
> DUPLICITY: INFO 7 'home/lukas/.jdownloader/libs/clink170.jar' reg
> DUPLICITY: . Writing home/lukas/.jdownloader/libs/clink170.jar of type reg
>
> DUPLICITY: DEBUG 1
> DUPLICITY: . Removing still remembered temporary file
> /tmp/duplicity-_y22bi-tempdir/mkstemp-_ZjUB0-1
>
> DUPLICITY: DEBUG 1
> DUPLICITY: . Removing still remembered temporary file
> /tmp/duplicity-_y22bi-tempdir/mktemp-wmFejx-6
>
> DUPLICITY: ERROR 30 IOError
> DUPLICITY: . Traceback (most recent call last):
> DUPLICITY: . File "/usr/bin/duplicity", line 1257, in <module>
> DUPLICITY: . with_tempdir(main)
> DUPLICITY: . File "/usr/bin/duplicity", line 1250, in with_tempdir
> DUPLICITY: . fn()
> DUPLICITY: . File "/usr/bin/duplicity", line 1204, in main
> DUPLICITY: . restore(col_stats)
> DUPLICITY: . File "/usr/bin/duplicity", line 539, in restore
> DUPLICITY: . restore_get_patched_rop_iter(col_stats)):
> DUPLICITY: . File
> "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 522, in
> Write_R...

Read more...

Michael Terry (mterry) wrote :

Someone else reported a similar error during a backup (dup bug 725117). Marking confirmed, but still not sure how the files got in this corrupted state.

Changed in deja-dup:
status: Incomplete → Confirmed
Michael Terry (mterry) on 2011-05-15
description: updated
Changed in deja-dup:
importance: Undecided → High
Michael Terry (mterry) on 2011-05-16
tags: added: restore
Michael Terry (mterry) on 2011-05-16
Changed in deja-dup:
importance: High → Critical
Michael Terry (mterry) wrote :

I still have been unable to reproduce. If anyone that hits this (or has hit it) still has a backup set that reproduces it, please contact me if your set can be shared. I'd also be interested in the contents of ~/.cache/deja-dup. Thanks.

Stephen (stevepetmonkey) wrote :
Download full text (4.3 KiB)

I don't think this will help but looking through the restored files a few
weeks after I noticed a couple had restored and had no name, logo or
properties, they are just files??? I really don't know what they were
originally, but chances are they were from an old windows program. I do
still have the backup set but its nearly 200gb.

On Wed, Jun 1, 2011 at 4:47 AM, Michael Terry
<email address hidden>wrote:

> I still have been unable to reproduce. If anyone that hits this (or has
> hit it) still has a backup set that reproduces it, please contact me if
> your set can be shared. I'd also be interested in the contents of
> ~/.cache/deja-dup. Thanks.
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/676767
>
> Title:
> IOError: CRC check failed
>
> Status in Déjà Dup Backup Tool:
> Confirmed
>
> Bug description:
> NOTE FOR PEOPLE EXPERIENCING THIS PROBLEM:
>
> The instructions at
> http://live.gnome.org/DejaDup/Help/Restore/WorstCase seem to help
> people recover from this. Duplicity works, but Deja Dup doesn't.
>
> Original report (edited for more brevity) below.
>
> -----------------
>
> Hi, I can't restore my backup, deja dup shows an error:
>
> -----------------
>
> DUPLICITY: ERROR 30 IOError
> DUPLICITY: . Traceback (most recent call last):
> DUPLICITY: . File "/usr/bin/duplicity", line 1257, in <module>
> DUPLICITY: . with_tempdir(main)
> DUPLICITY: . File "/usr/bin/duplicity", line 1250, in with_tempdir
> DUPLICITY: . fn()
> DUPLICITY: . File "/usr/bin/duplicity", line 1204, in main
> DUPLICITY: . restore(col_stats)
> DUPLICITY: . File "/usr/bin/duplicity", line 539, in restore
> DUPLICITY: . restore_get_patched_rop_iter(col_stats)):
> DUPLICITY: . File
> "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 522, in
> Write_ROPaths
> DUPLICITY: . ITR( ropath.index, ropath )
> DUPLICITY: . File "/usr/lib/python2.6/dist-packages/duplicity/lazy.py",
> line 335, in __call__
> DUPLICITY: . last_branch.fast_process, args)
> DUPLICITY: . File
> "/usr/lib/python2.6/dist-packages/duplicity/robust.py", line 37, in
> check_common_error
> DUPLICITY: . return function(*args)
> DUPLICITY: . File
> "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 575, in
> fast_process
> DUPLICITY: . ropath.copy( self.base_path.new_index( index ) )
> DUPLICITY: . File "/usr/lib/python2.6/dist-packages/duplicity/path.py",
> line 416, in copy
> DUPLICITY: . other.writefileobj(self.open("rb"))
> DUPLICITY: . File "/usr/lib/python2.6/dist-packages/duplicity/path.py",
> line 591, in writefileobj
> DUPLICITY: . buf = fin.read(_copy_blocksize)
> DUPLICITY: . File
> "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 200, in read
> DUPLICITY: . if not self.addtobuffer():
> DUPLICITY: . File
> "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 221, in
> addtobuffer
> DUPLICITY: . self.buffer += fp.read()
> DUPLICITY: . File
> "/usr/lib/python2.6/dist-packages/duplicity/tarfile.py", line 1338, in
> _readnormal
> DUPLICITY: . return...

Read more...

Michael Terry (mterry) wrote :

Note that rdiff-backup apparently has hit this issue enough to have an FAQ entry about it: http://www.nongnu.org/rdiff-backup/FAQ.html#crc

I'm very confused though that this affects just Deja Dup restores, not duplicity ones. Perhaps the cache directory is causing problems (duplicity and deja-dup use different cache locations).

For anyone that still has a backup set that reliably hits this bug, try the following:

mv ~/.cache/deja-dup ~/.cache/deja-dup.old

And try restoring again. I'm curious if that gets around the problem.

Tom Wright (twright-tdw) wrote :

@Michael Terry: I have seen this issue but deleting the cache as mentioned seems to fix it so it would appear it was the cache which was corrupted not the actual backups. Wouldn't just automatically wiping the cache in such cases be the best solution?

Michael Terry (mterry) wrote :

Excellent, then. I've added code to Deja Dup to clean the cache behind the scenes if we encounter this error. I'm going to open a duplicity task for the underlying problem though.

Changed in deja-dup:
milestone: none → 19.3
status: Confirmed → Fix Committed
Michael Terry (mterry) on 2011-06-24
Changed in deja-dup:
status: Fix Committed → Fix Released
Matej Kovacic (matej-kovacic) wrote :

I have the same error with DejaDup in Ubuntu 16.04.

I am trying to run a backup (not restore!), and got this error:

Ni mogoče brati /tmp/duplicity-gUZKem-tempdir/mktemp-jecFmU-5: (<type 'exceptions.IOError'>, IOError('CRC check failed 0x5528869f != 0x2aeb4411L',), <traceback object at 0x7fd5292a9830>)

It is in slovenian, but it means:
Cannot read /tmp/duplicity-gUZKem-tempdir/mktemp-jecFmU-5: (<type 'exceptions.IOError'>, IOError('CRC check failed 0x5528869f != 0x2aeb4411L',), <traceback object at 0x7fd5292a9830>)

I tried to delete .cache/deja-dup, but the error is still there.

I am doing a backup to a remote server via ssh.

Roberto Colombari (otrebor79) wrote :

Same error of Matej Kovacic (matej-kovacic) for me with DejaDup in Ubuntu 16.04.

"Impossibile leggere /tmp/duplicity-3h4aBK-tempdir/mktemp-fcwj6H-621: (<type 'exceptions.IOError'>, IOError('CRC check failed 0x69b0855d != 0x1fb21cc4L',), <traceback object at 0x7f3d6d2b86c8>)"

Clean .cache/deja-dup doesn't resolve.

Jonathan (desertshadow) wrote :

I get this error too. `\rm -r ~/.cache/duplicity` did not resolve it.

```
duplicity restore file://psm-images/ uploads
Synchronizing remote metadata to local cache...
Copying duplicity-full-signatures.20170826T023715Z.sigtar.gz to local cache.
Copying duplicity-full-signatures.20170826T024126Z.sigtar.gz to local cache.
Copying duplicity-full.20170826T023715Z.manifest to local cache.
Copying duplicity-full.20170826T024126Z.manifest to local cache.
Copying duplicity-new-signatures.20170826T024126Z.to.20170828T221324Z.sigtar.gz to local cache.
Failed to read /tmp/duplicity-nCtyW6-tempdir/mktemp-_AsaJn-10: (<type 'exceptions.IOError'>, IOError('CRC check failed 0x8833f151 != 0x465163a2L',), <traceback object at 0x7fcc82046518>)
```

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Related questions

Bug attachments