ReadError: missing or bad subsequent header

Bug #642279 reported by Matt on 2010-09-18
30
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Duplicity
Undecided
Unassigned

Bug Description

Binary package hint: deja-dup

I have deja-dup configured to backup to an external drive. I had it unmounted and disconnected today, and during the automatic backup, deja-dup reported this error (and continues to report it):

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1239, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1232, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1214, in main
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 478, in incremental_backup
    bytes_written = dummy_backup(tarblock_iter)
  File "/usr/bin/duplicity", line 160, in dummy_backup
    while tarblock_iter.next():
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 505, in next
    result = self.process(self.input_iter.next(), size)
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 187, in get_delta_iter
    for new_path, sig_path in collated:
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 275, in collate2iters
    relem2 = riter2.next()
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 344, in combine_path_iters
    refresh_triple_list(triple_list)
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 330, in refresh_triple_list
    new_triple = get_triple(old_triple[1])
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 316, in get_triple
    path = path_iter_list[iter_index].next()
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 229, in sigtar2path_iter
    for tarinfo in tf:
  File "/usr/lib/python2.6/dist-packages/duplicity/tarfile.py", line 1211, in next
    tarinfo = self.tarfile.next()
  File "/usr/lib/python2.6/dist-packages/duplicity/tarfile.py", line 564, in next
    tarinfo = self._proc_gnulong(tarinfo, tarinfo.type)
  File "/usr/lib/python2.6/dist-packages/duplicity/tarfile.py", line 1111, in _proc_gnulong
    tarinfo = self._buftoinfo(buf)
  File "/usr/lib/python2.6/dist-packages/duplicity/tarfile.py", line 1066, in _buftoinfo
    tarinfo.mode = int(buf[100:107], 8)
ValueError: invalid literal for int() with base 8: '\x00\x00\x00\x00\x00\x00\x00'

If this is happening because the drive is unmounted, surely there can be a more user-friendly way to report the problem?

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: deja-dup 14.2-0ubuntu0.1
ProcVersionSignature: Ubuntu 2.6.32-24.43-generic 2.6.32.15+drm33.5
Uname: Linux 2.6.32-24-generic x86_64
NonfreeKernelModules: nvidia
Architecture: amd64
Date: Sat Sep 18 11:07:23 2010
EcryptfsInUse: Yes
ExecutablePath: /usr/bin/deja-dup
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release amd64 (20091027)
ProcEnviron:
 LANG=en_US.utf8
 SHELL=/bin/bash
SourcePackage: deja-dup
XsessionErrors:
 (polkit-gnome-authentication-agent-1:1879): GLib-CRITICAL **: g_once_init_leave: assertion `initialization_value != 0' failed
 (Do:1914): Wnck-CRITICAL **: wnck_set_client_type got called multiple times.
 (trackerd:1901): Tracker-CRITICAL **: tracker_processor_mount_point_added: assertion `TRACKER_IS_PROCESSOR (processor)' failed

Matt (slower-shamble) wrote :
Matt (slower-shamble) wrote :

Forgot to add, this is Ubuntu 10.04 x86_64.

Michael Terry (mterry) wrote :

Ken, does this look familiar?

Matt (slower-shamble) wrote :

I apologize, this is NOT configured to backup to an external drive. It is backing up to an SFTP location, which is and has been available all along. However, many of the paths that are explicitly excluded in the backup set are symlinks to the external hard drive that was unplugged. In any case, the drive is reconnected and the backup is still failing with the same error.

I have attached the (sanitized) output of "deja-dup --backup" and the two log files generated by duplicity.

Matt (slower-shamble) wrote :
Matt (slower-shamble) wrote :
Michael Terry (mterry) on 2011-06-24
Changed in duplicity:
importance: Undecided → High
importance: High → Undecided
Changed in deja-dup:
importance: Undecided → High
Michael Terry (mterry) on 2011-06-24
summary: - deja-dup crashes during backup
+ ValueError: invalid literal for int() with base 8:
+ '\x00\x00\x00\x00\x00\x00\x00'

Closing deja-dup side of this. I would bet good money that this was fixed with the recent tarfile update in duplicity, which will be in 0.6.16. I would be interested to see if that version fixes this for you.

Changed in deja-dup:
status: New → Invalid
Michael Terry (mterry) on 2011-10-06
Changed in deja-dup (Ubuntu):
status: New → Invalid
Michael Terry (mterry) wrote :

A person on the mailing list experienced this bug, then upgraded to 0.6.19 and had basically the same problem, but new stacktrace:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1404, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1397, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1379, in main
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 574, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 380, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 320, in GPGWriteFile
    data = block_iter.next(min(block_size, bytes_to_go)).data
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 507, in next
    result = self.process(self.input_iter.next(), size)
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 188, in get_delta_iter
    for new_path, sig_path in collated:
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 277, in collate2iters
    relem2 = riter2.next()
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 346, in combine_path_iters
    refresh_triple_list(triple_list)
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 332, in refresh_triple_list
    new_triple = get_triple(old_triple[1])
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 318, in get_triple
    path = path_iter_list[iter_index].next()
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 230, in sigtar2path_iter
    for tarinfo in tf:
  File "/usr/lib/python2.7/dist-packages/duplicity/tarfile.py", line 2472, in next
    tarinfo = self.tarfile.next()
  File "/usr/lib/python2.7/dist-packages/duplicity/tarfile.py", line 2345, in next
    raise ReadError(str(e))
ReadError: missing or bad subsequent header

summary: - ValueError: invalid literal for int() with base 8:
- '\x00\x00\x00\x00\x00\x00\x00'
+ ReadError: missing or bad subsequent header
Michael Terry (mterry) on 2012-09-19
no longer affects: deja-dup (Ubuntu)
no longer affects: deja-dup
Changed in duplicity:
status: New → Confirmed
carloslp (carloslp) wrote :

Hi.

Same problem on Debian Squeeze with duplicity=0.6.18-3~bpo60+1

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1404, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1397, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1379, in main
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 574, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 348, in write_multivol
    restart_position_iterator(tarblock_iter)
  File "/usr/bin/duplicity", line 222, in restart_position_iterator
    while tarblock_iter.next():
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 507, in next
    result = self.process(self.input_iter.next(), size)
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 188, in get_delta_iter
    for new_path, sig_path in collated:
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 277, in collate2iters
    relem2 = riter2.next()
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 346, in combine_path_iters
    refresh_triple_list(triple_list)
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 332, in refresh_triple_list
    new_triple = get_triple(old_triple[1])
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 318, in get_triple
    path = path_iter_list[iter_index].next()
  File "/usr/lib/python2.6/dist-packages/duplicity/diffdir.py", line 230, in sigtar2path_iter
    for tarinfo in tf:
  File "/usr/lib/python2.6/dist-packages/duplicity/tarfile.py", line 2472, in next
    tarinfo = self.tarfile.next()
  File "/usr/lib/python2.6/dist-packages/duplicity/tarfile.py", line 2345, in next
    raise ReadError(str(e))
ReadError: missing or bad subsequent header

I tried to delete the 2 last increment files and retry it but gives the same error

carloslp (carloslp) wrote :

A clarification to previous comment.

I was doing a backup to an external usb drive but the machine hanged out and the backup was not completed.

After reboot, when i ran duplicity it detects the last increment but fails to complete the backup with the previous error.

carloslp (carloslp) wrote :

Seems that the problem is gone if you delete the uncompleted increment and you starts a new increment from scratch

I ran duplicity cleanup, this deleted all the files from the uncompleted increment and then re-run duplicity to do a new increment backup. This worked.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers