Possible data loss when restarting in the middle of a deleted file

[Michael Terry]

This was recently fixed in duplicity trunk. But I'm filing a bug for paperwork purposes and for Ubuntu SRUs.

[Impact]

When restarting a backup, duplicity may accidentally skip the first 65k chunk of one of the source files. This means that when it is restored, it will be incomplete/corrupted, resulting in data loss.

Note that symptoms of this bug are diverse and may look like:
assert len( patch_seq ) == 1, len( patch_seq )
or
assert first.difftype != "diff", patch_seq
or
librsync error 103 while in patch cycle

These are all different results from this root cause. In all cases, it is because chunks of files are missing in your backup. There is nothing we can do to get those chunks back. But a related patch in duplicity 0.6.23 will let you restore the rest of your files (instead of just crashing halfway through a restore like 0.6.22 does). So try duplicity 0.6.23 to get a little progress from your broken backups. And use duplicity 0.6.23 for all future backups.

[Test Case]

Download and install the 'test1.sh' file attached to this bug report, and run it like 'sh test1.sh'. If it prints the following line, the bug is present:
Binary files /tmp/source/newfile and /tmp/restore/newfile differ

Or, follow these manual steps:
mkdir /tmp/source
dd if=/dev/urandom of=/tmp/source/bigfile bs=1024 count=5000
# This next command will intentionally fail after the second volume!
duplicity full /tmp/source file:///tmp/backup --vol 1 --fail 2 --no-encryption
mv /tmp/source/bigfile /tmp/source/newfile
duplicity full /tmp/source file:///tmp/backup --no-encryption
duplicity restore file:///tmp/backup /tmp/restore --no-encryption
# This next line will say the files differ if the bug is present
diff /tmp/source/newfile /tmp/restore/newfile

[Regression Potential]

It's a relatively small patch, only affecting the specific case of restarting a backup when the file we were in the middle of is no longer there. I'd say minor regression potential.

[Launchpad Janitor (janitor)]

This bug was fixed in the package duplicity - 0.6.22-1ubuntu3

---------------
duplicity (0.6.22-1ubuntu3) trusty; urgency=low

  * debian/patches/04-dont-skip-first-chunk-on-restart.patch:
    - When restarting a backup, if the file we were in the middle of
      backing up is now deleted, don't skip the first 65k chunk of the
      next file. Patch backported from upstream trunk. LP: #1252484
 -- Michael Terry <email address hidden> Mon, 18 Nov 2013 16:51:52 -0500

[Stéphane Graber (stgraber)]

Hello Michael, or anyone else affected,

Accepted duplicity into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/duplicity/0.6.21-0ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Miklos Juhasz (mjuhasz)]

With the version in proposed the testcase in the description passed. Tested on Ubuntu Saucy (13.10) using duplicity 0.6.21-0ubuntu4.1

[Michael Terry (mterry)]

Thanks, Miklos!

[Brian Murray (brian-murray)]

Hello Michael, or anyone else affected,

Accepted duplicity into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/duplicity/0.6.21-0ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Michael, or anyone else affected,

Accepted duplicity into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/duplicity/0.6.19-0ubuntu2.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Michael, or anyone else affected,

Accepted duplicity into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/duplicity/0.6.18-0ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Michael, or anyone else affected,

Accepted duplicity into lucid-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/duplicity/0.6.08b-0ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package duplicity - 0.6.21-0ubuntu4.1

---------------
duplicity (0.6.21-0ubuntu4.1) saucy; urgency=low

  * debian/patches/03-dont-skip-first-chunk-on-restart.dpatch:
    - When restarting a backup, if the file we were in the middle of
      backing up is now deleted, don't skip the first 65k chunk of the
      next file. Patch backported from upstream trunk. LP: #1252484
 -- Michael Terry <email address hidden> Mon, 18 Nov 2013 18:40:41 -0500

[Michael Terry (mterry)]

I should note for anyone experiencing any of the symptoms of this bug when restoring [1], this fix does not let you get back the 65k chunk of data that this bug lost. But duplicity trunk contains some additional fixes that should let you restore the rest of your data. That is, duplicity trunk no longer throws an exception when encountering the symptoms of this bug. It will continue to restore the data not affected by this bug.

So while you can't experience full relief, duplicity trunk will let you recover most things.

[1] It can cause various exceptions depending on how/when this bug happened; see the duplicate bugs.

[Lennart Jütte (ljuette)]

I believe i'm still experiencing the problems.

I have a set of backups from a machine which started with Ubuntu 12.04 and was upgraded a few weeks ago to 13.10. The backups were created via deja-dup. I'm trying to restore on a Machine Running Linux Mint ( ~Ubuntu 13.10) with duplicity 0.6.21-0ubuntu4.1.

Here's what i get when i try to restore my backup manually (duplicity, not deja-dup):
python: ERROR: (rs_file_copy_cb) unexpected eof on fd122
python: ERROR: (rs_job_complete) patch job failed: unexpected end of input
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1414, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1407, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1341, in main
    restore(col_stats)
  File "/usr/bin/duplicity", line 635, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 526, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 499, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 479, in patch_seq2ropath
    misc.copyfileobj( current_file, tempfp )
  File "/usr/lib/python2.7/dist-packages/duplicity/misc.py", line 166, in copyfileobj
    buf = infp.read(blocksize)
  File "/usr/lib/python2.7/dist-packages/duplicity/librsync.py", line 80, in read
    self._add_to_outbuf_once()
  File "/usr/lib/python2.7/dist-packages/duplicity/librsync.py", line 94, in _add_to_outbuf_once
    raise librsyncError(str(e))
librsyncError: librsync error 103 while in patch cycle

deja-dup detects that there's a problem and bails out. If i try to restore via duplicity, i get this message and the application hangs.
A ps -ef ef shows that there's the duplicity process, which deja-dup started, is still lingering around. My current process is idling.

Did i miss something?

[Michael Terry (mterry)]

Lennart, see my comment #11. This fix/SRU is for the root cause when making a backup, preventing the data loss in the first place.

But if you are trying to restore a backup affected by the bug (as you are), the traceback you see when restoring is worked-around by a separate fix in duplicity trunk. You still will not be able to correctly restore the file missing its 65k chunk, but duplicity trunk does prevent the restore from bailing out.

[Dan Gillmor (dan-gillmor)]

I just tried a backup and restore, encrypting the backup. Failed again
on the restore, in exactly the same way as far as I can tell.

At this point I simply use Grsync to run a complete backup to an
encrypted drive. It works well enough.

On 12/15/2013 04:56 PM, Michael Terry wrote:
> Lennart, see my comment #11. This fix/SRU is for the root cause when
> making a backup, preventing the data loss in the first place.
>
> But if you are trying to restore a backup affected by the bug (as you
> are), the traceback you see when restoring is worked-around by a
> separate fix in duplicity trunk. You still will not be able to
> correctly restore the file missing its 65k chunk, but duplicity trunk
> does prevent the restore from bailing out.
>

[Michael Terry (mterry)]

Dan, simply backing up with a fixed duplicity will not retroactively fix the bug. That is, if you have a backup chain that is already affected by the bug, adding more incremental backups to the chain won't be able to recover the previous missing 65k chunk. We're just ensuring that future backups won't be affected.

[Dan Gillmor (dan-gillmor)]

I tried this with a brand-new backup, not an old one.

Dan

On 12/17/2013 11:53 AM, Michael Terry wrote:
> Dan, simply backing up with a fixed duplicity will not retroactively fix
> the bug. That is, if you have a backup chain that is already affected
> by the bug, adding more incremental backups to the chain won't be able
> to recover the previous missing 65k chunk. We're just ensuring that
> future backups won't be affected.
>

[Michael Terry (mterry)]

Dan, if you are getting an error using the patched version of duplicity, can you please file a new bug with the logs and details the bug form asks for? It may be this bug, it may be something else. I'll triage separately.

[Dan Gillmor (dan-gillmor)]

OK, I'll do this when I get home from a trip.

Upside is I'm learning a lot about how rsync works!

Dan

On 12/18/2013 09:45 AM, Michael Terry wrote:
> Dan, if you are getting an error using the patched version of duplicity,
> can you please file a new bug with the logs and details the bug form
> asks for? It may be this bug, it may be something else. I'll triage
> separately.
>

[Michael Terry (mterry)]

It's been a while, so I've verified for 10.04 myself.

[Michael Terry (mterry)]

And same for 12.04.

[Launchpad Janitor (janitor)]

This bug was fixed in the package duplicity - 0.6.08b-0ubuntu2.3

---------------
duplicity (0.6.08b-0ubuntu2.3) lucid; urgency=low

  * debian/patches/08-dont-skip-first-chunk-on-restart.dpatch:
    - When restarting a backup, if the file we were in the middle of
      backing up is now deleted, don't skip the first 65k chunk of the
      next file. Patch backported from upstream trunk. LP: #1252484
 -- Michael Terry <email address hidden> Tue, 19 Nov 2013 10:58:49 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package duplicity - 0.6.18-0ubuntu3.4

---------------
duplicity (0.6.18-0ubuntu3.4) precise; urgency=low

  * debian/patches/11-dont-skip-first-chunk-on-restart.dpatch:
    - When restarting a backup, if the file we were in the middle of
      backing up is now deleted, don't skip the first 65k chunk of the
      next file. Patch backported from upstream trunk. LP: #1252484
 -- Michael Terry <email address hidden> Tue, 19 Nov 2013 10:16:52 -0500