| 2012-12-17 15:06:51 |
Pascual Abellan |
bug |
|
|
added bug |
| 2012-12-17 15:06:51 |
Pascual Abellan |
attachment added |
|
no-encryption-corruption-fix.patch https://bugs.launchpad.net/bugs/1091269/+attachment/3462143/+files/no-encryption-corruption-fix.patch |
|
| 2012-12-17 15:07:14 |
Pascual Abellan |
attachment added |
|
manual-ctrl-c-test.patch https://bugs.launchpad.net/duplicity/+bug/1091269/+attachment/3462144/+files/manual-ctrl-c-test.patch |
|
| 2013-01-02 14:02:45 |
Launchpad Janitor |
branch linked |
|
lp:duplicity |
|
| 2013-01-02 14:02:55 |
Kenneth Loafman |
duplicity: status |
New |
Fix Committed |
|
| 2013-01-02 14:03:00 |
Kenneth Loafman |
duplicity: importance |
Undecided |
Medium |
|
| 2013-01-04 19:25:45 |
Michael Terry |
bug task added |
|
duplicity (Ubuntu) |
|
| 2013-01-04 20:19:03 |
Ubuntu Foundations Team Bug Bot |
tags |
|
patch |
|
| 2013-01-04 20:19:10 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Review Team |
| 2013-01-07 03:05:38 |
Michael Terry |
duplicity (Ubuntu): importance |
Undecided |
Critical |
|
| 2013-01-07 03:05:38 |
Michael Terry |
duplicity (Ubuntu): status |
New |
Triaged |
|
| 2013-01-07 03:05:38 |
Michael Terry |
duplicity (Ubuntu): assignee |
|
Michael Terry (mterry) |
|
| 2013-01-07 03:06:44 |
Michael Terry |
bug |
|
|
added subscriber Michael Terry |
| 2013-01-07 03:10:33 |
Michael Terry |
nominated for series |
|
Ubuntu Precise |
|
| 2013-01-07 03:10:33 |
Michael Terry |
bug task added |
|
duplicity (Ubuntu Precise) |
|
| 2013-01-07 03:10:33 |
Michael Terry |
nominated for series |
|
Ubuntu Quantal |
|
| 2013-01-07 03:10:33 |
Michael Terry |
bug task added |
|
duplicity (Ubuntu Quantal) |
|
| 2013-01-07 14:48:31 |
Michael Terry |
branch linked |
|
lp:~mterry/duplicity/static-corruption |
|
| 2013-01-07 15:36:57 |
Michael Terry |
summary |
Data corruption when resuming with --no-encryption option |
Data corruption when resuming |
|
| 2013-01-07 16:04:51 |
Michael Terry |
description |
I have found a bug similar to #613244.
It seems that some files are corrupted when the backup is interrupted and resumed when using --no-encryption option.
The make-ctrl-c-test.sh test fails if I add the --no-encryption option to duplicity. Without this option it works fine.
I have added an option to make-ctrl-c-test.sh test to run it with or without encryption (patch attached).
I have seen this bug in the current duplicity quantal version (0.6.19-0ubuntu2) and in the last bazaar revision (897).
I also attached a patch that seems to fix this issue. I don't know anything about duplicity internals, so I don't know if this patch is correct or it will break something. Actually, I have ran the run-test script and it fails in test_GzipWriteFile test. But hopefully this can help you to fix the problem.
These are the final log lines of the make-ctrl-c-test.sh script, with and without encryption.
make-ctrl-c-test.sh with encryption:
...
Restoring backups...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:07:59 2012
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:08:13 2012
Diff between /lib and /tmp/restore1
Diff between /tmp/restore1 and /tmp/restore2
make-ctrl-c-test.sh with --no-encryption and without fix applied:
...
Restoring backups...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:08:55 2012
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:09:16 2012
Diff between /lib and /tmp/restore1
Diff between /tmp/restore1 and /tmp/restore2
Files /tmp/restore1/modules/3.5.0-19-generic/kernel/drivers/infiniband/hw/qib/ib_qib.ko and /tmp/restore2/modules/3.5.0-19-generic/kernel/drivers/infiniband/hw/qib/ib_qib.ko differ
make-ctrl-c-test.sh with --no-encryption and fix applied:
...
Restoring backups...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:19:43 2012
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:20:14 2012
Diff between /lib and /tmp/restore1
Diff between /tmp/restore1 and /tmp/restore2
The test_GzipWriteFile fails with this error:
======================================================================
FAIL: test_GzipWriteFile (__main__.GPGTest)
Test GzipWriteFile
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests/gpgtest.py", line 135, in test_GzipWriteFile
assert size - 64 * 1024 <= os.stat("testfiles/output/gzwrite.gz").st_size <= size + 64 * 1024
AssertionError
----------------------------------------------------------------------
Ran 8 tests in 1.326s
FAILED (failures=1)
Test failed |
[Impact]
It's very easy to accidentally corrupt the backup copy of a file when resuming an interrupted backup.
Specifically, there are several ways to end up backing up corrupted data:
A) If resuming after a volume that ended in a one-block file, we would skip the first block of the next file.
B) If resuming after a volume that ended in a multi-block file, we would skip the first block of the next file.
C) If resuming a non-encrypted backup after a volume that spanned a multi-block file, we would skip some data inside the file.
These are all very similar but have slightly different code fixes. Together they amount to "if you resume a backup, it's very likely you will have corrupted data somewhere". The only situation that doesn't is resuming in the middle of a file while using encryption.
[Test Case]
Running the restart test suite from my fix branch, which includes new tests for these cases:
bzr branch lp:~mterry/duplicity/static-corruption
cd static-corruption
./testing/run-tests
[Regression Potential]
This patch changes (1) how much data we read from source files when backing up and (2) how we pick where to resume a backup.
If a regression did appear because of this branch, I would expect it to be a similar data corruption problem (not starting in the right place or reading too much/too little from source files).
[Original Report]
I have found a bug similar to #613244.
It seems that some files are corrupted when the backup is interrupted and resumed when using --no-encryption option.
The make-ctrl-c-test.sh test fails if I add the --no-encryption option to duplicity. Without this option it works fine.
I have added an option to make-ctrl-c-test.sh test to run it with or without encryption (patch attached).
I have seen this bug in the current duplicity quantal version (0.6.19-0ubuntu2) and in the last bazaar revision (897).
I also attached a patch that seems to fix this issue. I don't know anything about duplicity internals, so I don't know if this patch is correct or it will break something. Actually, I have ran the run-test script and it fails in test_GzipWriteFile test. But hopefully this can help you to fix the problem.
These are the final log lines of the make-ctrl-c-test.sh script, with and without encryption.
make-ctrl-c-test.sh with encryption:
...
Restoring backups...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:07:59 2012
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:08:13 2012
Diff between /lib and /tmp/restore1
Diff between /tmp/restore1 and /tmp/restore2
make-ctrl-c-test.sh with --no-encryption and without fix applied:
...
Restoring backups...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:08:55 2012
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:09:16 2012
Diff between /lib and /tmp/restore1
Diff between /tmp/restore1 and /tmp/restore2
Files /tmp/restore1/modules/3.5.0-19-generic/kernel/drivers/infiniband/hw/qib/ib_qib.ko and /tmp/restore2/modules/3.5.0-19-generic/kernel/drivers/infiniband/hw/qib/ib_qib.ko differ
make-ctrl-c-test.sh with --no-encryption and fix applied:
...
Restoring backups...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:19:43 2012
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:20:14 2012
Diff between /lib and /tmp/restore1
Diff between /tmp/restore1 and /tmp/restore2
The test_GzipWriteFile fails with this error:
======================================================================
FAIL: test_GzipWriteFile (__main__.GPGTest)
Test GzipWriteFile
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests/gpgtest.py", line 135, in test_GzipWriteFile
assert size - 64 * 1024 <= os.stat("testfiles/output/gzwrite.gz").st_size <= size + 64 * 1024
AssertionError
----------------------------------------------------------------------
Ran 8 tests in 1.326s
FAILED (failures=1)
Test failed |
|
| 2013-01-07 16:04:55 |
Michael Terry |
duplicity: status |
Fix Committed |
Triaged |
|
| 2013-01-07 17:24:12 |
Launchpad Janitor |
duplicity (Ubuntu): status |
Triaged |
Fix Released |
|
| 2013-01-07 18:41:53 |
Michael Terry |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2013-01-07 18:42:17 |
Michael Terry |
removed subscriber Ubuntu Review Team |
|
|
|
| 2013-01-07 18:42:33 |
Michael Terry |
nominated for series |
|
Ubuntu Lucid |
|
| 2013-01-07 18:42:33 |
Michael Terry |
bug task added |
|
duplicity (Ubuntu Lucid) |
|
| 2013-01-07 18:42:33 |
Michael Terry |
nominated for series |
|
Ubuntu Oneiric |
|
| 2013-01-07 18:42:33 |
Michael Terry |
bug task added |
|
duplicity (Ubuntu Oneiric) |
|
| 2013-01-07 23:44:59 |
Launchpad Janitor |
duplicity (Ubuntu Lucid): status |
New |
Confirmed |
|
| 2013-01-07 23:44:59 |
Launchpad Janitor |
duplicity (Ubuntu Oneiric): status |
New |
Confirmed |
|
| 2013-01-07 23:44:59 |
Launchpad Janitor |
duplicity (Ubuntu Precise): status |
New |
Confirmed |
|
| 2013-01-07 23:44:59 |
Launchpad Janitor |
duplicity (Ubuntu Quantal): status |
New |
Confirmed |
|
| 2013-01-08 14:21:33 |
Michael Terry |
duplicity: status |
Triaged |
Fix Committed |
|
| 2013-01-11 02:02:01 |
Michael Terry |
attachment added |
|
auto-ctrl-c-test.sh https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/1091269/+attachment/3479775/+files/auto-ctrl-c-test.sh |
|
| 2013-01-11 02:04:36 |
Michael Terry |
description |
[Impact]
It's very easy to accidentally corrupt the backup copy of a file when resuming an interrupted backup.
Specifically, there are several ways to end up backing up corrupted data:
A) If resuming after a volume that ended in a one-block file, we would skip the first block of the next file.
B) If resuming after a volume that ended in a multi-block file, we would skip the first block of the next file.
C) If resuming a non-encrypted backup after a volume that spanned a multi-block file, we would skip some data inside the file.
These are all very similar but have slightly different code fixes. Together they amount to "if you resume a backup, it's very likely you will have corrupted data somewhere". The only situation that doesn't is resuming in the middle of a file while using encryption.
[Test Case]
Running the restart test suite from my fix branch, which includes new tests for these cases:
bzr branch lp:~mterry/duplicity/static-corruption
cd static-corruption
./testing/run-tests
[Regression Potential]
This patch changes (1) how much data we read from source files when backing up and (2) how we pick where to resume a backup.
If a regression did appear because of this branch, I would expect it to be a similar data corruption problem (not starting in the right place or reading too much/too little from source files).
[Original Report]
I have found a bug similar to #613244.
It seems that some files are corrupted when the backup is interrupted and resumed when using --no-encryption option.
The make-ctrl-c-test.sh test fails if I add the --no-encryption option to duplicity. Without this option it works fine.
I have added an option to make-ctrl-c-test.sh test to run it with or without encryption (patch attached).
I have seen this bug in the current duplicity quantal version (0.6.19-0ubuntu2) and in the last bazaar revision (897).
I also attached a patch that seems to fix this issue. I don't know anything about duplicity internals, so I don't know if this patch is correct or it will break something. Actually, I have ran the run-test script and it fails in test_GzipWriteFile test. But hopefully this can help you to fix the problem.
These are the final log lines of the make-ctrl-c-test.sh script, with and without encryption.
make-ctrl-c-test.sh with encryption:
...
Restoring backups...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:07:59 2012
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:08:13 2012
Diff between /lib and /tmp/restore1
Diff between /tmp/restore1 and /tmp/restore2
make-ctrl-c-test.sh with --no-encryption and without fix applied:
...
Restoring backups...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:08:55 2012
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:09:16 2012
Diff between /lib and /tmp/restore1
Diff between /tmp/restore1 and /tmp/restore2
Files /tmp/restore1/modules/3.5.0-19-generic/kernel/drivers/infiniband/hw/qib/ib_qib.ko and /tmp/restore2/modules/3.5.0-19-generic/kernel/drivers/infiniband/hw/qib/ib_qib.ko differ
make-ctrl-c-test.sh with --no-encryption and fix applied:
...
Restoring backups...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:19:43 2012
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:20:14 2012
Diff between /lib and /tmp/restore1
Diff between /tmp/restore1 and /tmp/restore2
The test_GzipWriteFile fails with this error:
======================================================================
FAIL: test_GzipWriteFile (__main__.GPGTest)
Test GzipWriteFile
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests/gpgtest.py", line 135, in test_GzipWriteFile
assert size - 64 * 1024 <= os.stat("testfiles/output/gzwrite.gz").st_size <= size + 64 * 1024
AssertionError
----------------------------------------------------------------------
Ran 8 tests in 1.326s
FAILED (failures=1)
Test failed |
[Impact]
It's very easy to accidentally corrupt the backup copy of a file when resuming an interrupted backup.
Specifically, there are several ways to end up backing up corrupted data:
A) If resuming after a volume that ended in a one-block file, we would skip the first block of the next file.
B) If resuming after a volume that ended in a multi-block file, we would skip the first block of the next file.
C) If resuming a non-encrypted backup after a volume that spanned a multi-block file, we would skip some data inside the file.
These are all very similar but have slightly different code fixes. Together they amount to "if you resume a backup, it's very likely you will have corrupted data somewhere". The only situation that doesn't is resuming in the middle of a file while using encryption.
[Test Case]
Download the script 'auto-ctrl-c-test.sh' from this bug, and run it:
sudo /bin/bash ./auto-ctrl-c-test.sh -s
It should end with "***** Diff between /lib and /tmp/restore" and no lines following if the bug is fixed. If the bug is still there, there will be several lines explaining the difference between the original files and the restored ones.
[Regression Potential]
This patch changes (1) how much data we read from source files when backing up and (2) how we pick where to resume a backup.
If a regression did appear because of this branch, I would expect it to be a similar data corruption problem (not starting in the right place or reading too much/too little from source files).
[Original Report]
I have found a bug similar to #613244.
It seems that some files are corrupted when the backup is interrupted and resumed when using --no-encryption option.
The make-ctrl-c-test.sh test fails if I add the --no-encryption option to duplicity. Without this option it works fine.
I have added an option to make-ctrl-c-test.sh test to run it with or without encryption (patch attached).
I have seen this bug in the current duplicity quantal version (0.6.19-0ubuntu2) and in the last bazaar revision (897).
I also attached a patch that seems to fix this issue. I don't know anything about duplicity internals, so I don't know if this patch is correct or it will break something. Actually, I have ran the run-test script and it fails in test_GzipWriteFile test. But hopefully this can help you to fix the problem.
These are the final log lines of the make-ctrl-c-test.sh script, with and without encryption.
make-ctrl-c-test.sh with encryption:
...
Restoring backups...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:07:59 2012
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:08:13 2012
Diff between /lib and /tmp/restore1
Diff between /tmp/restore1 and /tmp/restore2
make-ctrl-c-test.sh with --no-encryption and without fix applied:
...
Restoring backups...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:08:55 2012
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:09:16 2012
Diff between /lib and /tmp/restore1
Diff between /tmp/restore1 and /tmp/restore2
Files /tmp/restore1/modules/3.5.0-19-generic/kernel/drivers/infiniband/hw/qib/ib_qib.ko and /tmp/restore2/modules/3.5.0-19-generic/kernel/drivers/infiniband/hw/qib/ib_qib.ko differ
make-ctrl-c-test.sh with --no-encryption and fix applied:
...
Restoring backups...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:19:43 2012
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Dec 17 14:20:14 2012
Diff between /lib and /tmp/restore1
Diff between /tmp/restore1 and /tmp/restore2
The test_GzipWriteFile fails with this error:
======================================================================
FAIL: test_GzipWriteFile (__main__.GPGTest)
Test GzipWriteFile
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests/gpgtest.py", line 135, in test_GzipWriteFile
assert size - 64 * 1024 <= os.stat("testfiles/output/gzwrite.gz").st_size <= size + 64 * 1024
AssertionError
----------------------------------------------------------------------
Ran 8 tests in 1.326s
FAILED (failures=1)
Test failed |
|
| 2013-01-11 17:11:16 |
Brian Murray |
duplicity (Ubuntu Quantal): status |
Confirmed |
Fix Committed |
|
| 2013-01-11 17:11:22 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2013-01-11 17:11:30 |
Brian Murray |
tags |
patch |
patch verification-needed |
|
| 2013-01-11 18:27:19 |
Brian Murray |
duplicity (Ubuntu Precise): status |
Confirmed |
Fix Committed |
|
| 2013-01-11 20:45:04 |
Miklos Juhasz |
bug |
|
|
added subscriber Miklos Juhasz |
| 2013-01-11 20:46:32 |
Miklos Juhasz |
tags |
patch verification-needed |
patch verification-done |
|
| 2013-01-19 13:24:33 |
Colin Watson |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2013-01-19 13:25:13 |
Launchpad Janitor |
duplicity (Ubuntu Precise): status |
Fix Committed |
Fix Released |
|
| 2013-01-19 13:29:09 |
Colin Watson |
duplicity (Ubuntu Quantal): status |
Fix Committed |
Fix Released |
|
| 2013-01-23 15:29:00 |
Kenneth Loafman |
duplicity: status |
Fix Committed |
Fix Released |
|
| 2013-01-23 15:29:00 |
Kenneth Loafman |
duplicity: milestone |
|
0.6.21 |
|
| 2013-02-07 21:12:01 |
Brian Murray |
duplicity (Ubuntu Lucid): status |
Confirmed |
Fix Committed |
|
| 2013-02-07 21:12:07 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2013-02-07 21:12:10 |
Brian Murray |
tags |
patch verification-done |
patch |
|
| 2013-02-07 21:12:12 |
Brian Murray |
tags |
patch |
patch verification-needed |
|
| 2013-02-07 21:13:55 |
Brian Murray |
duplicity (Ubuntu Oneiric): status |
Confirmed |
Fix Committed |
|
| 2013-03-18 00:54:16 |
Michael Terry |
bug watch added |
|
https://bugzilla.redhat.com/show_bug.cgi?id=922576 |
|
| 2013-03-18 00:54:16 |
Michael Terry |
bug task added |
|
duplicity (Fedora) |
|
| 2013-05-13 17:24:40 |
Brian Murray |
tags |
patch verification-needed |
patch removal-candidate verification-needed |
|
| 2013-05-13 17:30:20 |
Brian Murray |
duplicity (Ubuntu Oneiric): status |
Fix Committed |
Won't Fix |
|
| 2013-06-07 07:33:39 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/duplicity |
|
| 2013-06-07 07:33:52 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/lucid-proposed/duplicity |
|
| 2013-06-07 07:33:54 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-branches/ubuntu/oneiric/duplicity/oneiric-proposed |
|
| 2013-08-28 12:31:00 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/precise-proposed/duplicity |
|
| 2013-08-28 12:31:24 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-branches/ubuntu/quantal/duplicity/quantal-proposed |
|
| 2013-08-28 12:31:59 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/raring/duplicity |
|
| 2013-08-28 12:32:01 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/raring-proposed/duplicity |
|
| 2013-11-18 23:35:39 |
Michael Terry |
tags |
patch removal-candidate verification-needed |
patch verification-done |
|
| 2013-11-19 23:00:51 |
Launchpad Janitor |
duplicity (Ubuntu Lucid): status |
Fix Committed |
Fix Released |
|
| 2017-10-28 09:38:02 |
Bug Watch Updater |
duplicity (Fedora): status |
Unknown |
Fix Released |
|
| 2017-10-28 09:38:02 |
Bug Watch Updater |
duplicity (Fedora): importance |
Unknown |
Undecided |
|
