duplicity does not create PAR2 archives when '--par2-options' is used, and does not warn about failure
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Duplicity |
Fix Released
|
Medium
|
Unassigned |
Bug Description
Duplicity appears to fail to create any PAR2 archives if the user specifies an option like increasing RAM use to speed PAR2 up; it also appears to fail to warn about it, and exits normally without reporting the non-creation of PAR2 archives.
---
I installed a new version of duplicity today to get PAR2 & Backblaze support, the `ubuntu2` .deb:
$ uname -a
Linux craft 4.2.0-42-generic #49-Ubuntu SMP Tue Jun 28 21:26:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ duplicity --version
duplicity 0.7.06
Successful run with default PAR2 options:
$ rm /media/gwern/My\ Book/duplicity2/*; duplicity full --use-agent --sign-key=CBE7F55C --encrypt-
Using archive dir: /home/gwern/
Using backup name: 9c8ed3c601f6078
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Main action: full
===
duplicity 0.7.06 (December 07, 2015)
Args: /usr/bin/duplicity full --use-agent --sign-key=CBE7F55C --encrypt-
Linux craft 4.2.0-42-generic #49-Ubuntu SMP Tue Jun 28 21:26:26 UTC 2016 x86_64 x86_64
/usr/
[GCC 5.2.1 20151010]
===
Using temporary directory /tmp/duplicity-
Registering (mkstemp) temporary file /tmp/duplicity-
Temp has 182325923840 available, backup will use approx 1363148800.
Synchronizing remote metadata to local cache...
Deleting local /home/gwern/
Deleting local /home/gwern/
Deleting local /home/gwern/
Deleting local /home/gwern/
0 files exist on backend
4 files exist in cache
Extracting backup chains from list of files: []
Warning, found signatures but no corresponding backup files
Last full backup date: none
Collection Status
---
Connecting with backend: BackendWrapper
Archive dir: /home/gwern/
Found 0 secondary backup chains.
No backup chains with active signatures found
No orphaned or incomplete backup sets found.
Using temporary directory /home/gwern/
Registering (mktemp) temporary file /home/gwern/
Using temporary directory /home/gwern/
Registering (mktemp) temporary file /home/gwern/
AsyncScheduler: instantiating at concurrency 0
Registering (mktemp) temporary file /tmp/duplicity-
Selecting /home/gwern/dnm
Comparing . and None
Getting delta of (. dir) and None
A .
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Comparing 3-16-cr-00109-SRU and None
Getting delta of (3-16-cr-00109-SRU dir) and None
A 3-16-cr-00109-SRU
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Comparing 3-16-cr-
Getting delta of (3-16-cr-
A 3-16-cr-
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Comparing 3-16-cr-
Getting delta of (3-16-cr-
A 3-16-cr-
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Comparing 3-16-cr-
Getting delta of (3-16-cr-
A 3-16-cr-
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Comparing 3-16-cr-
Getting delta of (3-16-cr-
A 3-16-cr-
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Comparing 3-16-cr-
Getting delta of (3-16-cr-
A 3-16-cr-
Removing still remembered temporary file /home/gwern/
Removing still remembered temporary file /home/gwern/
AsyncScheduler: running task synchronously (asynchronicity disabled)
Writing duplicity-
Making directory /tmp/duplicity-
Create Par2 recovery files
Deleting /tmp/duplicity-
Deleting tree /tmp/duplicity-
Selecting /tmp/duplicity-
Selection: examining path /tmp/duplicity-
Selection: + no selection functions found. Including
Selecting /tmp/duplicity-
Deleting /tmp/duplicity-
Selection: examining path /tmp/duplicity-
Selection: + no selection functions found. Including
Selecting /tmp/duplicity-
Deleting /tmp/duplicity-
Deleting /tmp/duplicity-
Deleting /tmp/duplicity-
Forgetting temporary file /tmp/duplicity-
AsyncScheduler: task completed successfully
Processed volume 1
Making directory /home/gwern/
Create Par2 recovery files
Deleting /home/gwern/
Deleting tree /home/gwern/
Selecting /home/gwern/
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Deleting /home/gwern/
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Deleting /home/gwern/
Deleting /home/gwern/
Making directory /home/gwern/
Create Par2 recovery files
Deleting /home/gwern/
Deleting tree /home/gwern/
Selecting /home/gwern/
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Deleting /home/gwern/
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Deleting /home/gwern/
Deleting /home/gwern/
1 file exists on backend
8 files exist in cache
Extracting backup chains from list of files: [u'duplicity-
File duplicity-
Warning, found incomplete backup sets, probably left from aborted session
--------------[ Backup Statistics ]--------------
StartTime 1479257802.19 (Tue Nov 15 19:56:42 2016)
EndTime 1479257802.76 (Tue Nov 15 19:56:42 2016)
ElapsedTime 0.57 (0.57 seconds)
SourceFiles 7
SourceFileSize 11078834 (10.6 MB)
NewFiles 7
NewFileSize 11078834 (10.6 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDelt
DeltaEntries 7
RawDeltaSize 11070642 (10.6 MB)
TotalDestin
Errors 0
---
Releasing lockfile /home/gwern/
Removing still remembered temporary file /tmp/duplicity-
Loading "duplicity-
Loaded 4 new packets
Loading "duplicity-
Loaded 200 new packets including 200 recovery blocks
There are 1 recoverable files and 0 other files.
The block size used was 4288 bytes.
There are a total of 2000 data blocks.
The total size of the data files is 8573459 bytes.
Verifying source files:
Target: "duplicity-
All files are correct, repair is not required.
duplicity-
duplicity-
duplicity-
Should I try tweaking the redundancy, it works fine with `--par2-redundancy 5` etc. So that particular PAR2 Duplicity option is working.
It always fails when I ask Duplicity to pass onto PAR2 an option, the -m option which increases RAM use by PAR2:
-m<n> Memory (in MB) to use
Due to the space-time tradeoff, PAR2 runs *much* faster (as in, several times faster) if you give it a more reasonable amount of RAM commensurate with modern standards, like 2GB. I always use it because it makes such a difference and I always have a few GB of RAM to spare. (It might not be a bad idea at all to have duplicity call PAR2 with a changed default....) However when I try:
$ rm /media/gwern/My\ Book/duplicity2/*; duplicity full --use-agent --sign-key=CBE7F55C --encrypt-
Using archive dir: /home/gwern/
Using backup name: 9c8ed3c601f6078
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Main action: full
===
duplicity 0.7.06 (December 07, 2015)
Args: /usr/bin/duplicity full --use-agent --sign-key=CBE7F55C --encrypt-
Linux craft 4.2.0-42-generic #49-Ubuntu SMP Tue Jun 28 21:26:26 UTC 2016 x86_64 x86_64
/usr/
[GCC 5.2.1 20151010]
===
Using temporary directory /tmp/duplicity-
Registering (mkstemp) temporary file /tmp/duplicity-
Temp has 182325932032 available, backup will use approx 1363148800.
Synchronizing remote metadata to local cache...
Deleting local /home/gwern/
Deleting local /home/gwern/
Deleting local /home/gwern/
Deleting local /home/gwern/
0 files exist on backend
4 files exist in cache
Extracting backup chains from list of files: []
Warning, found signatures but no corresponding backup files
Last full backup date: none
Collection Status
---
Connecting with backend: BackendWrapper
Archive dir: /home/gwern/
Found 0 secondary backup chains.
No backup chains with active signatures found
No orphaned or incomplete backup sets found.
Using temporary directory /home/gwern/
Registering (mktemp) temporary file /home/gwern/
Using temporary directory /home/gwern/
Registering (mktemp) temporary file /home/gwern/
AsyncScheduler: instantiating at concurrency 0
Registering (mktemp) temporary file /tmp/duplicity-
Selecting /home/gwern/dnm
Comparing . and None
Getting delta of (. dir) and None
A .
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Comparing 3-16-cr-00109-SRU and None
Getting delta of (3-16-cr-00109-SRU dir) and None
A 3-16-cr-00109-SRU
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Comparing 3-16-cr-
Getting delta of (3-16-cr-
A 3-16-cr-
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Comparing 3-16-cr-
Getting delta of (3-16-cr-
A 3-16-cr-
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Comparing 3-16-cr-
Getting delta of (3-16-cr-
A 3-16-cr-
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Comparing 3-16-cr-
Getting delta of (3-16-cr-
A 3-16-cr-
Selection: examining path /home/gwern/
Selection: + no selection functions found. Including
Selecting /home/gwern/
Comparing 3-16-cr-
Getting delta of (3-16-cr-
A 3-16-cr-
Removing still remembered temporary file /home/gwern/
Removing still remembered temporary file /home/gwern/
AsyncScheduler: running task synchronously (asynchronicity disabled)
Writing duplicity-
Making directory /tmp/duplicity-
Create Par2 recovery files
Deleting /tmp/duplicity-
Deleting tree /tmp/duplicity-
Selecting /tmp/duplicity-
Deleting /tmp/duplicity-
Deleting /tmp/duplicity-
Forgetting temporary file /tmp/duplicity-
AsyncScheduler: task completed successfully
Processed volume 1
Making directory /home/gwern/
Create Par2 recovery files
Deleting /home/gwern/
Deleting tree /home/gwern/
Selecting /home/gwern/
Deleting /home/gwern/
Making directory /home/gwern/
Create Par2 recovery files
Deleting /home/gwern/
Deleting tree /home/gwern/
Selecting /home/gwern/
Deleting /home/gwern/
1 file exists on backend
8 files exist in cache
Extracting backup chains from list of files: [u'duplicity-
File duplicity-
Warning, found incomplete backup sets, probably left from aborted session
--------------[ Backup Statistics ]--------------
StartTime 1479257641.59 (Tue Nov 15 19:54:01 2016)
EndTime 1479257642.18 (Tue Nov 15 19:54:02 2016)
ElapsedTime 0.59 (0.59 seconds)
SourceFiles 7
SourceFileSize 11078834 (10.6 MB)
NewFiles 7
NewFileSize 11078834 (10.6 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDelt
DeltaEntries 7
RawDeltaSize 11070642 (10.6 MB)
TotalDestin
Errors 0
---
Releasing lockfile /home/gwern/
Removing still remembered temporary file /tmp/duplicity-
par2 file must not have a wildcard in it.
duplicity-
All variants I tried failed:
--par2-options "m2000"
--par2-options "-m2000"
--par2-
--par2-
--par2-options m2000
--par2-options -m2000
Further, duplicity appears to be oblivious to the PAR2 errors or to not be running par2 at all (even debug-level output is unclear on what it's actually doing, as it merely says "Create Par2 recovery files" and continues normally).
Changed in duplicity: | |
status: | New → Fix Committed |
Changed in duplicity: | |
importance: | Undecided → Medium |
Changed in duplicity: | |
status: | Fix Committed → Fix Released |
An additional detail: some of the invocations, while failing to produce PAR2s in the backup destination, did dump some .par2 files in ~/, like m2000-q.par2 m2000-q. vol000+ 190.par2 m2000-q. vol00+34. par2 m4000-q.par2 m4000-q. vol000+ 163.par2 m4000-q. vol000+ 200.par2. (Not sure which, I didn't notice them until just now.)