Please add support for par2 creating

[Plamen K. Kosseff]

OK, this add support for creating par2 recovery files for every file that gets uploaded to the backend.

- can be enabled by passing --par2. If the backend already contains par2 files there is no need to pass the --par2 option.
- the par2 files are created just before the upload of the file so this will not break the asynchronous mode.
- all par2 files are preserved in the local cache and uploaded to the backend.

todo:
- add comments and logging
- add support for customizing the pa2 options currently 20% redundancy is used which might be an overkill or too little for someone
- don't use os.system and add error handling.
- add support for recovery.

Code review will be greatly appreciated. There are some very ugly hacks with temp files

[Stéphane Lesimple (speed47)]

Very nice idea! I'm testing duplicity 0.6.05 with your patch right now on my server, with daily backups.

I'm not yet comfortable enough with python (or duplicity's code) to make a code review, but that still might happen eventually :)

[Kenneth Loafman (kenneth-loafman)]

I'm assigning this to the entire duplicity-team since we'll need all of your help in getting this into 0.7-series. I'm working on getting the initial patch to work correctly with all of the unit tests, in some cases changing the tests to fit, then I'll push a version to 0.7-series, close to what Plamen had in the patch.

[Stéphane Lesimple (speed47)]

I'm using this patch for a couple of days now. I've already made 2 modifications to the code.
2 more patches are attached, I've tested and they also work against the current 0.7-series branch.

The first patch avoids storing all the par2 files into the local cache. On my installation, the --archive-dir directory was growing up beyond the gigabyte limit since I applied the par2 patch, which was way too much !

[Stéphane Lesimple (speed47)]

This second patch skips the par2 files when building up the sets and chains of backups, used for the collection-status command. Without it, the par2 files were considered as being incomplete backups:

Also found 0 backup sets not part of any chain,
and 2 incomplete backup sets.
These may be deleted by running duplicity with the --cleanup option.

This is misleading because the cleanup command won't remove those files (of course!) and we'll still get the "incomplete backup sets" message on the next collection-status command.
After this patch is applied, a duplicity collection-status on the same backup profile says:

No orphaned or incomplete backup sets found.

Hopefully there's no side effect. Haven't noticed any for now, but it may require more testing.

[Kenneth Loafman (kenneth-loafman)]

Thanks for the patches. What exactly is the gigabyte limit you refer to?

[Stéphane Lesimple (speed47)]

Sorry, I wrote those comments a bit late at night :)
I meant something like "the gigabyte barrier", as in "wow, my folder is now one gigabyte large !". This is not referring to any limit in the software or OS.
I was just thinking that more than one gigabyte was a bit large for only metadata, and is the reason why I had a look into the folder, and found the par2 files into it.

[Plamen K. Kosseff (p-kosseff)]

Hi,

I definitely prefer having the par2 files locally. May be there have to be a command line option that determines that?

[Plamen K. Kosseff (p-kosseff)]

Any way I plan to start working on the recovery part asap so I'll add something like --cache-par2.

[gwern (gwern0)]

What is the status of these par2 patches? It's been almost 2 years since the last comment; has the functionality been incorporated into duplicity? (I've been generating the PAR2 files for my duplicity backups manually and it's fairly annoying.)

[Kenneth Loafman (kenneth-loafman)]

The par2 patches were merged into the 0.7 series, but that series has fallen
behind 0.6 and needs serious work. My last attempt at merging the recent
changes met with disaster, so I'm inclined to restart on par2, use some of
the ideas, but rewrite the code from scratch. For now, treat 0.7 as very
experimental.

On Sat, Jun 4, 2011 at 11:50 AM, gwern <email address hidden> wrote:

> What is the status of these par2 patches? It's been almost 2 years since
> the last comment; has the functionality been incorporated into
> duplicity? (I've been generating the PAR2 files for my duplicity backups
> manually and it's fairly annoying.)
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/426282
>
> Title:
> [PATCH] par2 creating support
>
> Status in Duplicity - Bandwidth Efficient Encrypted Backup:
> In Progress
>
> Bug description:
> OK, this add support for creating par2 recovery files for every file
> that gets uploaded to the backend.
>
> - can be enabled by passing --par2. If the backend already contains par2
> files there is no need to pass the --par2 option.
> - the par2 files are created just before the upload of the file so this
> will not break the asynchronous mode.
> - all par2 files are preserved in the local cache and uploaded to the
> backend.
>
>
> todo:
> - add comments and logging
> - add support for customizing the pa2 options currently 20% redundancy is
> used which might be an overkill or too little for someone
> - don't use os.system and add error handling.
> - add support for recovery.
>
> Code review will be greatly appreciated. There are some very ugly hacks
> with temp files
>

[Germar (germar)]

I wrote a wrapper backend that create Par2 files and write them to the wrapped backend. Just add a leading 'par2+' to your url_string to use this (e.g. 'par2+webdavs://USER@HOST/PATH')

pros:
- easier to maintain than build-in par2 functions

cons:
- incorrect size calculation as par2 files are not included in size
- no async creation of par2 files