[patch] Dropbox backend: update to SDK v2, use chunked upload

[Dmitry Nezhevenko]

Hi,

I've just updated Dropbox backend, since it looks like outdated and have a lot of problems:
1. It doesn't work out of the box due to hardcoded SDK key. Replacing it requires editing of backend source file in /usr.
2. Hardcoded authentication file (~/.dropbox_token_store.txt). That's why there is no way to backup to a few different dropbox accounts.
3. It doesn't work well with large files upload: tries to upload it without chunks. That's actually why I wasn't able to perform full backup (it was just impossible to upload signatures file).
4. Sometimes it produces broken backups on my machine due to dropbox 'autorename' feature. File uploaded as 'duplicity-full.20160301T074309Z.vol2.difftar.gz' may be stored as 'duplicity-full.20160301T074309Z.vol2.difftar.gz (1)' if previous attempt failed at certain point.
5. It uses deprecated v1 SDK.

I've tried to address all of these issues and updated backend code.

New code uses new SDKv2 and passes backendtest. It also uses chunked upload. So should support very large files.

Backend is tested on ~10GB backup. And I'm checking it right now for whole machine backup.

For now it's most likely RFC. Backend requires change of configuration due to different way to store authentication data. It seems that there is no way to gracefully migrate setups because old SDKv1 tokens are not compatbile with new OAuth2 way.

PS. Any suggestions what to do with old "duplicity-full.20160301T074309Z.vol2.difftar.gz (1)" files? it seems that duplicity cleanup don't delete them (even with --extra-clean)? It it any way to completely remove all files that are not understandable by duplicity?

[Dmitry Nezhevenko (dmitry.nezhevenko)]

Fixes followed bugs:
https://bugs.launchpad.net/duplicity/+bug/1388600
https://bugs.launchpad.net/duplicity/+bug/1379736

[Kenneth Loafman (kenneth-loafman)]

That autorename feature may be a problem. The regex for matching uses a
fence of '$' which stops the match before the added ' (1)'. Changing that
could have bad side effects. Not sure.

As to authentication, that part decidedly needs to be reworked in
duplicity. We probably need a config file for each backup and more.

On Tue, Mar 1, 2016 at 3:17 AM, Dmitry Nezhevenko <
<email address hidden>> wrote:

> Fixes followed bugs:
> https://bugs.launchpad.net/duplicity/+bug/1388600
> https://bugs.launchpad.net/duplicity/+bug/1379736
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1551620
>
> Title:
> [patch] Dropbox backend: update to SDK v2, use chunked upload
>
> Status in Duplicity:
> New
>
> Bug description:
> Hi,
>
> I've just updated Dropbox backend, since it looks like outdated and have
> a lot of problems:
> 1. It doesn't work out of the box due to hardcoded SDK key. Replacing it
> requires editing of backend source file in /usr.
> 2. Hardcoded authentication file (~/.dropbox_token_store.txt). That's
> why there is no way to backup to a few different dropbox accounts.
> 3. It doesn't work well with large files upload: tries to upload it
> without chunks. That's actually why I wasn't able to perform full backup
> (it was just impossible to upload signatures file).
> 4. Sometimes it produces broken backups on my machine due to dropbox
> 'autorename' feature. File uploaded as
> 'duplicity-full.20160301T074309Z.vol2.difftar.gz' may be stored as
> 'duplicity-full.20160301T074309Z.vol2.difftar.gz (1)' if previous attempt
> failed at certain point.
> 5. It uses deprecated v1 SDK.
>
> I've tried to address all of these issues and updated backend code.
>
> New code uses new SDKv2 and passes backendtest. It also uses chunked
> upload. So should support very large files.
>
> Backend is tested on ~10GB backup. And I'm checking it right now for
> whole machine backup.
>
> For now it's most likely RFC. Backend requires change of
> configuration due to different way to store authentication data. It
> seems that there is no way to gracefully migrate setups because old
> SDKv1 tokens are not compatbile with new OAuth2 way.
>
> PS. Any suggestions what to do with old "duplicity-
> full.20160301T074309Z.vol2.difftar.gz (1)" files? it seems that
> duplicity cleanup don't delete them (even with --extra-clean)? It it
> any way to completely remove all files that are not understandable by
> duplicity?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1551620/+subscriptions
>

[Dmitry Nezhevenko (dmitry.nezhevenko)]

Actually autorename is problem for old backup. With this patch applied new backups will not cause autorenaming since we explicitly are telling dropbox to avoid doing this.

It just will be hard to recover old backups because sometimes it's not clear what file to use: "difftar.gz" or difftar.gz (1)", etc.

Plus now I'm trying to be as robust as possible. So files are named AFTER 100% upload.

[Kenneth Loafman (kenneth-loafman)]

I would suggest that the user make the determination on what to recover.
Perhaps your list function could somehow log the renamed files? I'm
guessing the user could then rename the ones that were the longest. Just a
suggestion.

On Tue, Mar 1, 2016 at 7:04 AM, Dmitry Nezhevenko <
<email address hidden>> wrote:

> Actually autorename is problem for old backup. With this patch applied
> new backups will not cause autorenaming since we explicitly are telling
> dropbox to avoid doing this.
>
> It just will be hard to recover old backups because sometimes it's not
> clear what file to use: "difftar.gz" or difftar.gz (1)", etc.
>
> Plus now I'm trying to be as robust as possible. So files are named
> AFTER 100% upload.
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1551620
>
> Title:
> [patch] Dropbox backend: update to SDK v2, use chunked upload
>
> Status in Duplicity:
> New
>
> Bug description:
> Hi,
>
> I've just updated Dropbox backend, since it looks like outdated and have
> a lot of problems:
> 1. It doesn't work out of the box due to hardcoded SDK key. Replacing it
> requires editing of backend source file in /usr.
> 2. Hardcoded authentication file (~/.dropbox_token_store.txt). That's
> why there is no way to backup to a few different dropbox accounts.
> 3. It doesn't work well with large files upload: tries to upload it
> without chunks. That's actually why I wasn't able to perform full backup
> (it was just impossible to upload signatures file).
> 4. Sometimes it produces broken backups on my machine due to dropbox
> 'autorename' feature. File uploaded as
> 'duplicity-full.20160301T074309Z.vol2.difftar.gz' may be stored as
> 'duplicity-full.20160301T074309Z.vol2.difftar.gz (1)' if previous attempt
> failed at certain point.
> 5. It uses deprecated v1 SDK.
>
> I've tried to address all of these issues and updated backend code.
>
> New code uses new SDKv2 and passes backendtest. It also uses chunked
> upload. So should support very large files.
>
> Backend is tested on ~10GB backup. And I'm checking it right now for
> whole machine backup.
>
> For now it's most likely RFC. Backend requires change of
> configuration due to different way to store authentication data. It
> seems that there is no way to gracefully migrate setups because old
> SDKv1 tokens are not compatbile with new OAuth2 way.
>
> PS. Any suggestions what to do with old "duplicity-
> full.20160301T074309Z.vol2.difftar.gz (1)" files? it seems that
> duplicity cleanup don't delete them (even with --extra-clean)? It it
> any way to completely remove all files that are not understandable by
> duplicity?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1551620/+subscriptions
>

[Dmitry Nezhevenko (dmitry.nezhevenko)]

Ok. I can add some warning (or even just "print") output in _list method if it detects files with "([0-9]+)" pattern. Hope that this should be enough.

I'll upload updated patch once I'll test it.

[Dmitry Nezhevenko (dmitry.nezhevenko)]

Here is second version that adds warning about renamed files.

[Dmitry Nezhevenko (dmitry.nezhevenko)]

Ok. I've finally finished full backup and "verify --compare-data" after this.

Everything seems to be ok. The only remaining issue is that there is no way to configure network timeout for dropbox. I've already proposed patch to SDK to use socket.getdefaulttimeout() for this.

[Kenneth Loafman (kenneth-loafman)]

I applied the patch, but it seems that it's not passing the code tests. See attached.

You can run the code tests by themselves with:

$ RUN_CODE_TESTS=1 python -u setup.py test -s testing.test_code.CodeTest

The most concerning one was the pylint error found in the last test, use of a non-existent -- operator.

[Dmitry Nezhevenko (dmitry.nezhevenko)]

oops. I'll check and back with fixed version.
Sorry. Tried only backendtest.

[Dmitry Nezhevenko (dmitry.nezhevenko)]

now everything seems to be clear.

Thanks!