duplicity 0.7.18 issue with remote directories (B2 back-end)

[stamster]

duplicity: 0.7.18

Python: 2.7.12

B2 PIP version: 1.3.6

OS: Ubuntu 16.04.5 LTS

Using Backblaze B2 as back-end.

Using remote syntax example:

duplicity localDirName b2://id:key@bucket-name/machine/localDirName

duplicity 0.7.17 worked flawlessly with above mentioned syntax.

As of duplicity 0.7.18 - this does not work any longer, as duplicity simply uploads all files into root of the bucket, creating a real mess!

This seems to be bad regression bug.

Manual mentions:

  b2://account_id[:application_key]@bucket_name/[some_dir/]

well, [some_dir/] simply does not work any longer!

[stamster (stamster)]

I have tried to downgrade to previous version, but no luck:

apt install duplicity=0.7.17-0ubuntu0ppa1353~ubuntu16.04.1
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Version '0.7.17-0ubuntu0ppa1353~ubuntu16.04.1' for 'duplicity' was not found

So this is really no-go situation. :/

[stamster (stamster)]

On Backblaze bucket duplicity creates the following:

machine/localDirName/duplicity-full-signatures.20180823T124553Z.sigtar.gpg

So it fails badly to create the directory structure, instead it appends it to the filenames!

This is really a deal breaker for me. Please advise.

[stamster (stamster)]

Ahh. Even worse.

On the next run, it fails to read signatures, thus switches to full backup!

 Last full backup date: none
 No signatures found, switching to full backup.

This shall be classified as prio1 bug IMHO.

[Kenneth Loafman (kenneth-loafman)]

I've attached a patch that _could_ fix the problem. Please try it out and let me know how it goes.

[stamster (stamster)]

Thanks for response and a patch, but I would need to compile it and test it 1st, which I cannot afford ATM as I have to deal with non-working backups.

Instead, reverting to 0.7.17 in the PPA repo would be far more appreciated solutions for everyone IMO.

P.S are we sure this only relates to B2 back-end, or it's general issue with any back-end, even SFTP? I'll try it now as I doubt it's related to B2 only as I have used older version of backend and upgraded to the latest yesterday when this issue occurred.

[stamster (stamster)]

Well as I have assumed.

0.7.18 seems broken in other areas as well (sftp back-end)

duplicity -v4 myBackupDir sftp://mybkpsrv.mydomain

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
GnuPG passphrase:
Retype passphrase to confirm:
No signatures found, switching to full backup.
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1567, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1553, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1405, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1528, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 584, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 466, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 465, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 355, in put
    validate_block(putsize, dest_filename)
  File "/usr/bin/duplicity", line 337, in validate_block
    util.escape(dest_filename), orig_size))
 TypeError: %d format: a number is required, not NoneType

Facts:

- The files are actually transferred - but this error prevents duplicity from displaying stats etc.

- the path is respected in this case sftp://srv//tmp/dj/radio/ok/isit

Any info of reverting to 0.7.17 but unpacking manually TAR?

[stamster (stamster)]

FYI: I tried above mentioned commands from machine which still is not updated so it holds 0.7.17 version, and everything worked flawlessly - both B2 and SFTP. Other way around - machine which was upgraded - is not usable any longer for B2 backups.

[Andrew Berry (andrewberry)]

I'm seeing similar issues with B2.

Synchronizing remote metadata to local cache...
Copying duplicity-full-signatures.20170908T121820Z.sigtar.gpg to local cache.
Get: home/duplicity-full-signatures.20170908T121820Z.sigtar.gpg -> /tmp/duplicity-hLf6D0-tempdir/mktemp-gCDG92-2
Attempt 1 failed. UnknownError: Unknown error: 404 not_found bucket REDACTED does not have file: home%2Fduplicity-full-signatures.20170908T121820Z.sigtar.gpg

I was able to download the previous release from https://launchpad.net/~duplicity-team/+archive/ubuntu/ppa/+build/14396862

[stamster (stamster)]

@andrewberry thanks for deb link!

I was able to revert to 0.7.17, and it worked again!

But I decided to upgrade B2 back-end soon after, from 1.1.0 to 1.3.6 and now it fails with:

Attempt 1 failed. AttributeError: B2ProgressListener instance has no attribute '__exit__'
Attempt 2 failed. AttributeError: B2ProgressListener instance has no attribute '__exit__'
Attempt 3 failed. AttributeError: B2ProgressListener instance has no attribute '__exit__'
Attempt 4 failed. AttributeError: B2ProgressListener instance has no attribute '__exit__'
Giving up after 5 attempts. AttributeError: B2ProgressListener instance has no attribute '__exit__'

So this is really hard to mix and match. :/

[stamster (stamster)]

OKi, here's an update.

1.3.x release of B2 back-end / cmd line tool does not work with Duplicity (at least not with 0.7.17), e.g. it seems they have changed API in 1.3.x series, not conforming to the SemVer.

So here https://github.com/Backblaze/B2_Command_Line_Tool/releases we see a list of releases, my trial and error results are that this is the working setup:

> duplicity --version
duplicity 0.7.17

> pip show b2
Name: b2
Version: 1.2.0
Summary: Command Line Tool for Backblaze B2
Home-page: https://github.com/Backblaze/B2_Command_Line_Tool
Author: Backblaze, Inc.
Author-email: <email address hidden>
License: MIT

[Kenneth Loafman (kenneth-loafman)]

Has anyone tried the patch from #4 above. I don't have access to B2.

[Kenneth Loafman (kenneth-loafman)]

This is fixed in the trunk and will be in the daily PPAs soon.

[stamster (stamster)]

Hello Kenneth Loafman,

I have just fetched duplicity 0.7.18.1 from repo.

It works like a charm!

Last full backup date: Mon Aug 27 14:45:05 2018
--------------[ Backup Statistics ]--------------
StartTime 1535385546.49 (Mon Aug 27 17:59:06 2018)
EndTime 1535385546.98 (Mon Aug 27 17:59:06 2018)
ElapsedTime 0.49 (0.49 seconds)
SourceFiles 312
SourceFileSize 2878474139 (2.68 GB)
NewFiles 2
NewFileSize 48805 (47.7 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 2
RawDeltaSize 28325 (27.7 KB)
TotalDestinationSizeChange 28649 (28.0 KB)
Errors 0
-------------------------------------------------

Kudos to @kenneth-loafman