add backblaze's B2 cloud storage as backup destination

[wcchandler]

FEATURE REQUEST
======================

Under storage location, please add backblaze's B2 service. Support has been added to duplicity as of version 0.7.06 as reported by this bug: https://bugs.launchpad.net/duplicity/+bug/1498877

[Alexey Zagarin (zagarin)]

Hi,

I've made the patch to support B2 Cloud Storage. It works for me. However, further testing needed. Please, review and consider merging.

[Erik Entrich (eentrich)]

I just tested your patch, and the initial backup seems to work correctly until it suddenly switches from "Backing up" to "Cleaning up" and deletes everything uploaded so far after which it starts over from the beginning.
I'm not sure if this is a problem with duplicity or deja-dup, so I'm going to retry the backup with just duplicity. It might take a while to replicate the error though since the backup ran fine for around 2 days before it occurred.

[Alexey Zagarin (zagarin)]

There were improvements in duplicity regarding B2 backend since v0.7.6 so you may try to build it from git. I didn't hit the behavior you described even with v0.7.6, but had other issues so I suggest you update duplicity first.

[Alexey Zagarin (zagarin)]

This bug https://bugs.launchpad.net/duplicity/+bug/1529883 in particular prevented the successful backup in my case.

[Erik Entrich (eentrich)]

Thank you, I'll try that and do more tests using duplicity directly!

[Erik Entrich (eentrich)]

I finally got around to testing it with the release version of duplicity 0.7.7, and it worked well until the very end of the initial backup when it decided to delete all my uploaded files again. I'm not sure what happened since the backup had been running flawlessly for around 8 days at that point and I wasn't watching it. I stopped the backup a few times in between but it always resumed without problems.

I could see from the console output that there was an error message after it uploaded the last file, but when I tried copying it my terminal crashed... It contained an error about a missing file the message "This might be a bug", but I couldn't find any info about that online.

I tried recreating the error with smaller directories, but it worked without any problems.

I know is is probably a duplicity bug, but is there any way to make deja-dup not delete the entire backup when an error happens? I would hate for to lose my entire backups in a year or so just because uploading a file failed or something like that.

[Marcos Alano (mhalano)]

eentrich,

Could you test with Duplicity 0.7.08 and check if bug remains?

Thanks.

[Erik Purins (epu)]

I rebuilt with patch applied to rev 1563, with duplicity 0.7.09 installed first in PATH, and can enter B2 storage information (but not application key) into the dialog. When I try to connect, I am prompted for my B2 account information and application key. I entered them, and saved them using the remember checkbox. When I go to run a backup immediately, I get a Backup Failed dialog, with "FatalBackendException: Bucket cannot be created". This happens whether or not the named bucket and folder exist.

I'm not sure how to begin debugging the application. If it will help, you can contact me via e-mail and I can gather more information.

[Michael Terry (mterry)]

If support gets added to GNOME Online Accounts for B2, I'll add it to deja-dup too. But I'm trying to avoid adding new custom duplicity backend supports to deja-dup.

https://wiki.gnome.org/Apps/DejaDup/Clouds

[step21 (step22)]

Sorry, but how does this make sense? This basically makes deja-dup useless, especially if also deprecating S3 etc. Services like Google Drive and Nextcloud which you (and GOA) seem to be focused on are not well suited to handle proper backups in my experience. And if going by https://wiki.gnome.org/Projects/GnomeOnlineAccounts/Providers is any indication, you might as well change "when Gnome Online Accounts adds support" to "never" as they only seem to support very end-user focused services.

[Michael Terry (mterry)]

I hear what you're saying. But (A) S3 support hasn't been removed and there are no plans to do so right now. And (B) Deja Dup has always been meant for more casual users that wouldn't otherwise make a backup. So we share GOA's end-user focus. (I've long wished our default cloud backup was easier to set up than S3 is.) There are plenty of other more advanced backup solutions for advanced users.

[step21 (step22)]

Thanks for still replying :) It has not, but it looks like it will be at one point. My point is that even as a maybe not-so casual user, I would still like a nice interface for my backup, especially as this (like for many users) makes it much more likely for me to actually do backups. Sure I can use rclone or duplicity directly but it's super annoying especially if you have to automate it. I do not mind the 'default' solution being somethign else (like dropbox?) or for a long time I have used it for local backup to an external drive. But now that better online options are available, it seems too focused on one specific audience if I cannot use those, especially with duplicity suporting them and somebody already having supplied/started a patch. And no, there are not really many options available, especially if you like a GUI now and then. Sure I could figure out how to set up duplicity or rclone via a cron job or something, but that is much more annoying and probably unreliable, especially on a laptop. Then there are enterprise systems but they are mostly proprietary and well, focused on enterprise. So I really think there should be some middle ground, especially with a nice DE like Gnome.

[Michael Terry (mterry)]

Yeah, I hear ya. But I have limited resources and my own focus is on the casual end.

Plus, restricting it to gvfs-supported backends lets us do fun stuff like calculate how much space the backup folder is taking up or actually know whether we will be able to connect through the vpn or weird network setups, etc.

So there is a real cost to supporting new backends. I'd consider a patch for an advanced mode that allowed arbitrary duplicity backends, but it would have to be carefully done (duplicity backends are very idiosyncratic) and come with sufficient warnings about not having all features/poor support/etc.

[step21 (step22)]

Sure. I might actually think about doing a patch like this, though no idea about a timetable.
At least it seems to be easier than adding backends to gvfs/goa.
The simplest duplicity backend would probably be something like vuplicity does/did (https://github.com/johansatge/vuplicity) where basically you just compose the terminal command. ;) Though I do not mean to advocate that as a good idea...

[Wizek (123-wizek)]

Whoops, sorry about changing the 'information type' field. It was an accidental miss-click, and I hope I was able to undo the change fully (I'm not very familiar with launchpad for the moment).

[Michael (sparks305)]

Please add support for B2 as a destination!!!!!!!!!!!!!!!

[Thomas HERON (thomas-heron)]

Please add support for B2 as a destination.

[Alan (alanbix)]

Please add support for B2 as a destination.

[Robert Navarro (crshman)]

Please add support for B2 as a destination.

[Lewis Juggins (lwis)]

Has any progress been made on this?

[Michael Terry (mterry)]

@lwis: no, I'm not currently planning to work on this. See my other comments in this bug.

[paulwilson05 (paulwilson05)]

:( I'll pay you $100

[paulwilson05 (paulwilson05)]

how about $10k
https://www.gofundme.com/b2-as-a-deja-dup-destination

[Mentos1386 (tine-jozelj)]

Please add support for B2 as a destination.

[paulwilson05 (paulwilson05)]

COME ON!!!!

[Felipe Portella (felipeportella)]

+1 for B2 support (in fact a config file to add/config other providers)

[Endolith (endolith)]

Has this been fixed yet?

[Briggs (jba1978)]

Please add me to the list! I would LOVE to see this implemented.

[Carl van Tonder (carlvantonder)]

I updated the patch for latest deja-dup; currently on a very slow connection but it seems to be working fine

[Yonn Trimoreau (bobspongieux)]

Please add support for B2 as a destination.

[Damien Martins (damien.martins)]

I add my voice. Support for B2 in Déjà Dup would be great

[iblue (iblue)]

Cross-linking the issue in GitLab: https://gitlab.gnome.org/World/deja-dup/-/issues/81

[Daniel Rosehill (drosehill)]

Adding my voice. Would like to start moving some of my S3 buckets over to Backblaze but would need a non-proprietary GUI to be confident doing that.

[Dan Gillmor (dan-gillmor)]

Agree, adding B2 would be a great feature.

[Andras (andrewlucky)]

I also would love to see B2 cloud storage support in Déjà Dup. Thanks

[Danie van der Merwe (dvdmerwe)]

Yes I also wanted support for B2 - it's a show stopper for me as B2 is the most cost effective and is supported by Duplicity. Déjà Dup would have been perfect for me to use. So I'm going to try implement it via OMV with Duplicity now, and continue using LuckyBackup from my desktop.

[Ryan Micallef (micallef)]

I'm also interested. I've been backing up my Ubuntu machine to a local hard drive and I miss the convenience and offsite security of Backblaze. I don't want to have to get into the guts of Ubuntu and I don't want to have to worry about whether I've set everything up correctly, risking my potential future recovery being worthless.

[BlueT - Matthew Lien - 練喆明 (bluet)]

It'd be really nice if deja-dup could support B2

[Justin Lamp (modzilla)]

Yes, please add B2 or S3, that would make it insanely useful and the defacto best way to backup the Linux desktop!

[Ted Major (tidmarsh)]

Count me in for requesting deja-dup support for Backblaze. I use it to back up my family's Macs, and it sure would be nice to have one account for all my backups (and also for those of us who neither have nor want Google accounts).

Thanks for your consideration.

[juanejot (juanejot)]

I would like to add my voice to the chorus requesting the addition of B2 functionality to Déjà Dup. Seems like it can be done with either/both Duplicity and Restic (long-standing and optional up-and-coming backends for Déjà Dup, respectively), but the CLI setup for automation of/reporting on either of those is a bear depending on permissions. To have it handled in a tidy GUI that stays properly configured* would be great.
*(Duplicati, which is a great web-based UI for backing up to local disks on both Windows & Linux, seems to keep forgetting its B2 configs. But I digress.)

[Kathrin Passig (kathrinpassig)]

It would be great if you could add support for B2. I switched from Mac and I'd much prefer to stay with Backblaze/B2 than use Google Drive.

[Bret (baradhili)]

I'll add my support as well... the patch is now too old to apply, but it must be useful to a degree?
I run duplicity from the corn at the moment - I really want to be able to at least restore via a GUI!

[Bret (baradhili)]

Just in case its useful - here is the script I use - obviously there is some variable setting above

--# Remove files older than 90 days
duplicity \
 --sign-key $SGN_KEY --encrypt-key $ENC_KEY \
 remove-older-than 90D --force \
 b2://${B2_ACCOUNT}:${B2_KEY}@${B2_BUCKET}/${B2_DIR}

# Perform the backup, make a full backup if it's been over 30 days
duplicity --progress\
 --sign-key $SGN_KEY --encrypt-key $ENC_KEY \
 --full-if-older-than 30D \
 --include-filelist /home/bret/backups/document-files.txt \
 ${LOCAL_DIR} b2://${B2_ACCOUNT}:${B2_KEY}@${B2_BUCKET}/${B2_DIR}

# Cleanup failures
duplicity \
 cleanup --force \
 --sign-key $SGN_KEY --encrypt-key $ENC_KEY \
 b2://${B2_ACCOUNT}:${B2_KEY}@${B2_BUCKET}/${B2_DIR}

# Show collection-status
duplicity collection-status \
 --sign-key $SGN_KEY --encrypt-key $ENC_KEY \
  b2://${B2_ACCOUNT}:${B2_KEY}@${B2_BUCKET}/${B2_DIR}

# Unset variables
unset B2_ACCOUNT
unset B2_KEY
unset B2_BUCKET
unset B2_DIR
unset LOCAL_DIR
unset ENC_KEY
unset SGN_KEY
unset PASSPHRASE
unset SIGN_PASSPHRASE

--