[enhancement] please consider using rclone as backend

Bug #1769267 reported by c sights on 2018-05-04
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Duplicity
Medium
Unassigned

Bug Description

Using rclone as a backend might help simplify the setup of the many targets duplicity supports.
someone has written a rclone adapter for duplicity already:
https://github.com/GilGalaad/duplicity-rclone

I'm planning to use it with Google Drive (as the author above did) and not have to install pydrive, which is an ugly pip experience

rclone supports many other targets also (below).
Thanks for your work!

https://rclone.org/

    Amazon Drive
    Amazon S3
    Backblaze B2
    Box
    Ceph
    DigitalOcean Spaces
    Dreamhost
    Dropbox
    FTP
    Google Cloud Storage
    Google Drive
    HTTP
    Hubic
    IBM COS S3
    Memset Memstore
    Mega
    Microsoft Azure Blob Storage
    Microsoft OneDrive
    Minio
    Nextloud
    OVH
    Openstack Swift
    Oracle Cloud Storage
    ownCloud
    pCloud
    put.io
    QingStor
    Rackspace Cloud Files
    SFTP
    Wasabi
    WebDAV
    Yandex Disk
    The local filesystem

rclone looks good. Will have to investigate further.

Just curious... What makes 'pip install pydrive' an ugly experience?

On Fri, May 4, 2018 at 3:55 PM, c sights <email address hidden> wrote:

> Public bug reported:
>
> Using rclone as a backend might help simplify the setup of the many
> targets duplicity supports.
> someone has written a rclone adapter for duplicity already:
> https://github.com/GilGalaad/duplicity-rclone
>
> I'm planning to use it with Google Drive (as the author above did) and
> not have to install pydrive, which is an ugly pip experience
>
> rclone supports many other targets also (below).
> Thanks for your work!
>
> https://rclone.org/
>
> Amazon Drive
> Amazon S3
> Backblaze B2
> Box
> Ceph
> DigitalOcean Spaces
> Dreamhost
> Dropbox
> FTP
> Google Cloud Storage
> Google Drive
> HTTP
> Hubic
> IBM COS S3
> Memset Memstore
> Mega
> Microsoft Azure Blob Storage
> Microsoft OneDrive
> Minio
> Nextloud
> OVH
> Openstack Swift
> Oracle Cloud Storage
> ownCloud
> pCloud
> put.io
> QingStor
> Rackspace Cloud Files
> SFTP
> Wasabi
> WebDAV
> Yandex Disk
> The local filesystem
>
> ** Affects: duplicity
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1769267
>
> Title:
> [enhancement] please consider using rclone as backend
>
> Status in Duplicity:
> New
>
> Bug description:
> Using rclone as a backend might help simplify the setup of the many
> targets duplicity supports.
> someone has written a rclone adapter for duplicity already:
> https://github.com/GilGalaad/duplicity-rclone
>
> I'm planning to use it with Google Drive (as the author above did) and
> not have to install pydrive, which is an ugly pip experience
>
> rclone supports many other targets also (below).
> Thanks for your work!
>
> https://rclone.org/
>
> Amazon Drive
> Amazon S3
> Backblaze B2
> Box
> Ceph
> DigitalOcean Spaces
> Dreamhost
> Dropbox
> FTP
> Google Cloud Storage
> Google Drive
> HTTP
> Hubic
> IBM COS S3
> Memset Memstore
> Mega
> Microsoft Azure Blob Storage
> Microsoft OneDrive
> Minio
> Nextloud
> OVH
> Openstack Swift
> Oracle Cloud Storage
> ownCloud
> pCloud
> put.io
> QingStor
> Rackspace Cloud Files
> SFTP
> Wasabi
> WebDAV
> Yandex Disk
> The local filesystem
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1769267/+subscriptions
>

Changed in duplicity:
importance: Undecided → Wishlist

Hi Ken,

> Just curious... What makes 'pip install pydrive' an ugly experience?

It's been a couple years. I seem to remember pip splattering files
throughout the local file hierarchy (/usr/share/python ?) and not
tracking them so that uninstall didn't reverse the process.

I prefer rclone's one file approach. (Probably pip is very lovable once
you get to know it. :) )

Also, I think back then pydrive or google-python-api-wrapper was
stagnant and depended on old other pip packages. My old config shows
that I was specifically using oauth2client==1.5.2 . Not sure why
anymore. :/

Thanks for your work!
C.

Edison Wong (hswong3i) wrote :

Could we simply support rclone as backend by integrating https://github.com/GilGalaad/duplicity-rclone into duplicity upstream?

Changed in duplicity:
assignee: nobody → Kenneth Loafman (kenneth-loafman)
importance: Wishlist → Medium
milestone: none → 0.8.08
status: New → In Progress
edso (ed.so) wrote :

https://github.com/GilGalaad/duplicity-rclone

has some clutter and should be cleaned up first.
- method subprocess should be removed
- local renaming of temp files seems superfluous
- error reporting just shows first messag line

also the url scheme should be streamlined to current duplicity standard eg.
rclone://gdrive:/ => rclone+gdrive://

and agn. manpage doc is missing

..ede

Yes, it needs a bit of work, especially conversion to Python 3. I am
thinking that, with everything in the backends, here is a product built for
talking to all the remotes we have, and then some.

It will need a man page entry, but that will probably just refer to the
rclone docs (much better than ours). We will normalize the process and put
in some tests as well.

On Mon, Nov 18, 2019 at 12:01 PM edso <email address hidden> wrote:

> https://github.com/GilGalaad/duplicity-rclone
>
> has some clutter and should be cleaned up first.
> - method subprocess should be removed
> - local renaming of temp files seems superfluous
> - error reporting just shows first messag line
>
> also the url scheme should be streamlined to current duplicity standard eg.
> rclone://gdrive:/ => rclone+gdrive://
>
> and agn. manpage doc is missing
>
> ..ede
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1769267
>
> Title:
> [enhancement] please consider using rclone as backend
>
> Status in Duplicity:
> In Progress
>
> Bug description:
> Using rclone as a backend might help simplify the setup of the many
> targets duplicity supports.
> someone has written a rclone adapter for duplicity already:
> https://github.com/GilGalaad/duplicity-rclone
>
> I'm planning to use it with Google Drive (as the author above did) and
> not have to install pydrive, which is an ugly pip experience
>
> rclone supports many other targets also (below).
> Thanks for your work!
>
> https://rclone.org/
>
> Amazon Drive
> Amazon S3
> Backblaze B2
> Box
> Ceph
> DigitalOcean Spaces
> Dreamhost
> Dropbox
> FTP
> Google Cloud Storage
> Google Drive
> HTTP
> Hubic
> IBM COS S3
> Memset Memstore
> Mega
> Microsoft Azure Blob Storage
> Microsoft OneDrive
> Minio
> Nextloud
> OVH
> Openstack Swift
> Oracle Cloud Storage
> ownCloud
> pCloud
> put.io
> QingStor
> Rackspace Cloud Files
> SFTP
> Wasabi
> WebDAV
> Yandex Disk
> The local filesystem
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1769267/+subscriptions
>

One possible show-stopper, duplicity is under GPL v2, rclonebackend.py is under GPL v3. I don't think we can arbitrarily mix the two licenses. Thoughts?

Changed in duplicity:
milestone: 0.8.08 → none
Changed in duplicity:
milestone: none → 0.8.09

I've released a working version of rclonebackend.py, but am not happy yet.

1) The renames are a real sticking point. You can't 'rclone copy dir/a dir/b'. Instead of copying to dir/b, it will create dir/b/a. The renames are a point of confusion when debugging, especially with our Ctrl-C happy set of users. I am investigating the use of 'rclone cat' at the moment. That could be the solution.

2) No man page yet. Another sticking point.

3) There is a test case, so at least it works.

Changed in duplicity:
status: In Progress → Fix Committed
edso (ed.so) wrote :

wrt. the license issue

as duplicity is licensed
"
# Duplicity is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 2 of the License, or (at your
# option) any later version.
"
note the _any later version_ option, we can easily include GPL3 code, which will make the whole then a GPL2 and GPL3 mixture. totally legal ;)
the other way around won't work though, basing GPL2'd code on GPL3 code.

..ede/duply.net

c sights (cwseys) wrote :

> 1) The renames are a real sticking point. You can't 'rclone copy dir/a
> dir/b'. Instead of copying to dir/b, it will create dir/b/a. The
> renames are a point of confusion when debugging, especially with our
> Ctrl-C happy set of users. I am investigating the use of 'rclone cat'
> at the moment. That could be the solution.

rclone moveto ?
https://rclone.org/commands/rclone_moveto/

Changed in duplicity:
milestone: 0.8.09 → 0.8.10
Changed in duplicity:
milestone: 0.8.10 → 0.8.11
Changed in duplicity:
assignee: Kenneth Loafman (kenneth-loafman) → nobody
Changed in duplicity:
status: Fix Committed → In Progress
milestone: 0.8.11 → 0.8.12
Changed in duplicity:
assignee: nobody → Kenneth Loafman (kenneth-loafman)
Changed in duplicity:
status: In Progress → Fix Committed
assignee: Kenneth Loafman (kenneth-loafman) → nobody
Changed in duplicity:
status: Fix Committed → Fix Released
c sights (cwseys) wrote :

Congratulations! Very exciting.
C.

On 3/19/20 2:47 PM, Kenneth Loafman wrote:
> ** Changed in: duplicity
> Status: Fix Committed => Fix Released
>

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.