New MEGA.nz backend based on official "MEGAcmd" tools (patch attached)

Bug #1877885 reported by José L. Domingo López on 2020-05-10
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Duplicity
Low
Unassigned

Bug Description

Hi, first of all, a big thank you to all the developers behind this software, as it is for me the best balance between features and simplicity for taking and managing regular backups (in my case, for a few PCs acting as personal computer, home automation server, and even a RPi).

The existing MEGA backend in "duplicity" still relies on (old) "megatools" package, and I found out the hard way for accounts created since November 2018, due to authentication changes, "megatools" no longer work for any such users. See :
https://github.com/megous/megatools/issues/411

So without any prior Python experience, bit the bullet and created a new backend for MEGA.nz based around the current official tools "MEGAcmd" :
https://mega.nz/cmd

NOTE : the new backend works for all MEGA accounts. The difference is the existing backend only works for accounts created up to approximately November 2018.

Of course, I did so by just mimicking the code in the existing MEGA backend, and adapting it to the new commands (ie "mega-ls" instead of "megals"), but also changed the way the backend works, as with the new tools commands no longer rely on a config file with the credentials, and instead, expect a session to be established to MEGA first, and then using some persistent per-user resident daemon (mega-cmd-server) to dispatch commands to MEGA from the CLI commands (which are basically shell wrappers around a single "mega-exec" binary.

Although the new tools don't rely on a config file for authentication, to me it looked handy to keep the way credentials were kept on a per-user basis in a config file, and hence I made the new backend search for the username (email) and password in a $HOME/.megav2rc file, which follows the usual "megatools" .megarc syntax, of course, as long as you don't want your password to be shown on the "duplicity" target directory specification.

I have tested the attached code in a few (although not very diverse) installs running Python 2.7 and 3.7+, and they work (in Python 2.7 I had to remove the reference I used to "subprocess.TimeoutExpired" as a means to timeout the backend upon MEGA session command not returning). Anyways, a couple more details are mentioned within the README file in the attachment.

Note due to my lack of prior coding experience and command of version control software, I just send an attachment with the new "megav2backend.py" backend implementation, a README with a couple notes, the extended manpage (duplicity.1), and a couple other manually modified files where references to the backend seem to be made in the code (but for the backend to work, I just needed to drop the new "megav2backend.py" under /usr/lib/python3/dist-packages/duplicity/backends/ in my current installs.

If you feel this is a valuable addition to "duplicity" and if you'd like more details about the code (I highly doubt it :) )are necessary, just reply to this Bug report. I will do as possible to satisfy the request so the new backend can be added to mainline for others to benefit.

Thank you.

Changed in duplicity:
status: New → In Progress
assignee: nobody → Kenneth Loafman (kenneth-loafman)
importance: Undecided → Medium
milestone: none → 0.8.14

Were the changes based on the 0.7 series or the 0.8 series?

0.7 is EOL due to Python 2 going EOL. 0.8 is the current development series.

Hi, attached "patch" has been created on top of Ubuntu 20.04 LTS "duplicity" 0.8.11.1612-1 package, and running Python 3.8.2.

But it has also been tested and is working in the same way on Ubuntu 19.10 x86_64 with Python 3.7.5, on top of Ubuntu's "duplicity 0.8.04-2ubuntu1" package.

Further to this, used the same backend on a Raspbian GNU/Linux 10 armhf with Python 2.7.16 , on top of Raspbian's "duplicity 0.7.18.2-1" package, and other than removing the reference to "subprocess.check_output / timeout" and "subprocess.TimeoutExpired" (which don't exist in Python earlier than 3.3) the backend also works.

Finally, since the time I submitted this Bug, I made a minor change to the code, to discard "mega-login" progress output, which looks as simple as this:
"""
--- megav2backend.py 2020-05-09 11:05:52.382811192 +0200
+++ megav2backend_v1.1.py 2020-05-17 19:36:23.891145593 +0200
@@ -150,7 +150,7 @@
         except Exception as e:
             cmd = ['mega-login', self._username, self._password]
             try:
- subprocess.check_output(cmd)
+ subprocess.check_output(cmd, stderr=subprocess.DEVNULL)
             except Exception as e:
                 raise BackendException("Could not log in to MEGA, error : '%s'" % (e,))
"""

Other than that, the only "restriction" I have found in the submitted backend is, to workaround "mega-session" sometimes not returning at all, the "mega_login" function does a "subprocess.check_output" with a hardcoded 30 second timeout (timeout=30), which is only available for Python 3.3 and later as far as I know. But I guess at this time in history, Python 3.3 is not precisely bleeding edge and hence will not result in a problem for people resorting to this backend for storing duplicity backups. Can't remember if "stderr=subprocess.DEVNULL" is Python 3.3+ or Python 3.5+ though.

The code has moved on a lot since those versions. In fact, the removal of the unicode adornment on all the strings is problematic. It causes all sorts of problems with diffing the files and trying to isolate your changes. I've attached a sample diff to show point.

As to Python 3.3, duplicity has only been tested to work on Python 3.6 and above. All of the ones before that have oddball unicode issues.

The current source resides at https://gitlab.com/duplicity/duplicity. Given the number of incompatible changes, I'm going to need some help. If you could checkout that code and apply your changes, I would really appreciate it, otherwise, it will take a long while.

Changed in duplicity:
importance: Medium → Low
milestone: 0.8.14 → none

Diff between commandline.py versions.

No problem, give me some time to adapt the backend based on the latest code as in Gitlab and will post here once I have it tested on my computers for a few days.

I think I have it. It took more time to workaround several issues with my Ubuntu install which were preventing the Debian package from being created than doing the changes itself and testing.

Testing was done with very short-lived backups (to avoid having to wait for days), but it should make no difference. Testing proved two problems (bugs) with the new backend, which I have now hopefully fixed:
- Missing "decode()" for "_delete", and the usual mismatch between b'' and u''
- Had to use "-f" for "mega-rm", or else deleted files end up in the MEGA account Trash, which is obviously not correct for a backup backend

Diff attached was created against:
"""
commit ff4fb49512e3593156f348bc1306f308303d69e0 (origin/master, origin/HEAD)
Author: Kenneth Loafman <email address hidden>
Date: Wed May 20 15:22:37 2020 -0500
"""

Please have a look into it and tell me if more work on this is necessary. Developed and tested on Ubuntu 20.04 LTS running x86_64 and Python 3.8.2.

Looks good! I just committed to GitLab after making some minor code style changes.

Thanks much for the hard work!

To you for the software and the time to make the code adhere to proper coding style.

Doing further testing and using MEGA free tier run out of space and hence I tought it would be good to properly handle these situations from the code raising a backend exception when occurring.

So I caught the error on "upload()"ing any file and returned error and raised backend exception, printing the corresponding error message.

Did for both the old MEGA and the new MEGAv2 backends and tested. Seems to work fine.

Please apply attached patch against (hopefully) current head.

Thank you.

Changed in duplicity:
status: In Progress → Fix Committed
assignee: Kenneth Loafman (kenneth-loafman) → nobody
milestone: none → 0.8.14
Changed in duplicity:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers