Faulty OK with failing mega backend

[Michael]

Duplicity 0.7.16
Megatools 1.9.97
Xubuntu 16.04.3

I have found some problem while using the mega backend.
1) Not possible to specify user (mega://user.name%<email address hidden>/Folder) and FTP_PASSWORD, it always try to use the .megarc file.
2) Folder at cloud not automatic created.
3) OK even if mega backend commands fails! MAJOR
4) This is written in megabackend.py: """Connect to remote store using Mega.co.nz API"""
   However it isn't true. The megatools are used.
5) If no files found in backend -> "Local and Remote metadata are synchronized, no sync needed."
   How can it be in sync if no files found at one end.

Example:
"
$ PASSPHRASE="abc" duplicity --include '/home/user/folder/**' --exclude '**' /home/user mega://mega.nz/Tmp2
mkdir: Tmp2
megals: /Root/Tmp2
Local and Remote metadata are synchronized, no sync needed.
megals: /Root/Tmp2
Last full backup date: none
No signatures found, switching to full backup.
megarm: duplicity-full.20180127T123518Z.vol1.difftar.gpg
megaput: duplicity-full.20180127T123518Z.vol1.difftar.gpg
megarm: duplicity-full-signatures.20180127T123518Z.sigtar.gpg
megaput: duplicity-full-signatures.20180127T123518Z.sigtar.gpg
megarm: duplicity-full.20180127T123518Z.manifest.gpg
megaput: duplicity-full.20180127T123518Z.manifest.gpg
megals: /Root/Tmp2
--------------[ Backup Statistics ]--------------
StartTime 1517056520.00 (Sat Jan 27 13:35:19 2018)
EndTime 1517056520.57 (Sat Jan 27 13:35:20 2018)
ElapsedTime 0.57 (0.57 seconds)
SourceFiles 676
SourceFileSize 11869399 (11.3 MB)
NewFiles 676
NewFileSize 11869399 (11.3 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 676
RawDeltaSize 11599063 (11.1 MB)
TotalDestinationSizeChange 3335818 (3.18 MB)
Errors 0
-------------------------------------------------
$ echo $?
0
"

Note that all seems to be okay above. However no files have been uploaded to the cloud, none!

It turns out that the megatools is bad. However bugs have been fixed since the version in Ubuntu (1.9.97). This version of the tools always return 0 (OK). Latest source code doesn't.

So the biggest problem is the old and bad megatools. However I think duplicity should include some kind of check for this major fault. I notice duplicity do a _list in backend after put up files. Maybe it could be possible to use this answer (list of files) and check that the files that was just uploaded actually is in the list.

Here is a proposal for one quick check that can be added to discover when megatools fails. Check that megals have got an answer from the server. The "/Root" folder should always be there or the folder that is given as argument, if it exist.

I have done some changes in megabackend.py
"
--- duplicity/backends/megabackend.716 2018-01-27 14:02:02.675823111 +0100
+++ duplicity/backends/megabackend.py 2018-01-27 17:51:07.203823111 +0100
@@ -28,7 +28,7 @@

 class MegaBackend(duplicity.backend.Backend):
- """Connect to remote store using Mega.co.nz API"""
+ """Connect to MEGA cloud (mega.nz) using Megatools"""

     def __init__(self, parsed_url):
         duplicity.backend.Backend.__init__(self, parsed_url)
@@ -128,6 +128,10 @@
         files = subprocess.check_output(cmd)
         files = files.strip().split('\n')

+ # ensure communication with server
+ if self._folder not in files:
+ raise BackendException("folder '%s' not found or communication error!" % (self._folder,))
+
         # remove the folder name, including the path separator
         files = [f[len(self._folder) + 1:] for f in files]

@@ -161,7 +165,12 @@
             cmd = ['megaput', '-u', self._username, '-p', self._password, '--no-progress',
                    '--path', self._folder + '/' + remote_file, local_file]

- self.subprocess_popen(cmd)
+ result = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+ result = result.rstrip()
+ if result:
+ print (result)
+ if "ERROR" in result:
+ raise BackendException("error from megaput!")

     def delete(self, remote_file):

"

Note: It can be discussed if check for "ERROR" string should be done or check for not empty result string. I don't know if all kinds of faults print ERROR.

I have done some tests, here are printouts from failed executions (with my patch)...

megals 1.9.97 fails (this is the same typ of execution as above):
"
mkdir: Test22
megals: /Root/Test22
Attempt 1 failed. BackendException: folder '/Root/Test22' not found or communication error!
"

megaput 1.9.97 fails:
"
mkdir: Test22
megals: /Root/Test22
Local and Remote metadata are synchronized, no sync needed.
megals: /Root/Test22
Last full backup left a partial set, restarting.
Last full backup date: Sat Jan 27 17:50:51 2018
RESTART: The first volume failed to upload before termination.
         Restart is impossible...starting backup from beginning.
mkdir: Test22
megals: /Root/Test22
Local and Remote metadata are synchronized, no sync needed.
megals: /Root/Test22
Last full backup date: none
No signatures found, switching to full backup.
megarm: duplicity-full.20180128T122438Z.vol1.difftar.gpg
megaput: duplicity-full.20180128T122438Z.vol1.difftar.gpg
ERROR: Upload failed for '/tmp/duplicity-UppYZF-tempdir/mktemp-a2MlX5-2': Parent directory doesn't exist: /Root/Test22
Attempt 1 failed. BackendException: error from megaput!
"

megals 1.9.98 fails:
"
mkdir: Test22
megals: /Root/Test22
Attempt 1 failed. BackendException: folder '/Root/Test22' not found or communication error!
"

megaput 1.9.98 fails:
"
mkdir: Test22
megals: /Root/Test22
Local and Remote metadata are synchronized, no sync needed.
megals: /Root/Test22
Last full backup left a partial set, restarting.
Last full backup date: Sun Jan 28 13:24:38 2018
RESTART: The first volume failed to upload before termination.
         Restart is impossible...starting backup from beginning.
mkdir: Test22
megals: /Root/Test22
Local and Remote metadata are synchronized, no sync needed.
megals: /Root/Test22
Last full backup date: none
No signatures found, switching to full backup.
megarm: duplicity-full.20180128T130311Z.vol1.difftar.gpg
megaput: duplicity-full.20180128T130311Z.vol1.difftar.gpg
Attempt 1 failed. CalledProcessError: Command '['megaput', '--config', '/home/surf/.megarc', '--no-progress', '--path', '/Root/Test22/duplicity-full.20180128T130311Z.vol1.difftar.gpg', '/tmp/duplicity-94eemj-tempdir/mktemp-7Vui_I-2']' returned non-zero exit status 1
"

I think both old and new megatools should be supported. However maybe it can depend on which versions will be included in apt repos.

Now I test with duplicity 0.7.16 (without my patch) and newer megatools 1.9.98:
"
mkdir: Test22
megals: /Root/Test22
Local and Remote metadata are synchronized, no sync needed.
megals: /Root/Test22
Last full backup left a partial set, restarting.
Last full backup date: Sun Jan 28 14:03:11 2018
RESTART: The first volume failed to upload before termination.
         Restart is impossible...starting backup from beginning.
mkdir: Test22
megals: /Root/Test22
Local and Remote metadata are synchronized, no sync needed.
megals: /Root/Test22
Last full backup date: none
No signatures found, switching to full backup.
megarm: duplicity-full.20180128T130740Z.vol1.difftar.gpg
megaput: duplicity-full.20180128T130740Z.vol1.difftar.gpg
Attempt 1 failed. BackendException: Error running 'megaput --config /home/surf/.megarc --no-progress --path /Root/Test22/duplicity-full.20180128T130740Z.vol1.difftar.gpg /tmp/duplicity-tn9xZW-tempdir/mktemp-eXq1NT-2': returned 1, with output:

ERROR: Upload failed for '/tmp/duplicity-tn9xZW-tempdir/mktemp-eXq1NT-2': Parent directory doesn't exist: /Root/Test22
"

So here we actually get an error, however not on megals.

PS. Please excuse me, I'm a C/C++ developer and have never written python code. So I don't feel comfortable doing this. I hope some experianced python developer can check this out. Sorry, but I haven't made proposal to fix all problems I listed at top.

//Michael

[David Parkin (davidmichaelparkin)]

I think mega://user.name%<email address hidden>/Folder should be mega://user.name%40gmail.com:@mega.nz/Folder - colon after .com and before @mega.