Unknown error while uploading duplicity-full-signatures

[jeanmarc]

Hi,
I'am using Déjà dup since few days. I found it really convenient BTW :)
So, i was uploading my files since days and it finally reach 100% this morning. It was now uploading a duplicity-full-signatures file and i had to stop at 1,3G to continue later on.

Now when i restart Déjà-Dup i have an 'unknown error'. I suppose it's a BUG.. I attached the log and config file

Here my version :
deja-dup 11.1-0ubuntu0karmic1
duplicity 0.6.06-0ubuntu0karmic1
Description: Ubuntu 9.10

thanks indeed !

[Michael Terry (mterry)]

Definitely a bug. That's an odd error, though, because it indicates that duplicity exited with error 1. That shouldn't happen in modern duplicity, as any fatal error has a special code... Ken, do you know how duplicity might return code 1?

[Kenneth Loafman (kenneth-loafman)]

Michael Terry wrote:
> Definitely a bug. That's an odd error, though, because it indicates
> that duplicity exited with error 1. That shouldn't happen in modern
> duplicity, as any fatal error has a special code... Ken, do you know
> how duplicity might return code 1?

That's an EPERM error, permissions. All of the sys.exit() calls return
either 0 (no error), or a well-defined error number defined in log.py.
Except for one 'generic' error return, which would have produced a log
message anyway, I just don't see right off where it's coming from.

At this point, duplicity has not logged much since it is still trying to
determine the collection status, so there's not much to go on.

Does the user have read/write access to all of the files?

Try running 'duplicity cleanup --force <target>' to clean up the target
directory.

[jeanmarc (jeanmarc77)]

Hi,
I have permissions on the server :

jeanmarc@data2:~/backupduplicity$ pwd
/home/jeanmarc/backupduplicity

ls -l
(..)
-rw------- 1 jeanmarc jeanmarc 10518713 2009-12-28 11:58 duplicity-full.20091226T083713Z.vol999.difftar.gpg
-rw------- 1 jeanmarc jeanmarc 10535797 2009-12-26 09:24 duplicity-full.20091226T083713Z.vol99.difftar.gpg
-rw------- 1 jeanmarc jeanmarc 10515257 2009-12-26 04:00 duplicity-full.20091226T083713Z.vol9.difftar.gpg
-rw-r--r-- 1 jeanmarc jeanmarc 1410203648 2010-01-03 08:58 duplicity-full-signatures.20091226T083713Z.sigtar.gpg

I tired to run the cleanup without success :( :

jeanmarc@plop:~$ duplicity cleanup --force scp://jeanmarc:<email address hidden>:22/home/jeanmarc/backupduplicity/
GnuPG passphrase:
Remote file or directory does not exist in command='mkdir home/jeanmarc/backupduplicity/'
Running 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>' failed (attempt #1)
Remote file or directory does not exist in command='mkdir home/jeanmarc/backupduplicity/'
Running 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>' failed (attempt #2)
Remote file or directory does not exist in command='mkdir home/jeanmarc/backupduplicity/'
Running 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>' failed (attempt #3)
Remote file or directory does not exist in command='mkdir home/jeanmarc/backupduplicity/'
Running 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>' failed (attempt #4)
Remote file or directory does not exist in command='mkdir home/jeanmarc/backupduplicity/'
Running 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>' failed (attempt #5)
Giving up trying to execute 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>' after 5 attempts
BackendException: Error running 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>'

That's weird, the directory exist, i'am using duplicity 0.6.06...
thanks

[Kenneth Loafman (kenneth-loafman)]

jeanmarc wrote:
> Hi,
> I have permissions on the server :
>
> jeanmarc@data2:~/backupduplicity$ pwd
> /home/jeanmarc/backupduplicity
>
> ls -l
> (..)
> -rw------- 1 jeanmarc jeanmarc 10518713 2009-12-28 11:58 duplicity-full.20091226T083713Z.vol999.difftar.gpg
> -rw------- 1 jeanmarc jeanmarc 10535797 2009-12-26 09:24 duplicity-full.20091226T083713Z.vol99.difftar.gpg
> -rw------- 1 jeanmarc jeanmarc 10515257 2009-12-26 04:00 duplicity-full.20091226T083713Z.vol9.difftar.gpg
> -rw-r--r-- 1 jeanmarc jeanmarc 1410203648 2010-01-03 08:58 duplicity-full-signatures.20091226T083713Z.sigtar.gpg
>
> I tired to run the cleanup without success :( :
>
> jeanmarc@plop:~$ duplicity cleanup --force scp://jeanmarc:<email address hidden>:22/home/jeanmarc/backupduplicity/
> GnuPG passphrase:
> Remote file or directory does not exist in command='mkdir home/jeanmarc/backupduplicity/'
> Running 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>' failed (attempt #1)
> Remote file or directory does not exist in command='mkdir home/jeanmarc/backupduplicity/'
> Running 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>' failed (attempt #2)
> Remote file or directory does not exist in command='mkdir home/jeanmarc/backupduplicity/'
> Running 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>' failed (attempt #3)
> Remote file or directory does not exist in command='mkdir home/jeanmarc/backupduplicity/'
> Running 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>' failed (attempt #4)
> Remote file or directory does not exist in command='mkdir home/jeanmarc/backupduplicity/'
> Running 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>' failed (attempt #5)
> Giving up trying to execute 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>' after 5 attempts
> BackendException: Error running 'sftp -oPort=22 -oServerAliveInterval=15 -oServerAliveCountMax=1 <email address hidden>'
>
> That's weird, the directory exist, i'am using duplicity 0.6.06...
> thanks

The URL should be ...:22//home/jeanmarc/... two slashes to make it an
absolute pathname.

[jeanmarc (jeanmarc77)]

Hi,
Thanks for the clarification. I tried but it seem like the clean up delete all my files excepted the signature :/
Now i have to upload again GB of data..
Sorry if i couldn't be more helpful, you may close the ticket if you want.

Thanks

[Michael Terry (mterry)]

Is it now working in Deja Dup or just on the command line?

[jeanmarc (jeanmarc77)]

It will takes days before i upload again my data.
I tried a few MB and déjà dup didn't make any errors.

[jeanmarc (jeanmarc77)]

Hi,
I've just finished today to upload my several GB and i got the same 'unknown error' on déjà dup.
I've upload around 10GB of data and the full-signature stop (for an unknow reason) at 1,3GB.

I don't know what may be wrong except that the bandwidth with my provider (datastorageunit.com) is pretty slow.
Now i'am gonna run again the "duplicity cleanup" command and i'll keep you inform.

Bye

[jeanmarc (jeanmarc77)]

I finally got this error :

Synchronizing remote metadata to local cache...
Copying duplicity-full-signatures.20100106T163303Z.sigtar to local cache.
GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: données chiffrées avec CAST5
gpg: chiffré avec 1 phrase de passe
gpg: block_filter 0x2158420: read error (size=11740,a->size=11740)
gpg: [don't know]: invalid packet (ctb=57)
gpg: packet(5) with unknown version 46
gpg: mdc_packet with invalid encoding
gpg: le déchiffrement a échoué: paquet invalide
gpg: packet(14) with unknown version 19
gpg: block_filter: pending bytes!
gpg: AVERTISSEMENT: le propriétaire du fichier de configuration `/home/jeanmarc/.gnupg/gpg.conf'
est peu sûr
===== End GnuPG log =====

[Michael Terry (mterry)]

Silly babelfish translation:

gpg: statistical data with CAST5
gpg: quantified with 1 sentence of master key
gpg: block_filter 0x2158420: read error (size=11740, a-> size=11740)
gpg: [don' T know]: invalid packet (ctb=57)
gpg: packet (5) with unknown version 46
gpg: mdc_packet with invalid encoding
gpg: the deciphering failed: invalid package
gpg: packet (14) with unknown version 19
gpg: block_filter: pending bytes!
gpg: WARNING: the owner of the file of configuration `/home/jeanmarc/.gnupg/gpg.conf' is not very sure

Doesn't look familiar to me...

[Moritz Maisel (mail-maisel)]

I am also facing the "unknown error" problem. My setup is:
deja-dup 14.0.1-0ubuntu0karmic1
duplicity 0.6.08-0ubuntu0karmic1
Ubuntu karmic

After failing to find some logfile I ran "deja-dup --backup" on the commandline which caused the (same) error:
----8<----*snip*----
** (deja-dup:10556): DEBUG: DuplicityInstance.vala:552: duplicity (10618) exited with value 1
----8<----*snip*----

Then I ran the logged duplicity command directly in the commandline which failed with the following last lines:
----8<----*snip*----
No orphaned or incomplete backup sets found.
Removing still remembered temporary file /tmp/duplicity-4Q3Kri-tempdir/mkstemp-qHIqk9-1
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1239, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1232, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1214, in main
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 474, in incremental_backup
    assert dup_time.curtime != dup_time.prevtime, "time not moving forward at appropriate pace - system clock issues?"
AssertionError: time not moving forward at appropriate pace - system clock issues?
----8<----*snip*----

As deja dup complained "duplicity (10618) exited with value 1" I supose my problem might be related to this bug. If you need more debug information or want me to file an extra bug report, please let me know ...

BTW: Thanks for this great tool! :-)

[Olivier Berger (olivierberger)]

Hmmm.

It seems to me that @Moritz_Maisel did not experience the very same problem as @jeanmarc.

Maybe would be worth reporting separate issues on duplicity ?

[Olivier Berger (olivierberger)]

Actually, I got confused by the last comment reporting another error than the first problem, so this link to Debian's BTS ought to be removed... but I cannot seem to know how.

[Dirk Leinenbach (j-launchpad-dirkleinenbach-de)]

Hi Moritz,

I got the same duplicity error message (I'm using duplicity directly not via deja-dup):

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1251, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1244, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1226, in main
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 474, in incremental_backup
    assert dup_time.curtime != dup_time.prevtime, "time not moving forward at
appropriate pace - system clock issues?"
AssertionError: time not moving forward at appropriate pace - system clock
issues?
Local and Remote metadata are synchronized, no sync needed.

It turned out that the problem was caused by two .part files in duplicities local archive:

duplicity-inc.20100501T230102Z.to.20100502T230102Z.manifest.part
duplicity-new-signatures.20100501T230102Z.to.20100502T230102Z.sigtar.part

deleting these files solved the problem for me

[Olivier Berger (olivierberger)]

Hi.

I have created a new ticket about the "time not moving forward at appropriate pace - system clock issues?" problem, as there's 2 different problems discussed here :
https://bugs.launchpad.net/duplicity/+bug/579958

Please report to this other ticket about the time issues.

[Kenneth Loafman (kenneth-loafman)]

Any assert will now cause a full exception FataError termination. No more unknown errors.

[ben (ben-chgtaa3qpp)]

I got the same problem. I am not sure if anyone got the fixed:

==============log===========
Synchronising remote metadata to local cache...
Copying duplicity-full-signatures.20120710T022345Z.sigtar.gpg to local cache.
Registering (mktemp) temporary file /tmp/duplicity-EslPma-tempdir/mktemp-BTbPX3-2
Reading results of 'ncftpget -f /tmp/duplicity-EslPma-tempdir/mkstemp-8iEyK1-1 -F -t 30 -o useCLNT=0,useHELP_SITE=0 -V -C 'ftpserver' 'folder/duplicity-full-signatures.20120710T022345Z.sigtar.gpg' '/tmp/duplicity-EslPma-tempdir/mktemp-BTbPX3-2''
Registering (mktemp) temporary file /tmp/duplicity-EslPma-tempdir/mktemp-C5sa5l-3
Removing still remembered temporary file /tmp/duplicity-EslPma-tempdir/mktemp-C5sa5l-3
Removing still remembered temporary file /tmp/duplicity-EslPma-tempdir/mktemp-BTbPX3-2
Removing still remembered temporary file /tmp/duplicity-EslPma-tempdir/mkstemp-8iEyK1-1
GPG error detail: Traceback (most recent call last):
  File "/usr/local/bin/duplicity", line 1391, in <module>
    with_tempdir(main)
  File "/usr/local/bin/duplicity", line 1384, in with_tempdir
    fn()
  File "/usr/local/bin/duplicity", line 1260, in main
    sync_archive(decrypt)
  File "/usr/local/bin/duplicity", line 1060, in sync_archive
    copy_to_local(fn)
  File "/usr/local/bin/duplicity", line 1007, in copy_to_local
    gpg.GzipWriteFile(src_iter, tdp.name, size=sys.maxint)
  File "/usr/lib64/python2.6/site-packages/duplicity/gpg.py", line 372, in GzipWriteFile
    new_block = block_iter.next(min(128*1024, bytes_to_go))
  File "/usr/local/bin/duplicity", line 990, in next
    self.fileobj.close()
  File "/usr/lib64/python2.6/site-packages/duplicity/dup_temp.py", line 222, in close
    assert not self.fileobj.close()
  File "/usr/lib64/python2.6/site-packages/duplicity/gpg.py", line 239, in close
    self.gpg_failed()
  File "/usr/lib64/python2.6/site-packages/duplicity/gpg.py", line 206, in gpg_failed
    raise GPGError, msg
GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: encrypted with 2048-bit ELG key, ID 12345667, created 2012-07-09
"Duplicity Backup (gpg key for duplicity backup) <email address hidden>"
gpg: block_filter 0xe631e0: read error (size=10898,a->size=10898)
gpg: block_filter 0xe63320: read error (size=9792,a->size=9792)
gpg: mdc_packet with invalid encoding
gpg: decryption failed: Invalid packet
gpg: block_filter: pending bytes!
gpg: block_filter: pending bytes!
===== End GnuPG log =====

GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: encrypted with 2048-bit ELG key, ID 23424234, created 2012-07-09
"Duplicity Backup (gpg key for duplicity backup) <email address hidden>"
gpg: block_filter 0xe631e0: read error (size=10898,a->size=10898)
gpg: block_filter 0xe63320: read error (size=9792,a->size=9792)
gpg: mdc_packet with invalid encoding
gpg: decryption failed: Invalid packet
gpg: block_filter: pending bytes!
gpg: block_filter: pending bytes!
===== End GnuPG log =====

[ben (ben-chgtaa3qpp)]

and my version is :

Name : duplicity Relocations: (not relocatable)
Version : 0.6.18 Vendor: Fedora Project
Release : 1.el6 Build Date: Sun 08 Apr 2012 21:52:32 BST