Ubuntu
backupninja package

duplicity jobs not working

Bug #126417 reported by lstewart on 2007-07-16

Affects		Status	Importance	Assigned to	Milestone
	backupninja (Debian)	Fix Released	Unknown	debbugs #458816
	backupninja (Ubuntu)	Fix Released	Low	Unassigned

Bug Description

Binary package hint: backupninja

I get these error messages in the logs when there is already some data there:
Jul 15 01:24:20 Debug: duplicity --no-print-statistics --scp-command 'scp ' --ssh-command 'ssh ' --encrypt-key 83AF64BC --sig
n-key 83AF64BC --remove-older-than 60D --include '/home/lstewart/bin/**' --exclude '**' / scp://<email address hidden>/
/home/lstewart/backup/nuzetta
Jul 15 01:24:50 Debug: Traceback (most recent call last): File "/usr/bin/duplicity", line 373, in <module> if __name__ == "__m
ain__": main() File "/usr/bin/duplicity", line 362, in main check_last_manifest(col_stats) File "/usr/bin/duplicity", line 342
, in check_last_manifest last_backup_set.check_manifests() File "/usr/lib/python2.5/site-packages/duplicity/collections.py", l
ine 128, in check_manifests if local_manifest: remote_manifest = local_manifest UnboundLocalError: local variable 'local_manif
est' referenced before assignment

And this one when we are looking at the initial backup:
Jul 15 01:26:59 Debug: duplicity --no-print-statistics --scp-command 'scp ' --ssh-command 'ssh ' --encrypt-key 83AF64BC --sign-key 83AF64BC --remove-older-than 60D --include '/home/lstewart/bin/**' --exclude '**' / scp://<email address hidden>//home/lstewart/backup/nuzetta
Jul 15 01:27:12 Debug: No signatures found, switching to full backup. Traceback (most recent call last): File "/usr/bin/duplicity", line 373, in <module> if __name__ == "__main__": main() File "/usr/bin/duplicity", line 366, in main if not sig_chain: full_backup(col_stats) File "/usr/bin/duplicity", line 142, in full_backup bytes_written = write_multivol("full", tarblock_iter, globals.backend) File "/usr/bin/duplicity", line 78, in write_multivol globals.gpg_profile) File "/usr/lib/python2.5/site-packages/duplicity/gpg.py", line 213, in GPGWriteFile try: data = block_iter.next(bytes_to_go).data File "/usr/lib/python2.5/site-packages/duplicity/diffdir.py", line 407, in next result = self.process(self.input_iter.next(), size) File "/usr/lib/python2.5/site-packages/duplicity/diffdir.py", line 277, in get_delta_iter_w_sig (new_path, sig_path, sigTarFile)) File "/usr/lib/python2.5/site-packages/duplicity/robust.py", line 31, in check_common_error try: return function(*args) File "/usr/lib/python2.5/site-packages/duplicity/diffdir.py", line 309, in get_delta_path_w_sig if not new_path.isreg(): sigTarFile.addfile(ti) File "/usr/lib/python2.5/site-packages/duplicity/tarfile.py", line 762, in addfile self.fileobj.write(full_headers) File "/usr/lib/python2.5/site-packages/duplicity/dup_temp.py", line 120, in write return self.fileobj.write(buf) File "/usr/lib/python2.5/site-packages/duplicity/gpg.py", line 103, in write def write(self, buf): return self.gpg_input.write(buf) IOError: [Errno 32] Broken pipe close failed: [Errno 32] Broken pipe

I know what you're gonna say: its a duplicity problem. The problem with that is that duplicity works with the same arguments from the command line, and in a simple scheduled cron job (what I've fallen back to). Everyone knows duplicity is a little bit unstable, but I think that this is really some issue with the environment the duplicity command is being executed in or something.

Revision history for this message

Marco Rodrigues (gothicx) wrote on 2007-12-12:

Do you still have this problem ? You're using Gutsy ?

Changed in backupninja:
importance:	Undecided → Low
assignee:	nobody → gothicx
status:	New → Incomplete

Revision history for this message

lstewart (agrodellic) wrote on 2007-12-13:

Download full text (3.3 KiB)

Nope. Now I get what you see below. I abandoned backup ninja a while back and went with my shell scripts, and then just last weekend I tried backupninja again, but had little luck.

Yep I'm using Gutsy... You want me to file a different report for this one?

Also I had gpg dump a list of available keys ( gpg --list-secret-keys ), and the required key shows up. Also, as with the previous problem, the duplicity command that backupninja spits out works fine from the command line, as-is.

Cheers!

Dec 12 23:14:37 Info: >>>> starting action /etc/backup.d/action.dup (because of --now)
Dec 12 23:14:37 Debug: yes
Dec 12 23:14:37 Debug: ssh -o IdentityFile=/home/lstewart/.ssh/id_dsa -o PasswordAuthentication=no mail.sevaserver.org -l lstewart 'echo -n 1'
Dec 12 23:14:38 Debug: Connected to mail.sevaserver.org as lstewart successfully
Dec 12 23:14:39 Debug: Data will be encrypted with the GnuPG key 83AF64BC.
Dec 12 23:14:39 Debug: Data won't be signed.
Dec 12 23:14:39 Debug: duplicity -v9 --allow-source-mismatch --no-print-statistics --scp-command 'scp -o IdentityFile=/home/lstewart/.ssh/id_dsa' --sftp-command 'sftp -o IdentityFile=/home/lstewart/.ssh/id_dsa' --ssh-options '-o IdentityFile=/home/lstewart/.ssh/id_dsa' --encrypt-key 83AF64BC --remove-older-than 60D --exclude '/' --include '/home/lstewart/bin/**' --exclude '**' / scp://<email address hidden>//home/lstewart/backup/nuzetta
Dec 12 23:14:42 Debug: Main action: inc Running 'sftp -o IdentityFile=/home/lstewart/.ssh/id_dsa -o IdentityFile=/home/lstewart/.ssh/id_dsa <email address hidden>' (attempt #1) Warning, found incomplete backup sets, probably left from aborted session Collection Status ----------------- Connecting with backend: sshBackend Archive dir: None No backup chains with active signatures found Found 0 backup chains without signatures. Also found 0 backup sets not part of any chain, and 3 incomplete backup sets. These may be deleted by running duplicity with the --cleanup option. No signatures found, switching to full backup. Selecting / Comparing () and None Getting delta of (() / dir) and None Generating delta - new file: . gpg: 83AF64BC: skipped: public key not found gpg: [stdin]: encryption failed: public key not found gpg: 83AF64BC: skipped: public key not found gpg: [stdin]: encryption failed: public key not found Traceback (most recent call last): File "/usr/bin/duplicity", line 394, in <module> if __name__ == "__main__": main() File "/usr/bin/duplicity", line 387, in main if not sig_chain: full_backup(col_stats) File "/usr/bin/duplicity", line 146, in full_backup bytes_written = write_multivol("full", tarblock_iter, globals.backend) File "/usr/bin/duplicity", line 82, in write_multivol globals.gpg_profile,globals.volsize) File "/usr/lib/python2.5/site-packages/duplicity/gpg.py", line 208, in GPGWriteFile file.close() File "/usr/lib/python2.5/site-packages/duplicity/gpg.py", line 123, in close self.gpg_process.wait() File "/var/lib/python-support/python2.5/GnuPGInterface.py", line 639, in wait raise IOError, "GnuPG exited non-zero, with code %d" % (e << 8) IOError: GnuPG exited non-zero, with code 131072
Dec 12 23:14:42 Fatal: Duplicity failed.
Dec 12 23:14:42...

Nope. Now I get what you see below. I abandoned backup ninja a while back and went with my shell scripts, and then just last weekend I tried backupninja again, but had little luck.

Yep I'm using Gutsy... You want me to file a different report for this one?

Also I had gpg dump a list of available keys (  gpg --list-secret-keys ), and the required key shows up. Also, as with the previous problem, the duplicity command that backupninja spits out works fine from the command line, as-is.

Cheers!

Dec 12 23:14:37 Info: >>>> starting action /etc/backup.d/action.dup (because of --now)
Dec 12 23:14:37 Debug: yes
Dec 12 23:14:37 Debug: ssh -o IdentityFile=/home/lstewart/.ssh/id_dsa -o PasswordAuthentication=no mail.sevaserver.org -l lstewart 'echo -n 1'
Dec 12 23:14:38 Debug: Connected to mail.sevaserver.org as lstewart successfully
Dec 12 23:14:39 Debug: Data will be encrypted with the GnuPG key 83AF64BC.
Dec 12 23:14:39 Debug: Data won't be signed.
Dec 12 23:14:39 Debug: duplicity -v9 --allow-source-mismatch  --no-print-statistics --scp-command 'scp -o IdentityFile=/home/lstewart/.ssh/id_dsa' --sftp-command 'sftp -o IdentityFile=/home/lstewart/.ssh/id_dsa' --ssh-options '-o IdentityFile=/home/lstewart/.ssh/id_dsa' --encrypt-key 83AF64BC --remove-older-than 60D --exclude '/' --include '/home/lstewart/bin/**'  --exclude '**' / scp://lstewart@mail.sevaserver.org//home/lstewart/backup/nuzetta
Dec 12 23:14:42 Debug: Main action: inc Running 'sftp -o IdentityFile=/home/lstewart/.ssh/id_dsa -o IdentityFile=/home/lstewart/.ssh/id_dsa lstewart@mail.sevaserver.org' (attempt #1) Warning, found incomplete backup sets, probably left from aborted session Collection Status ----------------- Connecting with backend: sshBackend Archive dir: None No backup chains with active signatures found Found 0 backup chains without signatures. Also found 0 backup sets not part of any chain, and 3 incomplete backup sets. These may be deleted by running duplicity with the --cleanup option. No signatures found, switching to full backup. Selecting / Comparing () and None Getting delta of (() / dir) and None Generating delta - new file: . gpg: 83AF64BC: skipped: public key not found gpg: [stdin]: encryption failed: public key not found gpg: 83AF64BC: skipped: public key not found gpg: [stdin]: encryption failed: public key not found Traceback (most recent call last): File "/usr/bin/duplicity", line 394, in <module> if __name__ == "__main__": main() File "/usr/bin/duplicity", line 387, in main if not sig_chain: full_backup(col_stats) File "/usr/bin/duplicity", line 146, in full_backup bytes_written = write_multivol("full", tarblock_iter, globals.backend) File "/usr/bin/duplicity", line 82, in write_multivol globals.gpg_profile,globals.volsize) File "/usr/lib/python2.5/site-packages/duplicity/gpg.py", line 208, in GPGWriteFile file.close() File "/usr/lib/python2.5/site-packages/duplicity/gpg.py", line 123, in close self.gpg_process.wait() File "/var/lib/python-support/python2.5/GnuPGInterface.py", line 639, in wait raise IOError, "GnuPG exited non-zero, with code %d" % (e << 8) IOError: GnuPG exited non-zero, with code 131072
Dec 12 23:14:42 Fatal: Duplicity failed.
Dec 12 23:14:42 Fatal: <<<< finished action /etc/backup.d/action.dup: FAILED
Dec 12 23:14:42 Debug: send report to agrodellic@gmail.com
Dec 12 23:14:42 Info: FINISHED: 1 actions run. 1 fatal. 0 error. 0 warning.

Revision history for this message

lstewart (agrodellic) wrote on 2007-12-13:

"Also I had gpg dump a list of available keys ( gpg --list-secret-keys ), and the required key shows up. Also, as with the previous problem, the duplicity command that backupninja spits out works fine from the command line, as-is."

From within the dupl action, dupl.in, or w/e.

Revision history for this message

Buttay (cyril-buttay) wrote on 2008-04-15:

Download full text (4.2 KiB)

I also have a problem running duplicity via backupninja:

on its own, duplicity works:
$ export PASSPHRASE=my_passphrase
$ duplicity --full --no-print-statistics --encrypt-key XXXXXXX --sign-key XXXXXXXX --remove-older-than 60D --exclude '/home/*/.gnupg' --exclude '/home/*/.gnupg' --exclude '/home/*/.local/share/Trash' --exclude '/home/*/.Trash' --exclude '/home/*/.thumbnails' --exclude '/home/*/.beagle' --exclude '/home/*/.aMule' --exclude '/home/*/gtk-gnutella-downloads' --exclude '/home/cyril/backups' --include '/var/spool/cron/crontabs' --include '/var/backups' --include '/etc' --include '/root' --include '/home' --include '/usr/local/*bin' --include '/var/lib/dpkg/status*' --exclude '**' / scp://login@host//directory

(the duplicity command is a copy of the one generated by the backupninja script)

however, running backupninja fails (line breaks added for readability):

backupninja --debug --run /etc/backup.d/90.dup

Debug: check_perms /etc/backup.d
Debug: perms: drwxrwx---
Debug: gperm: rwx
Debug: wperm: ---
Debug: check_perms /etc/backup.d/90.dup
Debug: perms: -rw-------
Debug: gperm: ---
Debug: wperm: ---
Info: >>>> starting action /etc/backup.d/90.dup (because of --now)
Debug: yes
Debug: ssh -o PasswordAuthentication=no host -l login 'echo -n 1'
Debug: Connected to host as login successfully
Debug: Data will be encrypted with the GnuPG key XXXXXXXX.
Debug: Data will be signed will the GnuPG key XXXXXXXX.

duplicity --full --no-print-statistics --encrypt-key XXXXXXX --sign-key XXXXXXXX --remove-older-than 60D --exclude '/home/*/.gnupg' --exclude '/home/*/.gnupg' --exclude '/home/*/.local/share/Trash' --exclude '/home/*/.Trash' --exclude '/home/*/.thumbnails' --exclude '/home/*/.beagle' --exclude '/home/*/.aMule' --exclude '/home/*/gtk-gnutella-downloads' --exclude '/home/cyril/backups' --include '/var/spool/cron/crontabs' --include '/var/backups' --include '/etc' --include '/root' --include '/home' --include '/usr/local/*bin' --include '/var/lib/dpkg/status*' --exclude '**' / scp://login@host//directory

I also have a problem running duplicity via backupninja:

on its own, duplicity works:
$ export PASSPHRASE=my_passphrase
$ duplicity  --full --no-print-statistics --encrypt-key XXXXXXX --sign-key XXXXXXXX --remove-older-than 60D --exclude '/home/*/.gnupg' --exclude '/home/*/.gnupg' --exclude '/home/*/.local/share/Trash' --exclude '/home/*/.Trash' --exclude '/home/*/.thumbnails' --exclude '/home/*/.beagle' --exclude '/home/*/.aMule' --exclude '/home/*/gtk-gnutella-downloads' --exclude '/home/cyril/backups' --include '/var/spool/cron/crontabs' --include '/var/backups' --include '/etc' --include '/root' --include '/home' --include '/usr/local/*bin' --include '/var/lib/dpkg/status*'  --exclude '**' / scp://login@host//directory

(the duplicity command is a copy of the one generated by the backupninja script)

however, running backupninja fails (line breaks added for readability):

backupninja --debug --run /etc/backup.d/90.dup

Debug: check_perms /etc/backup.d
Debug: perms: drwxrwx---
Debug: gperm: rwx
Debug: wperm: ---
Debug: check_perms /etc/backup.d/90.dup
Debug: perms: -rw-------
Debug: gperm: ---
Debug: wperm: ---
Info: >>>> starting action /etc/backup.d/90.dup (because of --now)
Debug: yes
Debug: ssh  -o PasswordAuthentication=no host -l login 'echo -n 1'
Debug: Connected to host as login successfully
Debug: Data will be encrypted with the GnuPG key XXXXXXXX.
Debug: Data will be signed will the GnuPG key XXXXXXXX.

duplicity  --full --no-print-statistics --encrypt-key XXXXXXX --sign-key XXXXXXXX --remove-older-than 60D --exclude '/home/*/.gnupg' --exclude '/home/*/.gnupg' --exclude '/home/*/.local/share/Trash' --exclude '/home/*/.Trash' --exclude '/home/*/.thumbnails' --exclude '/home/*/.beagle' --exclude '/home/*/.aMule' --exclude '/home/*/gtk-gnutella-downloads' --exclude '/home/cyril/backups' --include '/var/spool/cron/crontabs' --include '/var/backups' --include '/etc' --include '/root' --include '/home' --include '/usr/local/*bin' --include '/var/lib/dpkg/status*'  --exclude '**' / scp://login@host//directory

Debug: Warning, found incomplete backup sets, probably left from aborted session No signatures found, switching to full backup. Traceback (most recent call last): 
File "/usr/bin/duplicity", line 394, in <module> if __name__ == "__main__": main() 
File "/usr/bin/duplicity", line 387, in main if not sig_chain: full_backup(col_stats) 
File "/usr/bin/duplicity", line 146, in full_backup bytes_written = write_multivol("full", tarblock_iter, globals.backend)
File "/usr/bin/duplicity", line 82, in write_multivol globals.gpg_profile,globals.volsize) 
File "/usr/lib/python2.5/site-packages/duplicity/gpg.py", line 198, in GPGWriteFile try: data = block_iter.next(bytes_to_go).data 
File "/usr/lib/python2.5/site-packages/duplicity/diffdir.py", line 407, in next result = self.process(self.input_iter.next(), size) 
File "/usr/lib/python2.5/site-packages/duplicity/diffdir.py", line 277, in get_delta_iter_w_sig (new_path, sig_path, sigTarFile)) 
File "/usr/lib/python2.5/site-packages/duplicity/robust.py", line 31, in check_common_error try: return function(*args) 
File "/usr/lib/python2.5/site-packages/duplicity/diffdir.py", line 309, in get_delta_path_w_sig if not new_path.isreg(): sigTarFile.addfile(ti) File "/usr/lib/python2.5/site-packages/duplicity/tarfile.py", line 762, in addfile self.fileobj.write(full_headers) 
File "/usr/lib/python2.5/site-packages/duplicity/dup_temp.py", line 120, in write return self.fileobj.write(buf) 
File "/usr/lib/python2.5/site-packages/duplicity/gpg.py", line 117, in write def write(self, buf): return self.gpg_input.write(buf) IOError: [Errno 32] Broken pipe close failed: [Errno 32] Broken pipe close failed: [Errno 32] Broken pipe

Fatal: Duplicity failed.
Fatal: <<<< finished action /etc/backup.d/90.dup: FAILED
Debug: send report to cyril@localhost
df: `': Aucun fichier ou répertoire de ce type
Info: FINISHED: 1 actions run. 1 fatal. 0 error. 0 warning.

also note that to get this far, I had to modify the duplicity script located in /usr/share/backupninja, to remove  --ssh-command 'ssh $sshoptions'  from lines 86 and 88, as this generated an error message because this option is not recognized (nor is it mentioned in the duplicity documentation)

Revision history for this message

Buttay (cyril-buttay) wrote on 2008-04-15:

I just noticed that the --ssh-command issue I mentionned at the end of the message above is already fixed by the latest version of backupninja upstream:

http://code.autistici.org/trac/backupninja/browser/trunk/ChangeLog

regards

Cyril

Rich Johnson (nixternal) on 2008-04-23

Changed in backupninja:
assignee:	gothicx → nobody

Revision history for this message

Mark Edgington (edgimar) wrote on 2008-06-03:

See http://bugs.debian.org/458816 . Backupninja 0.9.5-3 fixes the problem, and is in the Debian repository. The Ubuntu repository just needs to get updated with the latest Debian package.

Revision history for this message

Greg Grossmeier (greg.grossmeier) wrote on 2009-02-21:

Marking as fix released as Jaunty has 0.9.6.4 (which is newer than the version which is said to fix this issue).

If you still have problems, please feel free to open a new bug report.

Thanks!

Changed in backupninja:
status:	Incomplete → Fix Released

Bug Watch Updater (bug-watch-updater) on 2009-10-07

Changed in backupninja (Debian):
status:	Unknown → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

debbugs #458816
[done important fixed-upstream] Edit

Bug watches keep track of this bug in other bug trackers.

Ubuntubackupninja package

duplicity jobs not working

Bug Description

Other bug subscribers

Remote bug watches

Ubuntu
backupninja package