Ubuntu

incremental backup does not work

Reported by Martey Dodoo on 2007-02-28
42
This bug affects 4 people
Affects Status Importance Assigned to Milestone
duplicity (Ubuntu)
Medium
Lionel Porcheron
Feisty
Medium
Martin Pitt

Bug Description

Binary package hint: duplicity

Using Duplicity on Feisty through backupninja. "Incremental" backups do not work properly. This seems to be confirmed by other users, see <http://lists.gnu.org/archive/html/duplicity-talk/2007-02/msg00008.html>

Gregory Oschwald (osch0001) wrote :

I am also experiencing this. I am confirming the bug.

Changed in duplicity:
status: Unconfirmed → Confirmed
Rog (roger-mindsocket) wrote :

I'm experiencing this issue also.

Tom Helner (duffman) wrote :

I am experiencing this same incremental back up issue with duplicity on my Ubuntu Feisty servers. It appears that the files get backed up but --verify, restore, and the incremental backup does not “see” them. I do not have this problem on my Edgy servers (Version: 0.4.1-8ubuntu1).

Here is my output for a simple back up and --verify:

root@w2szswr601:~# duplicity --encrypt-key "91708B1B" /home/szswr6/vm_info file:///home/data/bur/backup
GnuPG passphrase:
Retype to confirm:
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1178639042.32 (Tue May 8 11:44:02 2007)
EndTime 1178639043.83 (Tue May 8 11:44:03 2007)
ElapsedTime 1.50 (1.50 seconds)
SourceFiles 4
SourceFileSize 3343424 (3.19 MB)
NewFiles 4
NewFileSize 3343424 (3.19 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 4
RawDeltaSize 3339328 (3.18 MB)
TotalDestinationSizeChange 2604284 (2.48 MB)
Errors 0
-------------------------------------------------

root@w2szswr601:~# duplicity --encrypt-key "91708B1B" --verbosity 4 --verify file:///home/data/bur/backup /home/szswr6/vm_info
GnuPG passphrase:
Retype to confirm:
Difference found: New file .
Difference found: New file VirtualBox_UserManual.pdf
Difference found: New file coe_build_into.txt~
Difference found: New file qemu_info_from_nick.txt
Verify complete: 4 files compared, 4 differences found.

Frank Ploss (frank-ploss) wrote :

The problem seems to have something to do with the python version, which is 2.5 on feisty, and 2.4 on edgy.

It works with python 2.4: if I copy the duplicity python modules into the python 2.4 tree and call duplicity with 'python2.4 /usr/bin/duplicity', everything works fine (despite a librsync version warning).

Using python2.4 to build and install duplicity from source also produces a working installation.

Jens (jens-launchpad-net) wrote :

Hi,
I am also experiencing this. I wanted to use duplicity for incremental backups of our clients, but the missing incremental backup function makes it just about impossible.

I would really appreciate if this could be investigated! :)

Jens

Jens (jens-launchpad-net) wrote :

Addendum: if you compile duplicity-0.4.2 from source using python2.4, detecing present backup sets fails, because of this (I think):

 Reading results of 'echo -e 'cd /users/benecke/KLAVIER
 ls -1' | sftp -b - benecke@orgel'

Full log:

jens@klavier:~$ duplicity --no-encryption --verbosity 9 --verify ssh://benecke@orgel//users/benecke/KLAVIER /
Reading globbing filelist /home/jens/.ftplicity/exclude
Main action: verify
Reading results of 'echo -e 'cd /users/benecke/KLAVIER
ls -1' | sftp -b - benecke@orgel'
Invalid command.
Ignoring file 'bin'
(... several other files ...)
Collection Status
-----------------
Connecting with backend: scpBackend
Archive dir: None
No backup chains with active signatures found

Found 0 backup chains without signatures.
No orphaned or incomplete backup sets found.
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 373, in ?
    if __name__ == "__main__": main()
  File "/usr/bin/duplicity", line 355, in main
    elif action == "verify": verify(col_stats)
  File "/usr/bin/duplicity", line 262, in verify
    collated = diffdir.collate2iters(restore_get_patched_rop_iter(col_stats),
  File "/usr/bin/duplicity", line 201, in restore_get_patched_rop_iter
    backup_chain = col_stats.get_backup_chain_at_time(time)
  File "/usr/lib/python2.4/site-packages/duplicity/collections.py", line 598, in get_backup_chain_at_time
    raise CollectionsError("No backup chains found")
duplicity.collections.CollectionsError: No backup chains found
Command exited with non-zero status 1

I am using SSH connections with the key stored in a key-agent (so passwordless connection is possible).

Duplicity 0.4.2 works fine on Debian Sarge, using python2.3.

Thanks,

Jens

Jyrki Muukkonen (jvtm) wrote :

There's something really evil going on with duplicity and python2.5. Unfortunately the duplicity codebase seems to be a bit messy, and it was easier to just fix the build scripts :)

Here's a fix debian/{rules,control} and friends which forces a build against python2.4. It also uses "$(PYTHON)" instead of "python" to call the build scripts, so it might be easier to step back and forth with different python versions.

Works for me.

Daniel Hahler (blueyed) on 2007-05-29
Changed in duplicity:
assignee: nobody → blueyed
Daniel Hahler (blueyed) wrote :

This is caused by a ValueError exception in duplicity's tarfile.py, because it uses int() on strings containing \0 (NUL) characters.
This exception does not happen with Python 2.4, and also is not specific to Ubuntu's Python 2.5.

The attached debdiff fixes this.

LaserJock (laserjock) on 2007-05-31
Changed in duplicity:
importance: Undecided → High
Daniel Hahler (blueyed) wrote :

The fix has been confirmed upstream.
It would be great if somebody could try my attached debdiff, which should probably work and fix the problem for you.
Please see https://wiki.ubuntu.com/UbuntuPackagingGuide/BuildFromDebdiff for creating a fixed package.

LaserJock (laserjock) on 2007-05-31
Changed in duplicity:
importance: High → Medium
Tom Helner (duffman) wrote :

I am unable to patch using duplicity_0.4.2-13ubuntu2.dsc.debdiff
Here is my output:

root@w2szswr601:/usr/src/duplicity# cd duplicity-0.4.2 && patch -p1 < ../duplicity_0.4.2-13ubuntu2.dsc.debdiff
can't find file to patch at input line 4
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff -Nru /tmp/QRGPt4qMTY/duplicity-0.4.2/debian/changelog /tmp/Lq0Slwsx2n/duplicity-0.4.2/debian/changelog
|--- /tmp/QRGPt4qMTY/duplicity-0.4.2/debian/changelog 2007-06-01 00:57:07.000000000 +0200
|+++ /tmp/Lq0Slwsx2n/duplicity-0.4.2/debian/changelog 2007-06-01 00:57:07.000000000 +0200
--------------------------
File to patch: debian/changelog
patching file debian/changelog
Hunk #1 FAILED at 1.
1 out of 1 hunk FAILED -- saving rejects to file debian/changelog.rej
patching file tmp/QRGPt4qMTY/duplicity-0.4.2/debian/patches/07fix-tarfile-_buftoinfo-for-python2.5.dpatch

Tom Helner (duffman) wrote :

OK I now see my mistake (other than trying to patch too late at night when the brain has already shut down!) but.....
I can patch using duplicity_0.4.2-10.1ubuntu1.dsc.debdiff but not duplicity_0.4.2-13ubuntu2.dsc.debdiff

root@w2szswr601:/usr/src/duplicity/duplicity-0.4.2# patch -p4 < ../duplicity_0.4.2-10.1ubuntu1.dsc.debdiff
patching file debian/patches/06_fix-tarfile-_buftoinfo-for-python2.5

root@w2szswr601:/usr/src/duplicity/duplicity-0.4.2# patch -p4 < ../duplicity_0.4.2-13ubuntu2.dsc.debdiff
patching file debian/changelog
Hunk #1 FAILED at 1.
1 out of 1 hunk FAILED -- saving rejects to file debian/changelog.rej
patching file debian/patches/07fix-tarfile-_buftoinfo-for-python2.5.dpatch

Daniel Hahler (blueyed) wrote :

Sorry Tom, the debdiff was against the gutsy package.

The one against feisty, with changelog entry is attached. You should apply this one with "patch -p1", as described in the wiki page linked above.

Daniel Hahler (blueyed) on 2007-06-01
Changed in duplicity:
assignee: blueyed → nobody
Tom Helner (duffman) wrote :

No problem dAniel.
Well...I have good news and bad news. It now patches ok, but I am getting the same result w/ the new package as I was w/ the old package.

Daniel Hahler (blueyed) wrote :

- Added patch to 00list (used dpatch-edit-patch)
- Taken patch from upstream CVS, should make merging in the future easier

Tom Helner (duffman) wrote :

This time I have only good news to report!
The new debdiff worked perfectly. My first (and quick) tests show a perfectly working Duplicity. I will do some more thorough testing later toady, when I have more time.

Tom Helner (duffman) wrote :

debdiff duplicity_0.4.2-10.1ubuntu1.dsc duplicity_0.4.2-10.1ubuntu1.1.dsc Looks to be the fix!

I have been using the patched duplicity pretty heavily over the last few days and everything appears to be working fine.

Thanks!

Mathias Gug (mathiaz) wrote :

Works for me too. Built and installed on feisty.

Emmet Hikory (persia) wrote :

My apologies, but I'm a little confused as to which debdiff needs to be applied to each of the gutsy and feisty packages. Could you please either re-prepare or reupload debdiffs for new candidate revisions for each of feisty-proposed and gutsy, with a comment indicating which is correct. Alternately, please delete any previously attached debdiffs that do not apply. I have unsubscribed ubuntu-universe-sponsors pending this clarification: please resubscribe to again request uploads. Thank you.

Changed in duplicity:
assignee: nobody → blueyed
status: Confirmed → Needs Info
Daniel Hahler (blueyed) wrote :

I've removed all my previous debdiffs, only left the final one.

There's only a debdiff for feisty-proposed: there will be a new release of duplicity with a lot more fixes in the next days, so no need to get this particular fix into gutsy alone IMHO.

Changed in duplicity:
assignee: blueyed → nobody
status: Needs Info → Confirmed
Mathias Gug (mathiaz) wrote :

The attached debdiff is the one working for me on feisty.

Emmet Hikory (persia) on 2007-06-11
Changed in duplicity:
importance: Undecided → Medium
status: Unconfirmed → Confirmed
Daniel T Chen (crimsun) wrote :

Format: 1.7
Date: Sun, 3 Jun 2007 16:54:31 +0200
Source: duplicity
Binary: duplicity
Architecture: source
Version: 0.4.2-10.1ubuntu1.1
Distribution: feisty-proposed
Urgency: low
Maintainer: Ubuntu MOTU Developers <email address hidden>
Changed-By: dAniel hAhler <email address hidden>
Description:
 duplicity - encrypted bandwidth-efficient backup
Launchpad-Bugs-Fixed: 88617
Changes:
 duplicity (0.4.2-10.1ubuntu1.1) feisty-proposed; urgency=low
 .
   * debian/patches/06_fix-tarfile-_buftoinfo-for-python2.5:
     Fixed "incremental backup does not work" for Python 2.5 (LP: #88617)
   * Applied changes according to https://wiki.ubuntu.com/DebianMaintainerField
Files:
 af337311f168266c6608a160caae5622 758 utils optional duplicity_0.4.2-10.1ubuntu1
.1.dsc
 5784a495edd517fb4fc07792b4c0ab1d 7464 utils optional duplicity_0.4.2-10.1ubuntu
1.1.diff.gz
Original-Maintainer: Martin Wuertele <email address hidden>

Changed in duplicity:
status: Confirmed → In Progress
Martin Pitt (pitti) wrote :

Accepted into feisty-proposed. Please go ahead with QA testing and also fix this in gutsy ASAP. I will refuse to let this into -updates without having it tested and fixed in gutsy.

Changed in duplicity:
status: In Progress → Fix Committed
Daniel Hahler (blueyed) wrote :

duplicity (0.4.2-10.1ubuntu1.1) feisty-proposed; urgency=low

  * debian/patches/06_fix-tarfile-_buftoinfo-for-python2.5:
    Fixed "incremental backup does not work" for Python 2.5 (LP: #88617)
  * Applied changes according to https://wiki.ubuntu.com/DebianMaintainerField

 -- dAniel hAhler <email address hidden> Sun, 3 Jun 2007 16:54:31 +0200

Changed in duplicity:
status: Fix Committed → Fix Released
Emmet Hikory (persia) on 2007-06-15
Changed in duplicity:
status: Fix Released → Fix Committed
Daniel Hahler (blueyed) wrote :

Please find attached the debdiff for Gutsy.

Format: 1.7
Date: Fri, 15 Jun 2007 15:29:08 +0200
Source: duplicity
Binary: duplicity
Architecture: source
Version: 0.4.2-14ubuntu2
Distribution: gutsy
Urgency: low
Maintainer: Ubuntu MOTU Developers <email address hidden>
Changed-By: dAniel hAhler <email address hidden>
Description:
 duplicity - encrypted bandwidth-efficient backup
Launchpad-Bugs-Fixed: 88617
Changes:
 duplicity (0.4.2-14ubuntu2) gutsy; urgency=low
 .
   * debian/patches/06_fix-tarfile-_buftoinfo-for-python2.5:
     Fixed "incremental backup does not work" for Python 2.5 (LP: #88617)
Files:
 a519da904c90d489459927cd7addf073 757 utils optional duplicity_0.4.2-14ubuntu2.dsc
 5c72964ab92f48257b591ef4306bc038 9137 utils optional duplicity_0.4.2-14ubuntu2.diff.gz
Original-Maintainer: Alexander Zangerl <email address hidden>

Changed in duplicity:
assignee: nobody → lionel.porcheron
status: Confirmed → Fix Committed
Daniel Hahler (blueyed) wrote :

duplicity (0.4.2-14ubuntu2) gutsy; urgency=low

  * debian/patches/06_fix-tarfile-_buftoinfo-for-python2.5:
    Fixed "incremental backup does not work" for Python 2.5 (LP: #88617)

 -- dAniel hAhler <email address hidden> Fri, 15 Jun 2007 15:29:08 +0200

Changed in duplicity:
status: Fix Committed → Fix Released
Daniel Hahler (blueyed) wrote :

To get the fix into feisty-updates, I need two people who acknowledge that it works.
So, please test it by adding
deb http://archive.ubuntu.com/ubuntu/ feisty-proposed universe
to your /etc/apt/sources.list, upgrade duplicity from there and give feedback here.

eyequeue (eyequeue-gmail) wrote :

Success!

"Verify complete: 3736 files compared, 16 differences found."

Earlier, with the original feisty version, (0.4.2-10-1ubuntu1,) that would have read something like "3736 files compared, 3736 differences found."

I can confirm that the 0.4.2-10.1ubuntu1.1 version does in fact fix the problem here. Thank you very much dAniel! This was a show-stopper bug on one of my systems.

Incremental backups now work.

Perhaps I should have included the following output in my last comment <a href="https://bugs.launchpad.net/ubuntu/feisty/+source/duplicity/+bug/88617/comments/33">a moment ago</a>:

"--------------[ Backup Statistics ]--------------
StartTime 1183342165.58 (Mon Jul 2 02:09:25 2007)
EndTime 1183342167.58 (Mon Jul 2 02:09:27 2007)
ElapsedTime 2.00 (2.00 seconds)
SourceFiles 3733
SourceFileSize 169660949 (162 MB)
NewFiles 9
NewFileSize 209738 (205 KB)
DeletedFiles 3
ChangedFiles 4
ChangedFileSize 26006 (25.4 KB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 16
RawDeltaSize 210716 (206 KB)
TotalDestinationSizeChange 124789 (122 KB)
Errors 0
-------------------------------------------------"

Thanks again, dAniel!

Tom Helner (duffman) wrote :

I have uninstalled the patched version that I have been running and installed duplicity from feisty-proposed. It is working perfectly for me.

Martin Pitt (pitti) wrote :

Copied -proposed version into -updates. Thank you!

Changed in duplicity:
assignee: nobody → pitti
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers