restore fails with "librsync error 103 while in patch cycle"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Duplicity |
Confirmed
|
Undecided
|
Unassigned | ||
duplicity (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
I'm doing a trial restore from an sftp backup, and it fails like so:
Writing .bazaar/
Deleting /tmp/duplicity-
Processed volume 2 of 2531
Running 'sftp -oServerAliveIn
sftp command: 'get "/backup/
Writing .bazaar/
......
Writing .byobu/status of type reg
Writing .byobu/windows of type reg
python: ERROR: (rs_file_copy_cb) unexpected eof on fd23
python: ERROR: (rs_job_complete) patch job failed: unexpected end of input
Traceback (most recent call last):
File "/home/
with_
File "/home/
fn()
File "/home/
restore(
File "/home/
restore_
File "/home/
for ropath in rop_iter:
File "/home/
final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
File "/home/
misc.
File "/home/
buf = infp.read(
File "/home/
self.
File "/home/
raise librsyncError(
librsyncError: librsync error 103 while in patch cycle
~/work/
Using the tip of the 0.6 branch on Ubuntu Maverick.
I think I've seen this once before too.
The 'unexpected eof' error suggests that one of the increment files could be truncated, but that doesn't seem likely: I haven't had any filesystem problems, and at the gzip level all of them seem to be ok.
esskov (esskov) wrote : | #1 |
Christopher Foo (chris.foo) wrote : | #2 |
I have this problem during verify as well. I can work around this by specifying subdirectories, for example,
ubuntu@
It might be possible to list out the files (list-current-
Christopher Foo (chris.foo) wrote : | #3 |
- Emergency hax patch that skips the unpatchable file Edit (761 bytes, text/plain)
Well, I made and attached an emergency hax patch that skips the file—at least I think that's what it does—for those who want it. :)
Christopher Foo (chris.foo) wrote : | #4 |
- Emergency hax patch that skips the unpatchable file (version 2) Edit (1.2 KiB, text/plain)
Attached is an updated hax patch that works for restore as well.
Robin Clark (robindtclark) wrote : | #5 |
How do you use the patch please?
Christopher Foo (chris.foo) wrote : | #6 |
- Emergency hax patch that skips the unpatchable file (version 3) Edit (1.1 KiB, text/plain)
In regards to https:/
patch -p0 < ../duplicity_
where you are in the source code directory and the patch is one level outside.
I'm attaching an updated hax patch for the latest version from the source code repo if the older patches don't work. It has not been tested as I don't have access to the faulty backup files any more.
Robin Clark (robindtclark) wrote : | #7 |
You sir are a legend, thank you kindly. I am now restored.
Lunarts (lunartsbr-gmail) wrote : | #8 |
Very smart, you saved my skin, it seems the issue was an uncaught python error, the double try except patch also did the trick for me, avoiding the software to colapse due to that issue, I was already accepting my doomed fate of losing all my software projects.
If it is an usefull information, I'm a Xubuntu 12.10 user; another person with the problem also used Xubuntu(12.04); I never had these backup corruption issues with plain ubuntu, may this be a Xubuntu specific issue? Or maybe it may be an issue elsewere during the backup process corrupted the backup, altought that only happened on Xubuntu as said above.
I wonder if Deja Dup is dangerous to use as backup tool on Xubuntu, version 24.0 and the current unstable presents backup issues sometimes, and what is a very bad, it cannot witstand a single backup file patch error without crumbling(without patch), I think the patch above should be applied to Deja dup permanenlty, together with an usefull gui log at the end of the backup restore showing which files couldn't be patched.
Christopher Foo (chris.foo) wrote : | #9 |
re: #8
That's a strange coincidence. I installed Xubuntu onto my regular Ubuntu install and I used Duplicity for backups.
As I mentioned earlier, I don't have physical access to my broken backups for testing. (I might have actually deleted them; can't remember.) It might be helpful if someone could post logs with and without my patches. I don't have time to actively debug this problem as I'm still in university, but if the devs arn't looking at this and more people have this issue, I'll dedicate some time to this bug.
Patch or no patch, I learned that it's best to use two backup systems at the expense of buying more external hard drives: one native and one cross-platform. (Currently I use Back In Time and Areca.)
Lunarts (lunartsbr-gmail) wrote : | #10 |
I installed pure Xubuntu from the beggining(no Ubuntu first), amd64 on an imac 2011(no backup issues with ubuntu on it thought). I still have my bugged backups set saved. I wonder to where the patch additional except: error log messages are being sent to(currently still running backup restore), as I wanted to see them; maybe I must use duplicity on terminal to see them. Not sure if as the error log line is it would output to the log the faulty files(used to program python games only), knowing the faulty files I can by logic have a notion of what may had corrupted the backups at least on my case, which could be:
1 - One of Xubuntu's constant energy manager errors which sometimes freezes the system for a couple of seconds on imac 2011
2 - During my own software testing, as sometimes I find bugs which freezes the system for around 30 min or at least severly drain system memory for one minute.
3 - Maybe having the file open, and using it while it is being backed up may be a problem
4 - Issues between Deja Dup and Xubuntu(at least on my hardware)
4 - Something else
I also discovered, like yourself, I will need an additional external drive, upon which I will not backup with any software, but manually instead weekly. I will check Back in Time and Areca as I'm interested in alternatives to Deja Dup.
Lunarts (lunartsbr-gmail) wrote : | #11 |
Allright, I saw some of the files which had to be ignored by the above patch; until now, all of them were open when Deja Dup was doing its backup process(a blender file and a python one), very probably their buggy backup was made after the november 15 full deja dup backup I did, which means only their changes compared to the source file were stored(not the full file). I could recover using the last option at Deja Dup 'Worst Case' web page the missing files, thought only the ones from november 15 full backup until now.
Ubuntu Foundations Team Bug Bot (crichton) wrote : | #12 |
The attachment "Emergency hax patch that skips the unpatchable file" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.
[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]
tags: | added: patch |
tlaurion (thierry-laurion) wrote : | #13 |
full file can be found here if patch gets rejected:
http://
Launchpad Janitor (janitor) wrote : | #14 |
Status changed to 'Confirmed' because the bug affects multiple users.
Changed in duplicity (Ubuntu): | |
status: | New → Confirmed |
Thom Wiggers (twiggers) wrote : | #15 |
I'm having the same issue on gentoo
Changed in duplicity: | |
status: | New → Confirmed |
Michael Terry (mterry) wrote : | #16 |
Here's a hint at what might be going wrong. While working on the Python 3 port of duplicity, I started getting this error. It turned out to be because I accidentally closed the patch basis file too soon.
Maybe some code path here is closing a file we need for rsync patching too early.
techno-mole (techno-mole) wrote : | #17 |
I have run into the same problem trying to restore my backups.
The error I get is -
Traceback (most recent call last):
File "/usr/bin/
with_
File "/usr/bin/
fn()
File "/usr/bin/
restore(
File "/usr/bin/
restore_
File "/usr/lib/
for ropath in rop_iter:
File "/usr/lib/
final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
File "/usr/lib/
misc.
File "/usr/lib/
buf = infp.read(
File "/usr/lib/
self.
File "/usr/lib/
raise librsyncError(
librsyncError: librsync error 103 while in patch cycle
I have tried using a more up to date version, tried using an earlier version, but I get the same error every time.
Is there a way I can get the backups back ? the entire backup folder is about 165gb and most of it I'd rather no lose if possible.
I noticed there's a patch mentioned, but in all honesty I'm not sure how to apply it, so I'm willing to give that a go, if someone has an idiot proof guide on how to apply the patch.
And if there's anything else I can try I'd be happy to have a go, really don't want to lose the data.
I'm running linux mint 14 (have previously run mint and used deja dup with no issues) current deja dup version - 26.0~bzr1443.
Constantin Gindele (c-gindele) wrote : | #18 |
I second the request on a quick how-to apply the patch for the less knowledgeable people (like my self) encountering the bug.
Christopher Foo (chris.foo) wrote : | #19 |
Sorry, I didn't see comment #17. This comment addresses #17 and #18 and any future readers.
First note, a disclaimer: I am *not* a developer nor maintainer of duplicity.
Ok, first open up a terminal. It should be in the Accessories menu.
Now type the following and hit enter. Do so for any commands listed from now on.
sudo apt-get install build-essential bzr
Enter your password if prompted and hit yes to install the software. Once it's installed, let's switch the current directory to somewhere obvious like the Desktop folder. Use command:
cd Desktop
Now download the software. Use command:
bzr branch lp:duplicity
There should be a duplicty folder on your Desktop now.
Now grab the my hax file patch and put it on the Desktop.
Ok, now switch to the duplicity folder. Run command:
cd duplicity
Before we apply to patch, we need to switch to older code. Run command:
bzr revert -r884
Now run the command to patch. The double dot indicates it's saved one level outside the current folder, the Desktop folder. Run command:
patch -p0 < ../duplicity_
It should say "patching file duplicity/
Now switch to the folder duplicity/ (aka ~/Desktop/
cd duplicity
Compile something. Run command:
python compilec.py
Now switch back to the parent dir. Run command:
cd ..
Now duplicity should be ready to run. You can run ./bin/duplicity.
This command will show the version. Run command:
./bin/duplicity -V
It will return "$version" which isn't set, but the main thing is that it successfully runs.
Hope this helps.
Constantin Gindele (c-gindele) wrote : | #20 |
Thank you for the detailed instructions! I managed to build however I had to install python-dev and librsync-dev as well two get around the errors
Python.h: No such file or directory
and
librsync.h: No such file or directory
The commands are
sudo apt-get install python-dev
and
sudo apt-get install librsync-dev
However I now get this error
Traceback (most recent call last):
File "./bin/duplicity", line 1403, in <module>
with_
File "./bin/duplicity", line 1396, in with_tempdir
fn()
File "./bin/duplicity", line 1276, in main
globals.
File "/home/
self.
File "/home/
map(
File "/home/
if set.add_
File "/home/
self.
File "/home/
remote_
AssertionError: ('duplicity-
So the quest to restore continues...
Adriano Oliveira (asoliveiramd) wrote : | #21 |
In which directory should i run "patch -p0 < ../duplicity_
Stuart Botha (zamorph) wrote : | #22 |
I also experience this bug. The backups I'm trying to restore are from my old Ubuntu 13.04 system - but actually one on which I installed the xubuntu-desktop package on top of - and trying to restore to a new computer with "plain" Ubuntu 13.04.
I saw a request for logs from before applying the patch, and afterwards. Below is my "before" log, whereafter I'll apply the patch mentioned above.
Log:
Traceback (most recent call last):
File "/usr/bin/
with_
File "/usr/bin/
fn()
File "/usr/bin/
restore(
File "/usr/bin/
restore_
File "/usr/lib/
for ropath in rop_iter:
File "/usr/lib/
final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
File "/usr/lib/
misc.
File "/usr/lib/
buf = infp.read(
File "/usr/lib/
self.
File "/usr/lib/
raise librsyncError(
librsyncError: librsync error 103 while in patch cycle
Stuart Botha (zamorph) wrote : | #23 |
Just to give an update:
I used the patch Chris indicated / attached to manually patch patchdir.py. I wasn't able to compile Duplicity, but when I tried restoring my backup again it ran past the error - but I'm experiencing a different error now (below)!
In terms of Adriano's question asking in which directory to run the patch: From the errors above, I found the patchdir.py file in /usr/lib/
The error I now get is a different error, I think unrelated to this specific bug report, but here is the new error anyway:
Traceback (most recent call last):
File "/usr/bin/
with_
File "/usr/bin/
fn()
File "/usr/bin/
restore(
File "/usr/bin/
restore_
File "/usr/lib/
for ropath in rop_iter:
File "/usr/lib/
final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
File "/usr/lib/
assert first.difftype != "diff", patch_seq
AssertionError: [(('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg)]
Michael Terry (mterry) wrote : | #24 |
If anyone has a copy of a backup with this bug that they'd be willing to share, please let me know. We haven't been able to reproduce this error yet. :(
sebastian-s (sebastian-s) wrote : | #25 |
Michael, all,
I'm trying to restore one of my backups that DejaDup saved to my NAS. Until now I have been unsuccessfull in doing so.
My machine is Ubuntu 12.10 and I haven't tried the patching thing yet.
here is the CLI output.... but I'm happy to dig further upon advice.
$ duplicity --file-to-restore home/seb ftp://backup@
NcFTP version is 3.2.5
Synchronising remote metadata to local cache...
GnuPG passphrase:
Copying duplicity-
Copying duplicity-
Copying duplicity-
Copying duplicity-
Copying duplicity-
Copying duplicity-
Copying duplicity-
Copying duplicity-
Copying duplicity-
Copying duplicity-
Copying duplicity-
Copying duplicity-
Copying duplicity-
Copying duplicity-
Copying duplicity-
Copying duplicity-
Warning, found incomplete backup sets, probably left from aborted session
Last full backup date: Thu Dec 13 16:47:32 2012
python: ERROR: (rs_file_copy_cb) unexpected eof on fd5
python: ERROR: (rs_job_complete) patch job failed: unexpected end of input
Traceback (most recent call last):
File "/usr/bin/
with_
File "/usr/bin/
fn()
File "/usr/bin/
restore(
File "/usr/bin/
restore_
File "/usr/lib/
for ropath in rop_iter:
File "/usr/lib/
final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
File "/usr/lib/
misc.
File "/usr/lib/
buf = infp.read(
File "/usr/lib/
self.
File "/usr/lib/py...
Constantin Gindele (c-gindele) wrote : | #26 |
I managed to compile but it still doesn't work. This really unfortunate since this is my only backup of all documents (well partially that's in the cloud), photos and music. I updated it before formatting and now I'm pretty much fucked.
Below's the output.
tantin@
Local and Remote metadata are synchronized, no sync needed.
Traceback (most recent call last):
File "./bin/duplicity", line 1403, in <module>
with_
File "./bin/duplicity", line 1396, in with_tempdir
fn()
File "./bin/duplicity", line 1276, in main
globals.
File "/home/
self.
File "/home/
map(
File "/home/
if set.add_
File "/home/
(self.
AssertionError: ({1: 'duplicity-
Constantin Gindele (c-gindele) wrote : | #27 |
Is there a completely manual way to restore? Removing the encryption shouldn't be to hard but it's an incremental backup and I worry about stitching the files together.
Dan Gillmor (dan-gillmor) wrote : Re: [Bug 662442] Re: restore fails with "librsync error 103 while in patch cycle" | #28 |
I tried and never could make it work. Lost a lot of work.
On 06/01/2013 09:41 AM, Constantin Gindele wrote:
> Is there a completely manual way to restore? Removing the encryption
> shouldn't be to hard but it's an incremental backup and I worry about
> stitching the files together.
>
Michael Terry (mterry) wrote : | #29 |
Constantin, to fix the error you see in your comment #26, delete the file duplicity-
Dennis Geus (dennis-famgeus) wrote : | #30 |
I have the same problem.
I really need to restore some files from my backup and it is ending with this error
librsyncError: librsync error 103 while in patch cycle
There is no file in my backup without .gpg
manuall recovery as suggested is causing the same error message.
Frederic Dreier (frederic-dreier) wrote : | #31 |
I tried the given patch but it still crash.
Error 'librsync error 103 while in patch cycle' processing home/john/
Traceback (most recent call last):
File "./bin/duplicity", line 1403, in <module>
with_
File "./bin/duplicity", line 1396, in with_tempdir
fn()
File "./bin/duplicity", line 1330, in main
restore(
File "./bin/duplicity", line 624, in restore
restore_
File "/tmp/duplicity
for ropath in rop_iter:
File "/tmp/duplicity
final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
File "/tmp/duplicity
assert first.difftype != "diff", patch_seq
AssertionError: [(('home', 'john', '.eclipse', 'org.eclipse.
Frederic Dreier (frederic-dreier) wrote : | #32 |
I added a try/catch around the faulty lines (like the one in the patch) and get it going through the restore. just missing sevral files (logs, gnome config DB, etc.).
I just got a bad feeling about it since my backup is my 'last ressort' when something goes wrong (here a HD crash). I expect it to be bullet-proof.
I hope duplicity will fix it since it is still a sooo great software for backup.
Regards
Dan Gillmor (dan-gillmor) wrote : | #33 |
Had this happen to me a year ago -- my mistake was failing to test the
restore immediately after the initial backup. I'm using grsync at this
point, a not-great but at least adequate substitute.
On 07/24/2013 12:53 AM, Frederic Dreier wrote:
> I added a try/catch around the faulty lines (like the one in the patch)
> and get it going through the restore. just missing sevral files (logs,
> gnome config DB, etc.).
>
> I just got a bad feeling about it since my backup is my 'last ressort'
> when something goes wrong (here a HD crash). I expect it to be bullet-
> proof.
>
> I hope duplicity will fix it since it is still a sooo great software for
> backup.
>
> Regards
>
gluca (gianluca-carlesso) wrote : | #34 |
I have the same problem. The restoration of the home gives the error "librsync error 103 while in patch cycle".
After applying the pacth, I can restore individual home folder, but not the entire home, it shows the error AssertionError: first.difftype! = "Diff"
Ulrike Hofbauer (ulderifata) wrote : | #35 |
I have the same problem. I used the patch Christopher Foo provided (many thanks for that!) - but unfortunately the outcome is the same as what gluca reports.
Would be very grateful for help
cuc (cuc+) wrote : | #36 |
thx for the patch.. rescued my stuff successfully...
now going for a more robust backup tool :)
David Huss (m-david-6) wrote : | #37 |
For those who still want (or have to) to restore their backups by hand, I found a solution for me.
The unpack-part is easy, the stich/join-part is not. So I wrote an application for that. More details here: http://
Martin Pool (mbp) wrote : | #38 |
The link to the source http://
to be broken
On 6 September 2013 02:18, David Huss <email address hidden> wrote:
> For those who still want (or have to) to restore their backups by hand, I
> found a solution for me.
> The unpack-part is easy, the stich/join-part is not. So I wrote an
> application for that. More details here:
> http://
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https:/
>
> Title:
> restore fails with "librsync error 103 while in patch cycle"
>
> To manage notifications about this bug go to:
> https:/
>
--
Martin
Michael Terry (mterry) wrote : | #39 |
A coworker (Steve Magoun) graciously lent me a backup data set of his that reproduces this problem.
What appears to have happened on his data set (and presumably others who are experiencing this) is that a file (say, test.txt) gets backed up in the initial full backup at time A. This initial backup seems fine.
Later, the file is included in an incremental backup at time B. But for some reason, the patch that duplicity creates is seemingly against a different basis file than the recorded at time A. So when restoring, the patch does not apply to the stored time A version of the file.
When restoring from time B, duplicity will complain but not error out. It will create a half-finished test.txt. If you restore from a later time point, duplicity will error out and create no file.
Here is the librsync output from my example data. (original basis file is an xml text file 68664 bytes large) You can see at the end, it tries to read from the basis file past its end. So the patch seems to be against a different (larger) basis file than we have. I'm not sure how this could happen yet, but I figured I'd dump these thoughts here.
---- original call to rs_job_iter here ----
python2.7: (rs_scoop_
python2.7: (rs_scoop_advance) advance over 4 bytes from input buffer
python2.7: (rs_patch_s_header) got patch magic 0x72730236
python2.7: (rs_scoop_
python2.7: (rs_scoop_advance) advance over 1 bytes from input buffer
python2.7: (rs_patch_
python2.7: (rs_scoop_
python2.7: (rs_scoop_
python2.7: (rs_scoop_advance) advance over 2 bytes from input buffer
python2.7: (rs_patch_s_run) running command 0x42, kind 1001
python2.7: (rs_patch_
python2.7: (rs_tube_
python2.7: (rs_scoop_
python2.7: (rs_scoop_advance) advance over 1 bytes from input buffer
python2.7: (rs_patch_
python2.7: (rs_scoop_
python2.7: (rs_scoop_
python2.7: (rs_scoop_advance) advance over 2 bytes from input buffer
python2.7: (rs_scoop_
python2.7: (rs_scoop_advance) advance over 2 bytes from input buffer
python2.7: (rs_patch_s_run) running command 0x4a, kind 1003
python2.7: (rs_patch_s_copy) COPY(where=512, len=24064)
python2.7: (rs_patch_
python2.7: (rs_patch_
python2.7: (rs_patch_
python2.7: (rs_scoop_
python2.7: (rs_scoop_advance) advance over 1 bytes from input buffer
python2.7: (rs_patch_
python2.7: (rs_scoop_
python2.7: (rs_scoop_
python2.7: (rs_scoop_advance) advance over 2 by...
Michael Terry (mterry) wrote : | #40 |
I still haven't found the source of this bug. But I have filed a branch [1] that is basically a glorified version of the patch in comment 6. This lets the rest of the restore continue when this error happens, though obviously without the problem file. (It will log an error about that file, so users will notice.)
Hopefully that will reduce the severity of this bug, but we should keep this bug open for the original problem.
[1] https:/
Michael Terry (mterry) wrote : | #41 |
Good news! I've confirmed this is the same underlying issue as bug 1252484 (which I will mark this bug a duplicate of). This is just one of its symptoms. I've attached the file 'test5.sh' which when run like 'sh test5.sh' should reliably reproduce this exception.
This bug has been fixed in duplicity trunk (and Ubuntu 13.10+). The fix will be in the upcoming duplicity 0.6.23 release.
I've prepared fixes for older Ubuntu releases, but the updates need testers before they can be released to everyone. If you are an Ubuntu user of an older release, please go to bug 1252484 and follow the instructions for testing the update. Thanks!
ddt (d-dt) wrote : | #42 |
Hi,
I am not sure if this is related, but I am certainly hoping someone can assist
I'm struggling to restore backups from an older installation of Ubuntu to a new installation on Ubuntu 16.04. I get the following error when trying to restore:
Failed with an unknown error Followed by:
Traceback (most recent call last):
File "/usr/bin/
with_
File "/usr/bin/
fn()
File "/usr/bin/
do_
File "/usr/bin/
list_
File "/usr/bin/
for path in path_iter:
File "/usr/lib/
refresh_
File "/usr/lib/
new_triple = get_triple(
File "/usr/lib/
path = path_iter_
File "/usr/lib/
for tarinfo in tf:
File "/usr/lib/
tarinfo = self.tarfile.next()
File "/usr/lib/
raise ReadError(
ReadError: unexpected end of data
The backup completed successfully but I cannot get it to restore.
Thanks in advance!
I'm also getting this while doing a "duplicity restore --no-encryption". The specific error I see is "python: ERROR: (rs_file_copy_cb) unexpected eof on fd8". This is duplicity 0.6.13.
In my case, one of the incremental backups might have been truncated/ interrupted as there were occasional read-errors on the source system in question. However, as this disk has now completely died and this backup is the only one, I'd like to hear if there is any way at all to make duplicity recover from this "unexpected end of input" and continue with the restore operation?