Could not restore the following files. Please make sure you are able to write to them

[leroy]

Hello.

1.distribution: ubuntu 16.04

2.deja-dup 34.2-0ubuntu1.1
duplicity 0.7.06-2ubuntu2

3.org.gnome.DejaDup last-restore '2017-08-13T21:47:37.916487Z'
org.gnome.DejaDup periodic true
org.gnome.DejaDup full-backup-period 90
org.gnome.DejaDup backend 'file'
org.gnome.DejaDup last-run '2017-08-13T21:47:37.916487Z'
org.gnome.DejaDup nag-check ''
org.gnome.DejaDup prompt-check '2017-08-13T15:59:20.448547Z'
org.gnome.DejaDup root-prompt true
org.gnome.DejaDup include-list ['$HOME']
org.gnome.DejaDup exclude-list ['$TRASH', '$DOWNLOAD']
org.gnome.DejaDup last-backup ''
org.gnome.DejaDup periodic-period 7
org.gnome.DejaDup delete-after 0
org.gnome.DejaDup.S3 id ''
org.gnome.DejaDup.S3 bucket ''
org.gnome.DejaDup.S3 folder 'too-N130BU'
org.gnome.DejaDup.OpenStack authurl ''
org.gnome.DejaDup.OpenStack tenant ''
org.gnome.DejaDup.OpenStack username ''
org.gnome.DejaDup.OpenStack container 'too-N130BU'
org.gnome.DejaDup.GCS id ''
org.gnome.DejaDup.GCS bucket ''
org.gnome.DejaDup.GCS folder 'too-N130BU'
org.gnome.DejaDup.GDrive email ''
org.gnome.DejaDup.GDrive folder '/deja-dup/too-N130BU'
org.gnome.DejaDup.Rackspace username ''
org.gnome.DejaDup.Rackspace container 'too-N130BU'
org.gnome.DejaDup.File path '/home/too/deja-dup'
org.gnome.DejaDup.File short-name 'SVECO'
org.gnome.DejaDup.File uuid '0F6655965B3C5375'
org.gnome.DejaDup.File icon '. GThemedIcon drive-harddisk-usb drive-harddisk drive'
org.gnome.DejaDup.File relpath b'sauvsystemEric'
org.gnome.DejaDup.File name 'Intenso External USB 3.0: SVECO'
org.gnome.DejaDup.File type 'volume'

4.file is empty

I have restored most of my files, from a Deja Dup backup.
Most of the backup restored properly, but I was given a list of 46 files, including some (thunderbird inboxes that prevent thunderbird from working properly), with the message : " with the following error message: "Could not restore the following files. Please make sure you are able to write to them."

I'm wondering:
1) Is there any way to get inside the duplicity archives and inspect for viable versions of these files?
2) What is meant by "Please make sure you are able to write to them."? What problem does this indicate?
3) Anything else I might try? Or are these files corrupted or lost? The log seems to remember them as belonging to the backup, they must have been there at some point.

Thanks!

(Yes, I know, I'll find a better backup solution for next round...
But can i, at least get all my previous emails so that i can keep my job ?)

[Michael Terry (mterry)]

The message might be because you chose "restore files to original locations" and for some reason the file permissions on your thunderbird folders are problematic?

Two thoughts:

1) Choose "Restore to specific folder" when restoring and restore all the files to some new directory in ~/Desktop or something -- something you know you can write to.

2) Read https://wiki.gnome.org/Apps/DejaDup/Help/Restore/WorstCase -- it might have good advice for restoring manually.

[leroy (leroyrene)]

Dear Michael,

Thank you for answering so fast.
It is my second night with no sleep facing the problem (it is 2:30 AM where i am)
I don't know why some directories are locked like that, randomly, without apparent reasons...
Idon't understand why the fact of restoring them in the original place creates a problem but restoring them elsewhere doesn't...

but, anyway, as it is a matter of life or death for my job and as, thanks to this, i didn't sleep during the week end, is there a fast solution to restore a directory, and only one, to a different location ?
Looks like it is one or the other...

Also, none of the solution you suggested on other topics (such like restoring in sudo mode or restoring elsewhere) could be used (first solution and Deja-dup says "back up media not connected" (although it is) and for the second solution, file is too big, i haven't got enough place to restore all my directories in one on my allready-restored-files'directory...

so if it would be possible to save one and only one file to a different place, and so cancel the "read-only" malediction, it would be great...

[Michael Terry (mterry)]

OK so you could use duplicity directly. But first, let's try using deja dup again. Deja dup doesn't have a mode that lets you restore just some files to a new location. But it does have a mode that lets you restore just some files to the original location.

So try this. Move your thunderbird directory out of he way (mv .thunderbird .thunderbird.bak -- or some such). Then use Files to restore missing files in your Home folder (or just call deja-dup --restore ~/.thunderbird). That should let you restore just a few files without dealing with whatever permission issue is there with your thunderbird dir.

[leroy (leroyrene)]

Thank you again for your help Michael.

I already tried that and had the same message (Could net restore the following files. Please make sure you are able to write to them) with the corresponding list of files that were in the original one.

For the moment i am saving some files to make room on the hard drive and try the seocond solution (save everything on a different location) as there's a lot of dir, it will take some hours. i think i will take the occasion to sleep a bit. ill keep you informed asap.

thanks again

[Michael Terry (mterry)]

Hmm, I'm sorry you're still having the problem. :( When this is done and you have your files, I'd be interested in digging into what's causing that.

If you can't make enough room.. I'd suggest digging into using duplicity directly.

It has a kind of crazy complicated command line setup though. Run deja-dup like so: "DEJA_DUP_DEBUG=1 deja-dup --restore ... etc" and it will print both the command lines it sends to duplicity and its output. But all you care about is the command to duplicity that looks like "duplicity restore ... A MILLION OPTIONS" Take that argument and edit as needed (make sure to take off --log-fd= option too, you don't want that).

Or just read duplicity man pages a bunch.

[leroy (leroyrene)]

ok, i'll do that, and i'll also give you all the info you want WHEN the problem is solved ;-)
For the moment, i let my copy by itself and i'm going to get some rest, i really need that...

[leroy (leroyrene)]

If it may help, list of the files that were not retired after " deja-dup --restore ~/.thunderbird ":

/q21d1m3p.erijou/Mail/Local Folders/Drafts
/q21d1m3p.erijou/Mail/Local Folders/Inbox
/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC
/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.msf
/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.sbd/BOULOT.msf
/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.sbd/CONSO.msf
/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.sbd/CONSO.sbd/Assurance.sbd/GMF
/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.sbd/CONSO.sbd/Assurance.sbd/GMF.msf
/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.sbd/CONSO.sbd/Telephone.sbd/Freemobile
/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.sbd/CONSO.sbd/Telephone.sbd/Freemobile.msf
/q21d1m3p.erijou/Mail/Local Folders/Junk
/q21d1m3p.erijou/Mail/Local Folders/Sent
/q21d1m3p.erijou/Mail/Local Folders/Sent.msf
/q21d1m3p.erijou/history.mab
/q21d1m3p.erijou/junklog.html
/q21d1m3p.erijou/prefs.js
/q21d1m3p.erijou/sessionCheckpoints.json

[leroy (leroyrene)]

I wrote "retired" i meant "restored".
Well, restoring all on a new place is in progress, don't have much hope in it: i already checked the .thunderbird dir size and it's the same as before...

So, to sum up:
1.restoration work ok except for few directories, but it concerns dir that are essential to thunderbird.
2. attempt to restore the directory alone failed
3. attempt to move the directory and restore it alone failed
4. attempt to make it via terminal failed
5. attempt to restore all the system in a different place is in progress
Looks like deja-dup tells me "it's ok: your files have been perfectly saved, don't worry, the only point is that you will never be able to restore or access them as we have locked them randomly with "for-no-reason-read-only-stuff", unless you learn to become an ingeneer in computer system. But don't worry, we'll keep them safe. You're welcome"

i've never been so desperate, frustrated and angry because of my computer.
I'm preparing to accept that i lost all my mails, and probably my job , as it was both my personnl and pro computer...
Thanks anyway, but i read you are involved in Deja-dup developpment, and i think there is a big issue here...

Also, i saw that i wasn't alone to meet this problem.
I took the post off another poor user and adapted to my case to make my first post here:
https://askubuntu.com/questions/886169/deja-dup-cant-restore-files?rq=1
https://askubuntu.com/questions/710340/trouble-with-deja-dup-restore

I didn't try the solution in the first answer of the second topic as it looks complicated, but that will be my next try.

[leroy (leroyrene)]

Ok, restore to a different location ended up the same.
Unrestored directories list:

/home/too/.config/libreoffice/4/user/uno_packages/cache/log.txt
/home/too/.local/share/gvfs-metadata/home
/home/too/.local/share/recently-used.xbel
/home/too/.local/share/zeitgeist/activity.sqlite
/home/too/.macromedia/Flash_Player/macromedia.com/support/flashplayer/sys/settings.sol
/home/too/.mozilla/firefox/4kwx3dm7.default-1446238324638/xulstore.json
/home/too/.thunderbird/q21d1m3p.erijou/Mail/Local Folders/Drafts
/home/too/.thunderbird/q21d1m3p.erijou/Mail/Local Folders/Inbox
/home/too/.thunderbird/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC
/home/too/.thunderbird/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.msf
/home/too/.thunderbird/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.sbd/BOULOT.msf
/home/too/.thunderbird/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.sbd/CONSO.msf
/home/too/.thunderbird/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.sbd/CONSO.sbd/Assurance.sbd/GMF
/home/too/.thunderbird/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.sbd/CONSO.sbd/Assurance.sbd/GMF.msf
/home/too/.thunderbird/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.sbd/CONSO.sbd/Telephone.sbd/Freemobile
/home/too/.thunderbird/q21d1m3p.erijou/Mail/Local Folders/Inbox.sbd/ERIC.sbd/CONSO.sbd/Telephone.sbd/Freemobile.msf
/home/too/.thunderbird/q21d1m3p.erijou/Mail/Local Folders/Junk
/home/too/.thunderbird/q21d1m3p.erijou/Mail/Local Folders/Sent
/home/too/.thunderbird/q21d1m3p.erijou/Mail/Local Folders/Sent.msf
/home/too/.thunderbird/q21d1m3p.erijou/history.mab
/home/too/.thunderbird/q21d1m3p.erijou/junklog.html
/home/too/.thunderbird/q21d1m3p.erijou/prefs.js
/home/too/.thunderbird/q21d1m3p.erijou/sessionCheckpoints.json
/home/too/Bureau/Velos/DSCN1604.JPG
/home/too/Bureau/Velos/DSCN1605.JPG
/home/too/Bureau/Velos/DSCN1606.JPG
/home/too/Bureau/Velos/DSCN1607.JPG

[Michael Terry (mterry)]

Ugh. I'm so sorry this is happening to you. For whatever reason, deja-dup isn't working. I think your best bet is digging into it manually with duplicity or, if even that still has the problem, manually unpacking the tarballs. Instructions on that are here:

https://wiki.gnome.org/Apps/DejaDup/Help/Restore/WorstCase

[leroy (leroyrene)]

Ok, thanks again, i'll try it as soon as i can.
But it is very strange that i found other examples on the web in which directories involved are always different but ALWAYS include vital thunderbird files.
It's also strange that among maybe 15000 + directories, it's only 25 of them that have the problem and among those 25, 17 are about Thunderbird...

Isn't the problem about an incompatibility between Thunderbird and DD ? Or TB and DD file format / authorization / file name, ... Or anything like that ?

Someone with your knowledge can surely find it and solve it, because, if people using DD professionally lose all their messages every 3 or 4 save, it cannot be trusted in...

[Michael Terry (mterry)]

Oh for sure, someone (thunderbird, duplicity, deja-dup) is screwing up. With the limited information I have so far, I'm going to guess duplicity (deja-dup doesn't deal too directly with issues of file permissions, but maybe we are trying to be too clever for our own good in some way or maybe we should have reported some error back when the original backup was being made that we didn't ).

But none of that helps you right now. I can set up a thunderbird account and back it up and restore a few times to see if I can trivially trigger this bug. But issues like this are hard to debug. Especially since we rarely get to have access to the files causing the problem.

[leroy (leroyrene)]

there are some information in this directory that are private and/or "confidential" professionally speaking (i don't work in the military or in the nuclear industry ;-) )
So it would bother me to send it to someone, but depending on where you are, i can come and give you access to the back up, as long as i can see what is done with it.
I work in Paris France, are you in the US or UK ? (seeing you answering at times that are 3am for me makes me think it's the first option...)

[leroy (leroyrene)]

Ok, now i've tried the "restore-missing-file" method at the lowest level of directory (/q21d1m3p.erijou/Mail/Local Folders/Inbox for example).

The message has changed, it is no longer "Could not restore the following files. Please make sure you are able to write to them", it is "Could not restore the following files. file not present in the back up"... (i translate from french which is my computer default language)
How can it be listed and be absent ?

It's worst, but, in some way, it is clearer.

Before i try anything else, i will try a complete restoration next week end with my other back up.
Chances of success ?

[Vej (vej)]

Hello Leroy!

You asked:

> How can it be listed and be absent ?

We have a list of files in the backup, which tells the program which file is stored in which archive (volume in backup programmer speech). Additionally you have a lot of these volumes, which contain the files.

I would speculate, that the files are listed in the list, but could not be found in the volume.

If these files are somehow important to you, you could try to "play duplicity" and do a restore by hand, following the instructions given in the Section "Restoring by hand" from our online help: https://wiki.gnome.org/Apps/DejaDup/Help/Restore/WorstCase

I know about one person from the German community which found more files doing so than were restored by duplicity and Déjà Dup, but chances are not very high.

Best Regards

Vej

[leroy (leroyrene)]

Thanks a lot Vej !
You are right, but i'm a bit unconfortable with this solution which i'm not at all familiar with and which will probably take time and nerves ;-)
Well, if nothing else works, i'll try the "duplicity" thing.

But i have another card in my pocket before that: Next monday i will be back from vacation and have access to my second back up, that i also do on a weekly basis.
I will try a total restore (once again) but from this second source.
I will see if it works as usual or if the probleme stays the same.

Probably, knowing if the problem does or doesn't happen then should be interesting to understand, and maybe solve it, in the future.
=> If it doesn't happen, we can assume it comes from an incident during back up, or from the media itself.
=> If it does, then the problem is systematic, and due to an error in my config, or between DD and TB...

I'll let you know the results as soon as i can try.

And if it doesn't , i will probably have tons of questions about duplicity commands ;-)

[leroy (leroyrene)]

Ok, After the "restore one file only" method, everything restored in Thunderbird from second back up !!!

Consequences of the problem are solved, but i have no satisfying explanation found.
And nothing garantees it won't happen again...

Is there something i can do to help more competent people to solve the problem ?

Thanks anyway Michael and Vej !