Authentication error with google drive

[Sam]

Hi,

I'm having an issue using deja-dup to back up my laptop to my google drive.

When backing only a small folder (eg: ~/Documents) every thing works fine.
However when trying to backup my whole home folder (/home/myusername) I get an "Invalid credentials" error after 5-10 min.

The back up seems to start normally and my destination folder starts filling with "duplicity-full...difftar.gz" files, but eventually I get the Invalid Credentials error.

lsb_release -d
Fedora release 27 (Twenty Seven)

rpm -q deja-dup duplicity
deja-dup-37.1-1.fc27.x86_64
duplicity-0.7.16-1.fc27.x86_64

deja-dup.log attached.
I cannot attach deja-dup.gsettings since i can only join 1 file. Will send if needed.

[Sam (scaum)]

I tried today using duplicity (followed this guide https://6ftdan.com/danielpclark/2016/04/21/encrypted-linux-backup-with-google-drive-and-duplicity/) and everything went fine.

For some reason the issue seems linked to deja-dup

[Michael Terry (mterry)]

Ah that blog post is using a different duplicity backend than deja-dup is using. Can you try something for me?

Instead of running "duplicity .... gdocs://<email address hidden>/backup" can you do "duplicity .... gio+google-drive://<email address hidden>/backup"? You'll need to do that inside your desktop session (i.e. not from cron).

Let me know if that hits the same bug you're seeing with deja-dup.

[Sam (scaum)]

Hi,

Thank you for your answer. Using your command gives me another error (not same as with deja-dup).

export GOOGLE_DRIVE_SETTINGS=~/.duplicity/credentials
duplicity --exclude-filelist ~/.duplicity/ignore ~/ gio+google-drive://myemail/test
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1559, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1545, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1381, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib64/python2.7/site-packages/duplicity/commandline.py", line 1140, in ProcessCommandLine
    backup, local_pathname = set_backend(args[0], args[1])
  File "/usr/lib64/python2.7/site-packages/duplicity/commandline.py", line 1015, in set_backend
    globals.backend = backend.get_backend(bend)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 223, in get_backend
    obj = get_backend_object(url_string)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 209, in get_backend_object
    return factory(pu)
  File "/usr/lib64/python2.7/site-packages/duplicity/backends/giobackend.py", line 97, in __init__
    self.remote_file.make_directory_with_parents(None)
 Error: g-io-error-quark: Operation unsupported (15)

The "test" folder was already created in my drive account at "root" level (next to the backup folder I am using with the other command).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jacob Marciniec (foxyjacob)]

Bumping this... I will gladly pitch in $10 to whoever fixes this. This bug affects me and I really would love to see it solved. I can help with any diagnostics.

https://www.bountysource.com/issues/58724362-authentication-error-with-google-drive

[Johannes Choo (jhanschoo)]

I noticed that I encounter this error when I am connecting via a VPN service, and do not encounter this error when I am not. If the reason is that Google rejects authenticating from suspicious IPs with the kind of authentication deja-dup and GNOME's online accounts use, I don't know if the team can do anything about it.

If this is indeed the case, I think a workaround by connecting from a non-blacklisted IP may work.

[Johannes Choo (jhanschoo)]

I would like to retract my report: my backups appear to not have completed.

[Usul_ (usul-)]

This happens to me to. I am not using a personal google account but a corporate one.
After 5 or 10 minutes I get an authentication error.
I am on ubuntu 18.04 using deja-dup 37-1 and duplicity 0.7.17

[Cody Hodges (cody35367)]

Perhaps the OAuth token is expiring during long backup jobs and needs to be renewed? Not sure what it is based on because our times vary widely.

https://developers.google.com/drive/api/v3/handle-errors#401_invalid_credentials

This happens to me after first-time backup running for about 45 minutes. Then it fails with (same as OP's log):
Giving up after 5 attempts. Error: gdata-service-error-quark: Authentication required: {
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}

Ubuntu 18.04.01 using:
deja-dup 37.1-2fakesync1
duplicity 0.7.17-0ubuntu1

[Prabhat Soni (ubuntudev2018)]

This bug seems to not have been fixed yet. Nobody seems to have posted its solution on the internet.
Description : I have a weekly backup scheduled. I do this on my google drive of my gmail account.

The backup happens for about 10 minutes. I'm unable to unmount my google drive from my computer this time and the computer tells me that some operations are keeping th volume busy.

Then I get this error message:-
This has been happening since 1 month continously and everytime I get this error message

Giving up after 5 attempts. Error: gdata-service-error-quark: Authentication required: {
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}
 (4)

[Clyton (clyton)]

Getting the same error on ubuntu-gnome 18.10 but I can see the backup files in google-drive. I don't know which part of the backup has failed
Giving up after 5 attempts. Error: gdata-service-error-quark: Authentication required: {
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}

[Andrew C. Dingman (adingman)]

I'm seeing exactly the same thing today on Fedora 29. It works fine on a new laptop with very little to back up, but a few gigs in to backing up my desktop I get the same error reported by the last two commenters.

[Andrew C. Dingman (adingman)]

Actually, slightly different. Probably not meaningfully so, but here it is just in case:

Giving up after 5 attempts. Error: Authentication required: {
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}

[Andrew C. Dingman (adingman)]

The same error happens with Duplicity on the command line, but it sucks less because resuming works.

$ duplicity --exclude=/home/<me>/{.cache,kolab,box.iu,Download/releases} --encrypt-key <mykey> /home/<me>/ gio+google-drive://<user>@gmail.com/<my_hostname>
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
Error accessing possibly locked file /home/<me>/Documents/<somedir>/<somefile>.pdf
Attempt 1 failed. Error: Authentication required: {
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}

Attempt 2 failed. Error: Authentication required: {
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}

Attempt 3 failed. Error: Authentication required: {
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}

Attempt 4 failed. Error: Authentication required: {
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}

Giving up after 5 attempts. Error: Authentication required: {
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}

$ duplicity --exclude=/home/<me>/{.cache,kolab,box.iu,Download/releases} --encrypt-key <mykey> /home/<me>/ gio+google-drive://<user>@gmail.com/<my_hostname>
Local and Remote metadata are synchronized, no sync needed.
Last full backup left a partial set, restarting.
Last full backup date: Mon Dec 31 15:56:49 2018
GnuPG passphrase:
RESTART: Volumes 220 to 221 failed to upload before termination.
         Restarting backup at volume 220.
Restarting after volume 219, file Music/<band>/<album>/<track>.flac, block 209
Error accessing possibly locked file /home/<me>/Documents/<somedir>/<somefile>.pdf

[Andrew C. Dingman (adingman)]

The bit about a possibly locked file can safely be ignored. For whatever reason, the PDF in question had mode 260.

[Michael Terry (mterry)]

I suspect this is an issue with gnome-online-accounts and how it is driving the authentication. I've filed an upstream bug: https://gitlab.gnome.org/GNOME/gnome-online-accounts/issues/36

[Andrew C. Dingman (adingman)]

Possibly, but I also notice that the gio backend doesn't implement _retry_cleanup(). Since re-running the same duplicity command can continue successfully without needing to re-start goa-daemon, I suspect there's something that could be done there to re-authenticate the connection.

Unfortunately, my Python skills are both limited and rusty. So far, all I've been able to do is confirm that simply re-invoking __init__() doesn't do the trick. I'll keep banging on it from that side as I have time.

[Michael Terry (mterry)]

I moved the above bug to GVFS: https://gitlab.gnome.org/GNOME/gvfs/issues/360

In that bug, I've attached a Python script that reproduces the problem. So this is not unique to duplicity or deja-dup.

However, we might reasonably work around it in duplicity by remounting or similar when such an error occurs. Maybe in _retry_cleanup as Andrew suggests.

[Adnan Iftekhar (livenicely)]

I am also facing this problem.

[Adnan Iftekhar (livenicely)]

Giving up after 5 attempts. Error: gdata-service-error-quark: Authentication required: {
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}
 (4)

[WcMinor (risingofthemoon)]

I can confirm, same issue here.

[Patrik Chynoransky (chynoranskypatrik)]

I had the same problem. I didn't figure out the cause of this issue, but found a very simple solution. The credentials had expired, so I needed to go to Settings > Online Accounts and select my Google account and log in again. After a few minutes the back-up restarted automatically and finished successfully.

[Yu Lou (louyu1999)]

The bug affects me even after I log out and log in again:

Giving up after 5 attempts. Error: gdata-service-error-quark: Authentication required: Unauthorized (4)

[Yu Lou (louyu1999)]

System information:

* Ubuntu 18.10
* Linux 4.18.0-16-generic #17-Ubuntu SMP Fri Feb 8 00:06:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
* duplicity 0.7.17-0ubuntu2
* deja-dup 38.0-1

[Vicente Oyanedel (vichoko)]

I have the same error while recovering a ~ 20 GB backup from Google Drive.

While recovering the encrypted .gpg files from the mounted Google Drive in Ubuntu's File Explorer directly to my disk i noticed that the connection hanged each 5 minutes at the same time i was trying to recover the data through Deja Dup Recovery function. The recovery through Deja Dup failed with the "Invalid Credentials" error, and the File Transfer through the file explorer raised an: "Error while copying *.gpg: Authentication required: Unauthorized"

So this fact support @mterry & cody35367's hypothesis of some kind of token expiration and renewal of the online accounts module.

Taking this is fact makes me wonder if maybe only 5 retries is too low? Maybe rising the timeout time to 30 sec-1 minute could be the solution meanwhile the token negotiation is slow.

[Francis (barcodes187)]

<email address hidden> is Francis

[Michael Terry (mterry)]

I have prepared a new beta release of deja-dup that might fix this issue. The 39.x series has switched from GNOME's Google keys to our own keys and from the GIO duplicity backend to the pydrive one. Which may alleviate some of these issues (on the presumption that part of it is that the GNOME key's quotas are being hit or how GIO is re-authorizing).

To test if the new 39.1 release helps, you can run:
snap install deja-dup --classic --beta

Or if you already had the snap installed:
snap refresh deja-dup --beta

When you next back up, you will be prompted to re-authorize deja-dup to access your Google account (with vastly diminished access than the GNOME keys, which is nice).

I will optimistically close the deja-dup side of this ticket. Please re-open and comment if the above release does not help matters.

[Robert C. Reynolds (robtcreynolds-o)]

I did get a full backup completed; however, there are a few details I need help working through.

Whereas the entire backup will now complete, there are unencryption errors when the volume is restore tested and there is where the backup fails. I apologize that I was unable to screenshot the message but it is a typical restore test fail due to inability to decrypt the volume message.

The other issue was relatively easy to mitigate but the file the backup goes to on Google Drive does not seem to be picked up by Deja-Dup and it will create a separate file series of the same name parallel to the existing.

[Robert C. Reynolds (robtcreynolds-o)]

I attempted to run a daily backup to supplement the failed backup. Oddly enough, the program recognized the original backup and attempted to build on it rather than clearing the volume and starting fresh. It did fail with this message:

Traceback (innermost last):
  File "/snap/deja-dup/126/bin/duplicity", line 1560, in <module>
    with_tempdir(main)
  File "/snap/deja-dup/126/bin/duplicity", line 1546, in with_tempdir
    fn()
  File "/snap/deja-dup/126/bin/duplicity", line 1398, in main
    do_backup(action)
  File "/snap/deja-dup/126/bin/duplicity", line 1528, in do_backup
    incremental_backup(sig_chain)
  File "/snap/deja-dup/126/bin/duplicity", line 664, in incremental_backup
    bytes_written = dummy_backup(tarblock_iter)
  File "/snap/deja-dup/126/bin/duplicity", line 227, in dummy_backup
    while tarblock_iter.next():
  File "/snap/deja-dup/126/lib/python2.7/site-packages/duplicity/diffdir.py", line 523, in next
    result = self.process(self.input_iter.next())
  File "/snap/deja-dup/126/lib/python2.7/site-packages/duplicity/diffdir.py", line 195, in get_delta_iter
    for new_path, sig_path in collated:
  File "/snap/deja-dup/126/lib/python2.7/site-packages/duplicity/diffdir.py", line 286, in collate2iters
    relem2 = riter2.next()
  File "/snap/deja-dup/126/lib/python2.7/site-packages/duplicity/diffdir.py", line 354, in combine_path_iters
    refresh_triple_list(triple_list)
  File "/snap/deja-dup/126/lib/python2.7/site-packages/duplicity/diffdir.py", line 341, in refresh_triple_list
    new_triple = get_triple(old_triple[1])
  File "/snap/deja-dup/126/lib/python2.7/site-packages/duplicity/diffdir.py", line 327, in get_triple
    path = path_iter_list[iter_index].next()
  File "/snap/deja-dup/126/lib/python2.7/site-packages/duplicity/diffdir.py", line 239, in sigtar2path_iter
    for tarinfo in tf:
  File "/usr/lib/python2.7/tarfile.py", line 2510, in next
    tarinfo = self.tarfile.next()
  File "/usr/lib/python2.7/tarfile.py", line 2352, in next
    raise ReadError("unexpected end of data")
 ReadError: unexpected end of data

[Ricardo Abreu (ricab)]

`snap install deja-dup --classic --beta` fixed it for me

[Michael Terry (mterry)]

Alright, the switch to our own keys seems to have fixed this. I’ll close the remaining tasks on this bug.

Ubuntu 19.10 uses the new key but has a separate issue with Google Drive: bug 1852069

[Besmir Zanaj (besmirzanaj-gmail)]

Using the default (apt) deja-dup with the online accoutns in ubuntu 18.04.
I found out a workaround for this. If you keep the session open manually by reading a file over and over you can reset the timer described here: https://gitlab.gnome.org/GNOME/gvfs/-/issues/360

Basically after keeping a session open on only a specific folder, there is a limit of 100 access things (reads, writes, etc.) and then the connection stales.

While the backup is running in the background, you have to manually browse the root of the google drive in nautilus every 5-10 minutes (file explorer) and reset that counter. You might get some errors like authentication failed but keep refreshing the root folder till you're able to browse files again. It finally completed my backup. This looks so easy to fix in the code tough and not sure why no one picked this up to fix this earlier.