Update to v1.56.0 to fix: googleapi: Error 403: Only files with binary content can be downloaded #5455

[Stephen Warren]

[Impact]
When using rclone to download or sync a google drive native file, like a google doc, it will fail with a generic 403 permission denied problem.

[Test Plan]
Setup rclone to access google drive. The steps are not straightforward unfortunately, but are described in detail here: https://rclone.org/drive/

Then either sync all of the drive, or a folder that contains a native google doc file. For example, to sync the entire google drive remote, if it's called "mygdrive":

rclone sync -v mydrive: ./target

Versions with the bug will yield a message like this when trying to fetch the google doc:

2022/03/29 19:29:32 ERROR : somefile: Failed to copy: failed to open source object: open file failed: googleapi: Error 403: Only files with binary content can be downloaded. Use Export with Docs Editors files., fileNotDownloadable

Bionic, with an older version, has a slightly different error message:

2022/03/29 19:16:17 ERROR : somefile: Failed to copy: failed to open source object: forbidden to download - check sharing permission

The fixed version will download those files without that error.

[Where problems could occur]
In the patch, the method used to identify what is a google doc file and what isn't seems flimsy: "google docs are identified solely by not having an MD5 checksum." I checked current git tip and the applied patch wasn't changed or reverted.

This file type check determines which download method is used, and if there are other files that match this check but require yet another way to download them, this could fail again in the future.

Something else that occurred to me is if there remains some google api endpoint which hasn't made this api change yet, if that would fail with the new rclone package. Seems unlikely after all this time (this bug is from july 2021), but I have no data to back that up other than my own testing with my own gdrive account.

[Other Info]
At one point while testing the bionic version, I was getting panics from rclone. I then reverted to the original bionic version, and still got a panic. After a while, both original, and patched versions, decided to work. My assumption is that there was some random network error, or maybe a google api response, that the bionic version didn't know how to handle, and crashed. Since it also happened with the rclone version as released with bionic, I don't think it's related to this fix.

[Original Description]

rclone version v1.56.0 (or at least, a beta release of that version) is required to fix rclone to adapt to a recent change in the Google Drive API. Older versions, such as the currently packaged 1.50.2-2 on focal, fail with the following error on some files:

ERROR : ...filename...: Failed to copy: failed to open source object: open file failed: googleapi: Error 403: Only files with binary content can be downloaded. Use Export with Docs Editors files., fileNotDownloadable

Reference: Upstream bug: https://github.com/rclone/rclone/issues/5455

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andreas Hasenack (ahasenack)]

Upstream commit: https://github.com/rclone/rclone/commit/c4aabaf490cdb6ecfacf379dc15adaf60a62d8f4

[Andreas Hasenack (ahasenack)]

Better yet, this one: https://github.com/rclone/rclone/commit/94b1439299e57f84dec27e865fba2b7356f3ea79

[Andreas Hasenack (ahasenack)]

I created this ppa with impish and hirsute packages for now, if you want to give them a try. It fixed the problem for me in my raspberry pi4 backup server:

https://launchpad.net/~ahasenack/+archive/ubuntu/rclone-gdrive-download-1937136

I may start an SRU for this in my spare time.

The git branches are at https://code.launchpad.net/~ahasenack/ubuntu/+source/rclone/+git/rclone

[Andreas Hasenack (ahasenack)]

https://github.com/rclone/rclone/issues/5455

[Launchpad Janitor (janitor)]

This bug was fixed in the package rclone - 1.53.3-4ubuntu1

---------------
rclone (1.53.3-4ubuntu1) jammy; urgency=medium

  * d/p/0008-fix-gdrive-download.patch: remove check for size, so google
    docs are identified solely by not having an MD5 checksum
    (LP: #1937136)

 -- Andreas Hasenack <email address hidden> Mon, 28 Mar 2022 18:45:23 -0300

[Brian Murray (brian-murray)]

Hello Stephen, or anyone else affected,

Accepted rclone into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/rclone/1.53.3-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Stephen, or anyone else affected,

Accepted rclone into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/rclone/1.50.2-2ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Stephen, or anyone else affected,

Accepted rclone into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/rclone/1.36-3ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Stephen Warren (srwarren)]

I tested https://launchpad.net/ubuntu/+source/rclone/1.50.2-2ubuntu0.1/+build/23542458 and it solves the problem.

Specifically:
- I had installed upstream build 1.56.0 to work around the problem, which was working fine.
- I uninstalled that version, and installed Ubuntu's 1.50.2-2, and reproduced the original problem.
- I upgraded to proposed package 1.50.2-2ubuntu0.1, and verified that it solves the problem.

[Andreas Hasenack (ahasenack)]

Impish verification

Reproducing the bug with:
$ apt-cache policy rclone
rclone:
  Installed: 1.53.3-1
  Candidate: 1.53.3-1
  Version table:
 *** 1.53.3-1 500
        500 http://br.archive.ubuntu.com/ubuntu impish/universe amd64 Packages

ubuntu@i-rclone-sru-1937136:~$ rclone sync -P gdrive_panlinux: $(pwd)/out
2022-07-14 12:45:42 ERROR : Compras Natal 2021: Failed to copy: failed to open source object: open file failed: googleapi: Error 403: Only files with binary content can be downloaded. Use Export with Docs Editors files., fileNotDownloadable
(...)

With the updated version:
ubuntu@i-rclone-sru-1937136:~$ apt-cache policy rclone
rclone:
  Installed: 1.53.3-1ubuntu0.1
  Candidate: 1.53.3-1ubuntu0.1
  Version table:
 *** 1.53.3-1ubuntu0.1 500
        500 http://br.archive.ubuntu.com/ubuntu impish-proposed/universe amd64 Packages

There are no such errors:
ubuntu@i-rclone-sru-1937136:~$ rclone sync -P gdrive_panlinux: $(pwd)/out
Transferred: 325.906M / 1.748 GBytes, 18%, 1.535 MBytes/s, ETA 15m53s
Transferred: 677 / 2273, 30%
(...)

I didn't let it finish just for the sake of the test, it's a lot to download. But the previous package failed almost immediately, and I can verify that the file that failed previously was already downloaded by this version:

$ find out/ -name 'Compras*2021*' -ls
   103022 9 -rw-rw-r-- 1 ubuntu ubuntu 13094 Dec 29 2021 out/Compras\ Natal\ 2021.xlsx

Impish verification succeeded.

[Andreas Hasenack (ahasenack)]

Bionic verification

First, reproducing the bug with the bionic package:
ubuntu@b-rclone-sru-1937136:~$ apt-cache policy rclone
rclone:
  Installed: 1.36-3
  Candidate: 1.36-3
  Version table:
 *** 1.36-3 500
        500 http://br.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

It fails to download a specific file:
ubuntu@b-rclone-sru-1937136:~$ rclone sync -v gdrive_panlinux: $(pwd)/out
2022/07/14 13:00:00 INFO : Local file system at /home/ubuntu/out: Modify window is 1ms
(...)
2022/07/14 13:00:23 ERROR : Compras Natal 2021: Failed to copy: failed to open source object: forbidden to download - check sharing permission
(...)

ubuntu@b-rclone-sru-1937136:~$ find out/ -name 'Compras*2021*' -ls
ubuntu@b-rclone-sru-1937136:~$

Then I update the package to what is in bionic-proposed:
ubuntu@b-rclone-sru-1937136:~$ apt-cache policy rclone
rclone:
  Installed: 1.36-3ubuntu0.1
  Candidate: 1.36-3ubuntu0.1
  Version table:
 *** 1.36-3ubuntu0.1 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-proposed/universe amd64 Package

And the file is downloaded this time:
ubuntu@b-rclone-sru-1937136:~$ rclone sync -v gdrive_panlinux: $(pwd)/out
2022/07/14 13:02:41 INFO : Local file system at /home/ubuntu/out: Modify window is 1ms
(...)
2022/07/14 13:03:07 INFO : Compras Natal 2021.xlsx: Copied (new)
(...)

ubuntu@b-rclone-sru-1937136:~$ find out/ -name 'Compras*2021*' -ls
    77230 9 -rw-rw-r-- 1 ubuntu ubuntu 13094 Dec 29 2021 out/Compras\ Natal\ 2021.xlsx

Bionic verification succeeded

[Launchpad Janitor (janitor)]

This bug was fixed in the package rclone - 1.53.3-1ubuntu0.1

---------------
rclone (1.53.3-1ubuntu0.1) impish; urgency=medium

  * d/p/0008-fix-gdrive-download.patch: remove check for size, so google
    docs are identified solely by not having an MD5 checksum
    (LP: #1937136)

 -- Andreas Hasenack <email address hidden> Mon, 28 Mar 2022 18:47:53 -0300

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for rclone has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package rclone - 1.50.2-2ubuntu0.1

---------------
rclone (1.50.2-2ubuntu0.1) focal; urgency=medium

  * d/p/0008-fix-gdrive-download.patch: remove check for size, so google
    docs are identified solely by not having an MD5 checksum
    (LP: #1937136)
  * d/control: add missing build-depends for
    golang-github-mattn-go-colorable-dev (LP: #1966950)

 -- Andreas Hasenack <email address hidden> Tue, 29 Mar 2022 10:28:41 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package rclone - 1.36-3ubuntu0.1

---------------
rclone (1.36-3ubuntu0.1) bionic; urgency=medium

  * d/p/0008-fix-gdrive-download.patch: remove check for size, so google
    docs are identified solely by not having an MD5 checksum
    (LP: #1937136)

 -- Andreas Hasenack <email address hidden> Tue, 29 Mar 2022 11:07:53 -0300