[github] 20.04: Apt fails to download URLs with non-encoded querystrings

[TJ]

I've just helped a group of Ubuntu 20.04 users with Microsoft Surface devices. They rely upon a github repository releases pocket as the apt archive. Those users recently hit a bug "Ubuntu: Apt update fails with Error 401 Unauthorized" [1]

The initial simple Github URL gets a Location: redirect to a complex URL with un-escaped query-string. The resulting complex URL causes apt to fail to fetch the resource.

$ sudo apt upgrade -y
Err:1 https://pkg.surfacelinux.com/debian release/main amd64 libwacom-surface amd64 1.12-2
  401 Unauthorized [IP: 185.199.110.133 443]

The URL can be manually corrected. One of the Surface users provided this example:

bad: https://objects.githubusercontent.com/github-production-release-asset-2e65be/139604852/86019e52-7bfa-4bc6-8cc1-52147027aee6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A/20211105/us-east-1/s3/aws4_request&X-Amz-Date=20211105T161053Z&X-Amz-Expires=300&X-Amz-Signature=2bc0c28946db539ada250b1030c37249dae909d73a68c90b5e7bfe7fecd5d347&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=139604852&response-content-disposition=attachment; filename=libwacom-surface_1.12-2_amd64.deb&response-content-type=application/octet-stream

good: https://objects.githubusercontent.com/github-production-release-asset-2e65be/139604852/86019e52-7bfa-4bc6-8cc1-52147027aee6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211105%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211105T160935Z&X-Amz-Expires=300&X-Amz-Signature=44d9307e66dfb5b3672ee0082b8801ad2532ac4b6be61c3442fb265ffce72852&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=139604852&response-content-disposition=attachment%3B%20filename%3Dlibwacom-surface_1.12-2_amd64.deb&response-content-type=application%2Foctet-stream

I found this also affects a github repository I recently added for Zotero on 20.04.

The problem is fixed in later versions of apt upstream, in Debian and Ubuntu releases. I cherry-picked the 4 commits [3] and provided a package for Focal in my PPA [2] which multiple users have reported (in [1]) solves the issue.

It would be really good to get those patches included in 20.04 as an SRU.

As the code is in later versions of apt and is focused on the URL encoding only it has minimal potential for causing regressions.

[1] https://github.com/linux-surface/linux-surface/issues/625

[2] https://launchpad.net/~tj/+archive/ubuntu/bugfixes

[3] https://salsa.debian.org/apt-team/apt/-/commit/06ec0067057e0578f3bc515f6a97d6a9d70824f6

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[David Kalnischkies (donkult)]

"minimal potential for causing regressions" is a big claim given I had to fix regressions in later commits like 149b23c2b9697bc262c0af1934c7a3f6114d903f and 2b0369a5d1673d9e40f2af4db7677b040a26ee58. There might be more, that is just what I remember directly. It is certainly not the most complicated code in the world, but it's quite a bit of it as I was not trying for minimal, but instead maximized for forward and backward compat.

(Disclaimer: I am the upstream author of the patch set in question. Not involved enough with Ubuntu to know and/or predict if this qualifies or not for backport, so not commenting on that part. Pretty sure Debian would refuse if we tried including that in a stable update through).

[Dave Fenichel (boo-radley)]

This was indeed a cross-platform bug, for which the patch resolved the days-long issue with apt as described for me, running Linux Mint 20.2 (Uma). If not for the patch I would not have been able to apply my latest kernel update deb package.

[Julian Andres Klode (juliank)]

Marking as fix released. Do not plan to backport this due to significant regression potential

[sem (semitones)]

I installed apt from TJ's PPA, hopefully there are no regressions but it fixed the URL problem.

[Bart Groeneveld (bartgpx)]

I have used TJ's ppa for some months, and it worked flawlessly. Thanks TJ!

Unfortunately, Ubuntu released 2.0.8 for focal, which overwrites 2.0.6.2 from TJ's ppa.
Therefore, I created my own ppa[1]. My ppa is based on 2.0.8 and includes the same fixes as TJ's ppa.

I've just tested it while updating the linux-surface kernel for one of my surface devices, and it works as expected.

I expect to maintain the ppa as long I use focal myself, which I expect to be at least one year from now, and possibly longer. No guarantees, tho.

[1]: https://launchpad.net/~gpxbv/+archive/ubuntu/apt-urlfix