apt https method decodes redirect locations and sends them to the destination undecoded.

[Nico Orrù]

[Impact]
Downloads via HTTPS fail if the URL contains a space (before yakkety only if there is no redirect from a previous space-free https URL). This breaks packages like ttf-mscorefonts-installer and various third party hosters.

[Test case]
Install/Upgrade apt-transport-https, that's where the fix is.

Check that

/usr/lib/apt/apt-helper download-file http://kxstudio.linuxaudio.org/repo/pool/free/ardour4_4.7.0-1kxstudio1_i386.deb test.deb

can successfully download the file (or at least start downloading it) and does not fail early with a 505 HTTP version not supported error message.

This problem does not occur with that file on xenial, as it first redirects to an https URI without a space which then redirects to an HTTPS uri with a space (http w/o space -> https w/o space -> https w/ space). In xenial, https->https redirects where handled internally by curl.

Another test (applicable to xenial) is to install ttf-mscorefonts-installer.

[Regression potential]
The added code is:
   Uri.Path = QuoteString(Uri.Path, "+~ ");

Some servers might not like + or ~ being quoted. We use the same quoting call for the http method too, though, so it seems highly unlikely to cause an issue.

[Original bug report]
Distributor ID: Ubuntu
Description: Ubuntu 16.10
Release: 16.10
Codename: yakkety

apt version 1.3.3 (also tried 1.4-beta2 .deb, same results)

When trying to install a package hosted on s3 from the kxstudio repo, the download fails with an HTTP error:

nico@nico-lenovo-ubuntu:~/Downloads$ sudo apt-get install wineasio-amd64
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  wine1.6-amd64
The following NEW packages will be installed
  wine1.6-amd64 wineasio-amd64
0 to upgrade, 2 to newly install, 0 to remove and 1 not to upgrade.
Need to get 30.9 kB/32.6 kB of archives.
After this operation, 184 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Err:1 http://kxstudio.linuxaudio.org/repo stable/free amd64 wineasio-amd64 amd64 0.9.0+git20110613-2kxstudio3
  505 HTTP Version not supported
E: Failed to fetch https://github-cloud.s3.amazonaws.com/releases/39372848/0f048802-2fb5-11e5-9d8c-907ec7b97c46.deb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ/20161222/us-east-1/s3/aws4_request&X-Amz-Date=20161222T022041Z&X-Amz-Expires=300&X-Amz-Signature=750f9b2ee076dcb8ae6992cae911f43208b3eec41976362cebf694e3c72b7aef&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment; filename=wineasio-amd64_0.9.0.git20110613-2kxstudio3_amd64.deb&response-content-type=application/octet-stream 505 HTTP Version not supported
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Error allegedly not present in Ubuntu 14.04 and 16.04

More details in these forum posts:

https://github.com/KXStudio/Repository/issues/73#issuecomment-268649503

https://www.linuxmusicians.com/viewtopic.php?t=16056

https://www.drupal.org/node/2324991 (clues on root cause)

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: apt 1.3.3
ProcVersionSignature: Ubuntu 4.8.0-30.32-lowlatency 4.8.6
Uname: Linux 4.8.0-30-lowlatency x86_64
ApportVersion: 2.20.3-0ubuntu8.2
Architecture: amd64
CurrentDesktop: X-Cinnamon
Date: Thu Dec 22 02:31:47 2016
InstallationDate: Installed on 2016-10-20 (62 days ago)
InstallationMedia: Ubuntu 16.10 "Yakkety Yak" - Release amd64 (20161012.2)
SourcePackage: apt
UpgradeStatus: No upgrade log present (probably fresh install)

[Julian Andres Klode (juliank)]

Right:

 /usr/lib/apt/apt-helper download-file http://kxstudio.linuxaudio.org/repo/pool/free/ardour4_4.7.0-1kxstudio1_i386.deb test.deb
Err:1 http://kxstudio.linuxaudio.org/repo/pool/free/ardour4_4.7.0-1kxstudio1_i386.deb
  505 HTTP Version not supported

[Julian Andres Klode (juliank)]

Adding curl to the loop. It works fine on Debian, but fails in yakkety, and there have been no changes in our https module.

Attaching a log: We send the server a HTTP/1.1 request and it responds with one

[Julian Andres Klode (juliank)]

Ah no, my Debian was just using a proxy in between. It fails there too.

[Julian Andres Klode (juliank)]

I think the problem is that there is a space in the URI before "filename=", and that is not encoded before it being sent out, causing the server to think "filename=..." is the http version...

[Nico Orrù (nigu-orru)]

t does appear that the redirect url does contain an encoded space, and that at some point in the apt pipe it gets decoded and not re-encoded back.

see below:

Location: https://github-cloud.s3.amazonaws.com/releases/39372848/4074c29a-d734-11e5-8632-2259aafc2c91.deb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ%2F20161222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20161222T161758Z&X-Amz-Expires=300&X-Amz-Signature=9b01bd5373576686821a64246a49f2aa743bd61a8a263344903424b177c58a20&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dardour4_4.7.0-1kxstudio1_i386.deb&response-content-type=application%2Foctet-stream

nico@nico-lenovo-ubuntu:~$ curl -vvv http://kxstudio.linuxaudio.org/repo/pool/free/ardour4_4.7.0-1kxstudio1_i386.deb
* Trying 198.82.152.114...
* Connected to kxstudio.linuxaudio.org (198.82.152.114) port 80 (#0)
> GET /repo/pool/free/ardour4_4.7.0-1kxstudio1_i386.deb HTTP/1.1
> Host: kxstudio.linuxaudio.org
> User-Agent: curl/7.50.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 22 Dec 2016 16:17:10 GMT
< Server: Apache
< Location: https://github.com/KXStudio/Repository/releases/download/initial/ardour4_4.7.0-1kxstudio1_i386.deb
< Vary: Accept-Encoding
< Content-Length: 379
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://github.com/KXStudio/Repository/releases/download/initial/ardour4_4.7.0-1kxstudio1_i386.deb">here</a>.</p>
<hr>
<address>Apache Server at kxstudio.linuxaudio.org Port 80</address>
</body></html>
* Connection #0 to host kxstudio.linuxaudio.org left intact
nico@nico-lenovo-ubuntu:~$ curl https://github.com/KXStudio/Repository/releases/download/initial/ardour4_4.7.0-1kxstudio1_i386.deb
<html><body>You are being <a href="https://github-cloud.s3.amazonaws.com/releases/39372848/4074c29a-d734-11e5-8632-2259aafc2c91.deb?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ%2F20161222%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20161222T161741Z&amp;X-Amz-Expires=300&amp;X-Amz-Signature=3faf5a0d96bf1e4762cc6ab598a8497c40cf439c33ce7b592705e60b9486d5d1&amp;X-Amz-SignedHeaders=host&amp;actor_id=0&amp;response-content-disposition=attachment%3B%20filename%3Dardour4_4.7.0-1kxstudio1_i386.deb&amp;response-content-type=application%2Foctet-stream">redirected</a>.</body></html>nico@nico-lenovo-ubuntu:~$
nico@nico-lenovo-ubuntu:~$ curl -vvv https://github.com/KXStudio/Repository/releases/download/initial/ardour4_4.7.0-1kxstudio1_i386.deb
* Trying 192.30.253.113...
* Connected to github.com (192.30.253.113) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 697 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: github.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
...

[Nico Orrù (nigu-orru)]

Command line curl does not seem to be affected:

```
nico@nico-lenovo-ubuntu:~$ curl -L -vvv http://kxstudio.linuxaudio.org/repo/pool/free/ardour4_4.7.0-1kxstudio1_i386.deb -o ardour.deb
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 198.82.152.114...
* Connected to kxstudio.linuxaudio.org (198.82.152.114) port 80 (#0)
> GET /repo/pool/free/ardour4_4.7.0-1kxstudio1_i386.deb HTTP/1.1
> Host: kxstudio.linuxaudio.org
> User-Agent: curl/7.50.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 22 Dec 2016 21:45:57 GMT
< Server: Apache
< Location: https://github.com/KXStudio/Repository/releases/download/initial/ardour4_4.7.0-1kxstudio1_i386.deb
< Vary: Accept-Encoding
< Content-Length: 379
< Content-Type: text/html; charset=iso-8859-1
<
* Ignoring the response-body
{ [379 bytes data]
100 379 100 379 0 0 1029 0 --:--:-- --:--:-- --:--:-- 1029
* Connection #0 to host kxstudio.linuxaudio.org left intact
* Issue another request to this URL: 'https://github.com/KXStudio/Repository/releases/download/initial/ardour4_4.7.0-1kxstudio1_i386.deb'
* Trying 192.30.253.113...
* Connected to github.com (192.30.253.113) port 443 (#1)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 697 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: github.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject:
* start date: Thu, 10 Mar 2016 00:00:00 GMT
* expire date: Thu, 17 May 2018 12:00:00 GMT
* issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 Extended Validation Server CA
* compression: NULL
* ALPN, server accepted to use http/1.1
> GET /KXStudio/Repository/releases/download/initial/ardour4_4.7.0-1kxstudio1_i386.deb HTTP/1.1
> Host: github.com
> User-Agent: curl/7.50.1
> Accept: */*
>
< HTTP/1.1 302 Found
< Server: GitHub.com
< Date: Thu, 22 Dec 2016 21:45:58 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Status: 302 Found
< Cache-Control: no-cache
< Vary: X-PJAX
< Location: https://github-cloud.s3.amazonaws.com/releases/39372848/4074c29a-d734-11e5-8632-2259aafc2c91.deb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ%2F20161222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20161222T214558Z&X-Amz-Expires=300&X-Amz-Signature=c42514a341b5775702cfa0e797d09d3ae3388257c936128fe26809f1ac04018f&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dardour4_4.7.0-1kxstudio1_i386.deb&response-content-type=application%2Foctet-stream
< X-UA-Compatible: IE=Edge,chrome=1
< Set-Cookie: logged_in=no; domain=.github.com; path=/; expires=Mon, 22 Dec 2036 21:45:58 -00...

[Nico Orrù (nigu-orru)]

nico@nico-lenovo-ubuntu:~$ curl --version
curl 7.50.1 (x86_64-pc-linux-gnu) libcurl/7.50.1 GnuTLS/3.5.3 zlib/1.2.8 libidn/1.33 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets

[Nico Orrù (nigu-orru)]

The redirect URL comes directly from the github server, by the way.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Julian Andres Klode (juliank)]

The problem is: the http methods URL-encodes URLs before sending them, the https one does not. And our redirecting code decodes the locations given, because the http method encodes them.

This is of course horribly broken: We should not decode the location and re-encode it in the first place. That said, we can't fix this right now, so a simple work around for this issue is probably to just do the quoting in the https method as well.

I hope we can get rid of the https method, and just add TLS support to our own method - that will get rid of a lot of problems with the https one.

[Julian Andres Klode (juliank)]

Currently running CI on https://github.com/Debian/apt/compare/master...julian-klode:bugfix/lp-1653094-https-quote?expand=1 let's see if that simple change works or if it needs more work.

[Nico Orrù (nigu-orru)]

Good news, thanks! Would it be possible to schedule a backported patch to 16.10? The problem is significant in the general case.

[Julian Andres Klode (juliank)]

I believe this will take quite some time to fix. A correct fix will have to wait until the end of the year, but we might be able to hack something in like what I did there - but really only quoting the local part and not the entire URL - which obviously fails.

[nmaxx (nmaxx)]

"A correct fix will have to wait until the end of the year" - do you really mean the end of 2017?
But even if you can only release a workaround for now, please backport to 16.04 LTS as well, where it broke e.g. the ttf-mscorefonts-installer.

[Nico Orrù (nigu-orru)]

A year to fix a regression (bug not apparent in 14.04)? Oh, dear! :(

[T. Meyer (tm-an)]

+1 for a backport, even if it's a "dirty hack"

[Julian Andres Klode (juliank)]

No regression. The quick hack we can try for now is parsing the URI we get and then encoding the local part. This is what I'm aiming for this month and it will fix this issue.

The correct fix requires restructuring the whole acquire system to not decode URIs in redirect requests and encode given URIs before sending the request. Since APT is freezing down for Debian stretch soon, that's likely not going to happen anytime soon (that targets apt 1.5, so 17.10 or 18.04).

[Julian Andres Klode (juliank)]

Bah, I was not clear. Github apparently started adding spaces into their URI. That's what is causing this issue, it's not really a change in apt that is causing it. That said, the redirect handling changed a bit - https redirects used to be handled by curl itself prior to 1.3.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Julian Andres Klode (juliank)]

I just uploaded 1.4~rc3ubuntu1 to zesty. It's building now, and should hit zesty-proposed soon, and hopefully migrate without issues to zesty release repository.

This is the included workaround for now:

commit 12d5863a6ecd358db5645a4c1ca75576ef3c6232
Author: Julian Andres Klode <email address hidden>
Date: Thu Dec 29 14:16:07 2016 +0100

    https: Quote path in URL before passing it to curl

    Curl requires URLs to be urlencoded. We are however giving it
    undecoded URLs. This causes it go completely nuts if there is
    a space in the URI, producing requests like:

        GET /a file HTTP/1.1

    which the servers then interpret as a GET request for "/a" with
    HTTP version "file" or some other non-sense.

    This works around the issue by encoding the path component of
    the URL. I'm not sure if we should encode other parts of the URL
    as well, this one seems to do the trick for the actual issue at
    hand.

    A more correct fix is to avoid the dequoting and (re-)quoting
    of URLs when a redirect occurs / a new request is sent. That's
    been on the radar for probably a year or two now, but nobody
    bothered implementing that yet.

    LP: #1651923

diff --git a/methods/https.cc b/methods/https.cc
index d60bc6fbc..d71ef0bf0 100644
--- a/methods/https.cc
+++ b/methods/https.cc
@@ -275,6 +275,10 @@ bool HttpsMethod::Fetch(FetchItem *Itm)
    if (Server == nullptr || Server->Comp(Itm->Uri) == false)
       Server = CreateServerState(Itm->Uri);

+ // The "+" is encoded as a workaround for a amazon S3 bug
+ // see LP bugs #1003633 and #1086997. (taken from http method)
+ Uri.Path = QuoteString(Uri.Path, "+~ ");
+
    FetchResult Res;
    RequestState Req(this, Server.get());
    CURLUserPointer userp(this, &Res, Itm, &Req);

If that works fine for everyone, I can cherry-pick this into xenial and yakkety. The SRU procedure takes a few weeks though - in fact, many of the current changes pending for xenial are waiting since 2 months already.

Trusty also has an issue with the downloads, BTW, but it just hangs completely, so there's probably a different issue there.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.4~beta3ubuntu1

---------------
apt (1.4~beta3ubuntu1) zesty; urgency=medium

  * https: Quote path in URL before passing it to curl (LP: #1651923)

 -- Julian Andres Klode <email address hidden> Wed, 11 Jan 2017 00:13:59 +0100

[Jeremy Bícha (jbicha)]

Bumping the severity because this is causing problems for a lot of people. There are hundreds of people who have marked this bug (or a duplicate) as affecting them.

[Catalin Popescu (catalinp86)]

As far as ttf-mscorefonts-installer is concerned, I looked on the sourceforge website and it's no longer at http://downloads.sourceforge.net/corefonts/andale32.exe . The file has been moved and I velieve is located at : https://sourceforge.net/projects/corefonts/files/the fonts/final/andale32.exe

[Naël (nathanael-naeri)]

downloads.sourceforge.net is just a redirection service to an auto-selected mirror. Using the full URL (sourceforge.net/projects/corefonts/files/...) also auto-selects a mirror (credited on the right of the web page: "Mirror provided by ...").

[Julian Andres Klode (juliank)]

Fix has been committed upstream with an additional test case:

https://anonscm.debian.org/cgit/apt/apt.git/commit/?id=994515e689dcc5f963f5fed58284831750a5da03

I'll sync the new version from Debian unstable once I have uploaded and it is known by Launchpad. I will also upload SRUs tomorrow - the versions are:

  1.2.19 for xenial
  1.3.4 for yakkety

As the change is small we should hopefully get this reviewed easily by the SRU team.

[Julian Andres Klode (juliank)]

Releases uploaded. The fix is exactly the same as in zesty, I just added an additional test case based on the new 1.4~beta4 release (the test case also needed some porting because the framework between 1.4 and 1.3 is a bit different).

[Naël (nathanael-naeri)]

Thanks for the hard work!

[Chris J Arges (arges)]

Hello Nico, or anyone else affected,

Accepted apt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.2.19 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Nico, or anyone else affected,

Accepted apt into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[luca (llucax)]

Should we test that package too in xenial? Or there will be another test package for xenial?

[Julian Andres Klode (juliank)]

@llucax there are two versions (and two calls for testing). 1.2.19 for xenial and 1.3.4 for yakkety.

[Robin (robingape)]

Just checked apt 1.2.9 from proposed on an x86_64 16.04 system . The package ttf-mscorefonts-installer version 3.4+nmu1ubuntu2 fails to download font files, but rather returns a 403 error on the first file download attempt.
Directly accessing the failed link through a browser gives the expected result of a download being offered.

[Brian Murray (brian-murray)]

While the new version of the package worked for me, so did apt version 1.2.18 - shouldn't it have failed?

bdmurray@clean-xenial-amd64:~$ apt-cache policy apt
apt:
  Installed: 1.2.18
  Candidate: 1.2.18
  Version table:
 *** 1.2.18 500
        500 http://192.168.10.7/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.2.15ubuntu0.2 500
        500 http://192.168.10.7/ubuntu xenial-security/main amd64 Packages
     1.2.10ubuntu1 500
        500 http://192.168.10.7/ubuntu xenial/main amd64 Packages
bdmurray@clean-xenial-amd64:~$ /usr/lib/apt/apt-helper download-file http://kxstudio.linuxaudio.org/repo/pool/free/ardour4_4.7.0-1kxstudio1_i386.deb test.deb
Get:1 http://kxstudio.linuxaudio.org/repo/pool/free/ardour4_4.7.0-1kxstudio1_i386.deb [60.2 MB]
Fetched 60.2 MB in 10s (5,701 kB/s)

[Julian Andres Klode (juliank)]

Could you provide logs with -o debug::acquire::https=1 and debug::acquire::http=1

Maybe thw redirect changed and the test case this started working again? I'll check soon.

[Julian Andres Klode (juliank)]

@Robin: I just confirmed that apt 1.2.19 fixes the corefonts issue:

All fonts downloaded and installed.
Setting up ttf-mscorefonts-installer (3.4+nmu1ubuntu2) ...

Did you really upgrade apt-transport-https or only apt?

But bdmurray is right, the ardour thing works fine in 1.2.18. In xenial with 1.3.3 it fails with
"505 HTTP Version not supported" though.

[Brian Murray (brian-murray)]

Here is the debug output with 1.2.18.

[Julian Andres Klode (juliank)]

This problem does not occur with that file on xenial, as it first redirects to an https URI without a space which then redirects to an HTTPS uri with a space (http w/o space -> https w/o space -> https w/ space). In xenial, https->https redirects where handled internally by curl.

Another test (applicable to xenial) is to install ttf-mscorefonts-installer.

And please note: You need to upgrade apt-transport-https for testing, just apt does nothing.

[Elvis Stansvik (elvstone)]

I'm afraid the 1.2.19 in xenial-proposed does not seem to solve the problem here:

estan@newton:~$ apt-cache policy apt
apt:
  Installerad: 1.2.19
  Kandidat: 1.2.19
  Versionstabell:
 *** 1.2.19 500
        500 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1.2.18 500
        500 http://se.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
     1.2.15ubuntu0.2 500
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
     1.2.10ubuntu1 500
        500 http://se.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
estan@newton:~$ /usr/lib/apt/apt-helper download-file http://downloads.sourceforge.net/corefonts/andale32.exe test.exe
Fel:1 http://downloads.sourceforge.net/corefonts/andale32.exe
  404 Not Found
E: Misslyckades med att hämta https://vorboss.dl.sourceforge.net/project/corefonts/the fonts/final/andale32.exe 404 Not Found

E: Hämtning misslyckades
estan@newton:~$ wget http://downloads.sourceforge.net/corefonts/andale32.exe
--2017-01-18 19:10:15-- http://downloads.sourceforge.net/corefonts/andale32.exe
Slår upp downloads.sourceforge.net (downloads.sourceforge.net)... 216.34.181.59
Ansluter till downloads.sourceforge.net (downloads.sourceforge.net)|216.34.181.59|:80... ansluten.
HTTP-begäran skickad, väntar på svar... 301 Moved Permanently
Adress: http://downloads.sourceforge.net/project/corefonts/the%20fonts/final/andale32.exe [följer]
--2017-01-18 19:10:15-- http://downloads.sourceforge.net/project/corefonts/the%20fonts/final/andale32.exe
Ansluter till downloads.sourceforge.net (downloads.sourceforge.net)|216.34.181.59|:80... ansluten.
HTTP-begäran skickad, väntar på svar... 302 Found
Adress: https://netcologne.dl.sourceforge.net/project/corefonts/the%20fonts/final/andale32.exe [följer]
--2017-01-18 19:10:16-- https://netcologne.dl.sourceforge.net/project/corefonts/the%20fonts/final/andale32.exe
Slår upp netcologne.dl.sourceforge.net (netcologne.dl.sourceforge.net)... 78.35.24.46, 2001:4dd0:1234:6::5f
Ansluter till netcologne.dl.sourceforge.net (netcologne.dl.sourceforge.net)|78.35.24.46|:443... ansluten.
HTTP-begäran skickad, väntar på svar... 200 OK
Längd: 198384 (194K) [application/octet-stream]
Sparar till: "andale32.exe"

andale32.exe 100%[========================================================================================================================================>] 193,73K 1,25MB/s in 0,2s

2017-01-18 19:10:16 (1,25 MB/s) - "andale32.exe" sparades [198384/198384]

estan@newton:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
estan@newton:~$

And I'm afraid it's not deterministic; this is the output from three runs:

estan@newton:~$ /usr/lib/apt/apt-helper download-file http://downloads.sourceforge.net/corefonts/andale32.exe test.exe
Fel:1 http://downloads.sourceforge.net/corefonts/andale32.exe
  404 Not Found
E: Misslyckades med att hämta https://netix.dl.sourceforge.net/project/corefonts/the fonts/final/andale32.exe 404 Not Found

E: Hämtning misslyckades
estan@newton:~$ /usr...

[Julian Andres Klode (juliank)]

Again: You need to install apt-transport-https, not apt. Nobody cares about your version of the apt package, the fix is in apt-transport-https.

[Julian Andres Klode (juliank)]

Another test case is downloading

"https://people.debian.org/~jak/a b/c"

with apt-helper. That fails in 1.2.18 and succeeds in 1.2.19.

[Elvis Stansvik (elvstone)]

"And please note: You need to upgrade apt-transport-https for testing, just apt does nothing."

Aha, I might have forgotten to upgrade apt-transport-https... But, I just tested, and now I'm getting:

estan@newton:~$ /usr/lib/apt/apt-helper download-file http://downloads.sourceforge.net/corefonts/andale32.exe test.exe
Err:1 http://downloads.sourceforge.net/corefonts/andale32.exe
  The HTTP server sent an invalid Content-Range header
E: Failed to fetch https://netassist.dl.sourceforge.net/project/corefonts/the fonts/final/andale32.exe The HTTP server sent an invalid Content-Range header

E: Download Failed
estan@newton:~$ /usr/lib/apt/apt-helper download-file http://downloads.sourceforge.net/corefonts/andale32.exe test.exe
Err:1 http://downloads.sourceforge.net/corefonts/andale32.exe
  The HTTP server sent an invalid Content-Range header
E: Failed to fetch https://netcologne.dl.sourceforge.net/project/corefonts/the fonts/final/andale32.exe The HTTP server sent an invalid Content-Range header

E: Download Failed
estan@newton:~$ /usr/lib/apt/apt-helper download-file http://downloads.sourceforge.net/corefonts/andale32.exe test.exe
Err:1 http://downloads.sourceforge.net/corefonts/andale32.exe
  The HTTP server sent an invalid Content-Range header
E: Failed to fetch https://netcologne.dl.sourceforge.net/project/corefonts/the fonts/final/andale32.exe The HTTP server sent an invalid Content-Range header

E: Download Failed
estan@newton:~$ apt-cache policy apt apt-transport-https
apt:
  Installed: 1.2.19
  Candidate: 1.2.19
  Version table:
 *** 1.2.19 500
        500 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1.2.18 500
        500 http://se.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
     1.2.15ubuntu0.2 500
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
     1.2.10ubuntu1 500
        500 http://se.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
apt-transport-https:
  Installed: 1.2.19
  Candidate: 1.2.19
  Version table:
 *** 1.2.19 500
        500 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1.2.18 500
        500 http://se.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
     1.2.15ubuntu0.2 500
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
     1.2.10ubuntu1 500
        500 http://se.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
estan@newton:~$

With debug output, I first got:

estan@newton:~$ /usr/lib/apt/apt-helper -o debug::acquire::https=1 -o debug::acquire::http=1 download-file http://downloads.sourceforge.net/corefonts/andale32.exe test.exe
0% [Working]GET /corefonts/andale32.exe HTTP/1.1
Host: downloads.sourceforge.net
Range: bytes=198384-
If-Range: Thu, 15 Aug 2002 14:33:49 GMT
User-Agent: Debian APT-HTTP/1.3 (1.2.19)

Answer for: http://downloads.sourceforge.net/corefonts/andale32.exe
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Wed, 18 Jan 2017 18:30:14 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Location: http://downloads.sourceforge.net/project/corefonts/the%20fonts/final...

[Nico Orrù (nigu-orru)]

Yep, fixed! Great work! Thanks for your time!

```
norru@GBWWSRUNUBWS02:~$ sudo apt install apt-transport-https
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  apt-transport-https
1 to upgrade, 0 to newly install, 0 to remove and 76 not to upgrade.
Need to get 34.3 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://gb.archive.ubuntu.com/ubuntu yakkety-proposed/main amd64 apt-transport-https amd64 1.3.4 [34.3 kB]
Fetched 34.3 kB in 0s (679 kB/s)
(Reading database ... 348718 files and directories currently installed.)
Preparing to unpack .../apt-transport-https_1.3.4_amd64.deb ...
Unpacking apt-transport-https (1.3.4) over (1.3.3) ...
Setting up apt-transport-https (1.3.4) ...
norru@GBWWSRUNUBWS02:~$ /usr/lib/apt/apt-helper download-file http://kxstudio.linuxaudio.org/repo/pool/free/ardour4_4.7.0-1kxstudio1_i386.deb test.deb
Get:1 http://kxstudio.linuxaudio.org/repo/pool/free/ardour4_4.7.0-1kxstudio1_i386.deb [60.2 MB]
Fetched 60.2 MB in 6s (8,814 kB/s)
```

[Julian Andres Klode (juliank)]

The first log looks entirely correct. The server is broken, vorboss does not support partial requests, but instead of responding correctly with all the content, it just redirects to another location. That other location is http, so unencrypted which we do not allow for security reasons.

Re the invalid content-range - Delete the file and it will work again. Their servers have another weird thing they do when replying to a request for partial content:

HTTP/1.1 302 Moved Temporarily
[...]
Content-Range: bytes */198384

They are responding with a Content-Range field for a redirect which really makes no sense. That's another issue I should be fixing, but it's really unrelated to that one and a bit less urgent.

[Julian Andres Klode (juliank)]

And note that the redirect from vorboss is to a site that indicates a failure, not the file.

Or in short: Sourceforge sucks, don't use it.

[Adrien Beau (adrienbeau)]

I have tested the new package on Xenial, following the instructions in https://wiki.ubuntu.com/Testing/EnableProposed

APT packages installed before updating:

apt 1.2.18
apt-transport-https 1.2.18
apt-utils 1.2.18
libapt-inst2.0 1.2.18
libapt-pkg5.0 1.2.18

Bug cases reproduced before updating:

1) dpkg-reconfigure update-notifier-common (calls apt-helper)
2) apt-helper download-file downloads.sourceforge.net
3) apt-helper download-file people.debian.org/~jak

Bug case *not* reproduced before updating:

4) apt-helper download-file kxstudio.linuxaudio.org

In this case, it appears that GitHub fixed its redirect response so it does not include spaces anymore (I see a %20 in the reply now).

APT packages installed after updating:

apt 1.2.19
apt-transport-https 1.2.19
apt-utils 1.2.19
libapt-inst2.0 1.2.19
libapt-pkg5.0 1.2.19

Bug cases fixed by updating:

1) dpkg-reconfigure update-notifier-common (calls apt-helper)
2) apt-helper download-file downloads.sourceforge.net
3) apt-helper download-file people.debian.org/~jak

[Elvis Stansvik (elvstone)]

Julian: Ah, indeed. If I make sure to use https against the SF, and make sure the file doesn't exist, it works reliably. Sorry for the confusion!

So, LGTM from xenial.

Thanks a lot for working this out!

[Julian Andres Klode (juliank)]

OK. We now have comments of success from yakkety and xenial, and I have also checked both, so marking it verified-done.

I'll open up a new bug for the partial download issue.

[Julian Andres Klode (juliank)]

The Content-Range issue with partial downloads on SF is tracked in bug 1657567 now. That's less urgent though, so we may just want to roll that one out when I do the next "scheduled" bugfix update.

[ComputeroBz (computerobz)]

I have this problem still. What I found to work was to remove the failed install and install it using the deb. I wrote an article about it on my blog for future reference...
https://computerobz.wordpress.com/2016/12/15/ttf-mscorefonts-installer-fails-to-installupgrade/

Basically:

1.) Launch a terminal and remove the failed install:

sudo apt-get remove --purge ttf-mscorefonts-installer

2.) Get the package and install it using the following three commands:

cd ~/Downloads/

sudo wget http://mirrors.linux.iu.edu/linux/debian/pool/contrib/m/msttcorefonts/ttf-mscorefonts-installer_3.6_all.deb

sudo dpkg -i ttf-mscorefonts-installer_3.6_all.deb

Problem solved.

[luca (llucax)]

Xenial's package worked for me, although I got these warnings (I guess is an unrelated problem though):

/etc/cron.daily/update-notifier-common:
Get:1 http://downloads.sourceforge.net/corefonts/andale32.exe [198 kB]
Fetched 198 kB in 2s (69,3 kB/s)
W: Can't drop privileges for downloading as file '/var/lib/update-notifier/package-data-downloads/partial/andale32.exe' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
Get:1 http://downloads.sourceforge.net/corefonts/arial32.exe [554 kB]
Fetched 554 kB in 3s (155 kB/s)
W: Can't drop privileges for downloading as file '/var/lib/update-notifier/package-data-downloads/partial/arial32.exe' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
Get:1 http://downloads.sourceforge.net/corefonts/arialb32.exe [168 kB]
Fetched 168 kB in 2s (62,1 kB/s)
W: Can't drop privileges for downloading as file '/var/lib/update-notifier/package-data-downloads/partial/arialb32.exe' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
Get:1 http://downloads.sourceforge.net/corefonts/comic32.exe [246 kB]
Fetched 246 kB in 2s (87,9 kB/s)
W: Can't drop privileges for downloading as file '/var/lib/update-notifier/package-data-downloads/partial/comic32.exe' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
Get:1 http://downloads.sourceforge.net/corefonts/courie32.exe [646 kB]
Fetched 646 kB in 3s (165 kB/s)
W: Can't drop privileges for downloading as file '/var/lib/update-notifier/package-data-downloads/partial/courie32.exe' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
Get:1 http://downloads.sourceforge.net/corefonts/georgi32.exe [392 kB]
Fetched 392 kB in 3s (119 kB/s)
W: Can't drop privileges for downloading as file '/var/lib/update-notifier/package-data-downloads/partial/georgi32.exe' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
Get:1 http://downloads.sourceforge.net/corefonts/impact32.exe [173 kB]
Fetched 173 kB in 2s (62,5 kB/s)
W: Can't drop privileges for downloading as file '/var/lib/update-notifier/package-data-downloads/partial/impact32.exe' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
Get:1 http://downloads.sourceforge.net/corefonts/times32.exe [662 kB]
Fetched 662 kB in 3s (177 kB/s)
W: Can't drop privileges for downloading as file '/var/lib/update-notifier/package-data-downloads/partial/times32.exe' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
Get:1 http://downloads.sourceforge.net/corefonts/trebuc32.exe [357 kB]
Fetched 357 kB in 3s (115 kB/s)
W: Can't drop privileges for downloading as file '/var/lib/update-notifier/package-data-downloads/partial/trebuc32.exe' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
Get:1 http://downloads.sourceforge.net/corefonts/verdan32.exe [352 kB]
Fetched 352 kB in 3s (113 kB/s)
W: Can't drop privileges for downloading as file '/var/lib/update-notifier/package-data-downloads/partial/verdan32.exe' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
Get:1 http://downloads.sourceforge.ne...

[Kevin O'Gorman (kogorman-pacbell)]

I got those error reports too. It happens when you run wget as root, and
use that directory. I changed it to use /var/tmp and the message went
away. You could also run wget without the sudo (some of the instructions I
saw have done that).

On Sun, Jan 22, 2017 at 1:13 PM, luca <email address hidden> wrote:

> Xenial's package worked for me, although I got these warnings (I guess
> is an unrelated problem though):
>
> /etc/cron.daily/update-notifier-common:
> Get:1 http://downloads.sourceforge.net/corefonts/andale32.exe [198 kB]
> Fetched 198 kB in 2s (69,3 kB/s)
> W: Can't drop privileges for downloading as file '/var/lib/update-notifier/
> package-data-downloads/partial/andale32.exe' couldn't be accessed by user
> '_apt'. - pkgAcquire::Run (13: Permission denied)
> Get:1 http://downloads.sourceforge.net/corefonts/arial32.exe [554 kB]
> Fetched 554 kB in 3s (155 kB/s)
> W: Can't drop privileges for downloading as file '/var/lib/update-notifier/
> package-data-downloads/partial/arial32.exe' couldn't be accessed by user
> '_apt'. - pkgAcquire::Run (13: Permission denied)
> Get:1 http://downloads.sourceforge.net/corefonts/arialb32.exe [168 kB]
> Fetched 168 kB in 2s (62,1 kB/s)
> W: Can't drop privileges for downloading as file '/var/lib/update-notifier/
> package-data-downloads/partial/arialb32.exe' couldn't be accessed by user
> '_apt'. - pkgAcquire::Run (13: Permission denied)
> Get:1 http://downloads.sourceforge.net/corefonts/comic32.exe [246 kB]
> Fetched 246 kB in 2s (87,9 kB/s)
> W: Can't drop privileges for downloading as file '/var/lib/update-notifier/
> package-data-downloads/partial/comic32.exe' couldn't be accessed by user
> '_apt'. - pkgAcquire::Run (13: Permission denied)
> Get:1 http://downloads.sourceforge.net/corefonts/courie32.exe [646 kB]
> Fetched 646 kB in 3s (165 kB/s)
> W: Can't drop privileges for downloading as file '/var/lib/update-notifier/
> package-data-downloads/partial/courie32.exe' couldn't be accessed by user
> '_apt'. - pkgAcquire::Run (13: Permission denied)
> Get:1 http://downloads.sourceforge.net/corefonts/georgi32.exe [392 kB]
> Fetched 392 kB in 3s (119 kB/s)
> W: Can't drop privileges for downloading as file '/var/lib/update-notifier/
> package-data-downloads/partial/georgi32.exe' couldn't be accessed by user
> '_apt'. - pkgAcquire::Run (13: Permission denied)
> Get:1 http://downloads.sourceforge.net/corefonts/impact32.exe [173 kB]
> Fetched 173 kB in 2s (62,5 kB/s)
> W: Can't drop privileges for downloading as file '/var/lib/update-notifier/
> package-data-downloads/partial/impact32.exe' couldn't be accessed by user
> '_apt'. - pkgAcquire::Run (13: Permission denied)
> Get:1 http://downloads.sourceforge.net/corefonts/times32.exe [662 kB]
> Fetched 662 kB in 3s (177 kB/s)
> W: Can't drop privileges for downloading as file '/var/lib/update-notifier/
> package-data-downloads/partial/times32.exe' couldn't be accessed by user
> '_apt'. - pkgAcquire::Run (13: Permission denied)
> Get:1 http://downloads.sourceforge.net/corefonts/trebuc32.exe [357 kB]
> Fetched 357 kB in 3s (115 kB/s)
> W: Can't drop privileges for downloading as file '/var/lib/update-notifier/
> package-data-downloads/partial/trebuc32.exe' couldn...

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for apt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.3.4

---------------
apt (1.3.4) yakkety; urgency=medium

  * https: Quote path in URL before passing it to curl (LP: #1651923)

 -- Julian Andres Klode <email address hidden> Tue, 17 Jan 2017 15:46:33 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.2.19

---------------
apt (1.2.19) xenial; urgency=medium

  * https: Quote path in URL before passing it to curl (LP: #1651923)

 -- Julian Andres Klode <email address hidden> Tue, 17 Jan 2017 15:48:51 +0100

[Nico Orrù (nigu-orru)]

Many thanks!

Cheers,
Nico

[Martin Burger (q8q6cw5f-8-ks2gs09p)]

Before version 1.2.19, I was able to automatically upgrade package "gitlab-ce" using unattended-upgrades. The update to version 1.2.19 seems to render my configuration useless.

From file `apt/apt.conf.d/50unattended-upgrades`:

    Unattended-Upgrade::Origins-Pattern {
      "origin=https://packages.gitlab.com/gitlab/gitlab-ce,archive=${distro_codename}";
    };

With version 1.2.19, `unattended-upgrade --debug` outputs:

[...]
    Allowed origins are: ['o=Ubuntu,a=xenial', 'o=Ubuntu,a=xenial-security', 'o=Ubuntu,a=xenial-updates', 'origin=https://packages.gitlab.com/gitlab/gitlab-ce,archive=xenial']
    Checking: gitlab-ce ([<Origin component:'main' archive:'xenial' origin:'packages.gitlab.com/gitlab/gitlab-ce' label:'gitlab-ce' site:'packages.gitlab.com' isTrusted:True>])
[...]

Thus, 'https://' seems to be missing now from checked property 'origin'.

While it is quite easy to fix this, version 1.2.19 might inadvertently break the configuration of many people.

[Adrien Beau (adrienbeau)]

Martin, I think you should file a new issue for what you described.

[Julian Andres Klode (juliank)]

Yes, please file a new bug. And that really seems more like an unattended-upgrades bug, I can't believe it's a regression in 1.2.19 - the change in 1.2.19 is just that:

+ Uri.Path = QuoteString(Uri.Path, "+~ ");

- just quoting the path component of the Uri before downloading it (in the https method process, not in the main apt process or when parsing an URI either), not doing any other change.

[aaronfranke (arnfranke)]

I'm still having this issue. Has the fix been pushed out yet?

I'm using Xubuntu 16.04 LTS 64-bit.

[John (webdevelopmentusa)]

I still have this issue. I upgraded from 16.04.4 LTS yesterday and now have 18.04LTS and still have this issue.

I'd rather get it fixed than resulting to cheap tricks.

Is anyone going to work on this?

[John (webdevelopmentusa)]

Additionally I ran this workaround but when running apt-get upgrade it still tries to upgrade this ttf install and fails the same way

workaround:

wget http://ftp.de.debian.org/debian/pool/contrib/m/msttcorefonts/ttf-mscorefonts-installer_3.6_all.deb

sudo dpkg -i ttf-mscorefonts-installer_3.6_all.deb

still get the failed download from sourceforge problem that everyone around the internet also reports getting, please fix this

[Dominique Meeùs (dominiquem)]

I still had the mscorefonts problem (errors in sourceforge mirrors) in 19.04. Found this workaround in #1655431. Quoting: Boris Rybalkin (ribalkin) wrote on 2017-11-26: #18
Was able to fix sf mirrors issue by replacing sf mirror with some github copy:
sudo sed -i 's#http://downloads.sourceforge.net/corefonts#https://github.com/pushcx/corefonts/raw/master#g' /usr/share/package-data-downloads/ttf-mscorefonts-installer
Then run:
sudo /usr/lib/update-notifier/package-data-downloader
All downloaded and good.

[Danny Sauer (ubuntu-dannysauer)]

This fix breaks CloudFront URLs, because the reencoding of URLs results in some HTML entities being replaced by their plain characters. CloudFront signing requires that the the whole URL matches the signature, as compared to S3 URLs which prune the querystring before validating the signature. That explains why this fix worked ok on S3.

I'll open a new bug since this is so old that reopening doesn't make sense, but also making a note here for future web search purposes.