Apt-cacher-ng fails on https repo

[wiredfool]

Following the instructions to add the docker repo here: https://docs.docker.com/engine/installation/linux/ubuntu/

/etc/apt/sources.list contains the repo:
deb https://apt.dockerproject.org/repo/ ubuntu-xenial main

When running through apt-cacher-ng, apt-get update returns:
$ sudo apt-get update
Ign:1 https://apt.dockerproject.org/repo ubuntu-xenial InRelease
Ign:2 https://apt.dockerproject.org/repo ubuntu-xenial Release
Ign:3 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages.diff/Index
Ign:4 https://apt.dockerproject.org/repo ubuntu-xenial/main i386 Packages
Ign:5 https://apt.dockerproject.org/repo ubuntu-xenial/main all Packages
Ign:6 https://apt.dockerproject.org/repo ubuntu-xenial/main Translation-en_US
Ign:7 https://apt.dockerproject.org/repo ubuntu-xenial/main Translation-en
Ign:8 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
Ign:4 https://apt.dockerproject.org/repo ubuntu-xenial/main i386 Packages
Ign:5 https://apt.dockerproject.org/repo ubuntu-xenial/main all Packages
Ign:6 https://apt.dockerproject.org/repo ubuntu-xenial/main Translation-en_US
Ign:7 https://apt.dockerproject.org/repo ubuntu-xenial/main Translation-en
Ign:8 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
Ign:4 https://apt.dockerproject.org/repo ubuntu-xenial/main i386 Packages
Ign:5 https://apt.dockerproject.org/repo ubuntu-xenial/main all Packages
Ign:6 https://apt.dockerproject.org/repo ubuntu-xenial/main Translation-en_US
Ign:7 https://apt.dockerproject.org/repo ubuntu-xenial/main Translation-en
Ign:8 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
Ign:4 https://apt.dockerproject.org/repo ubuntu-xenial/main i386 Packages
Ign:5 https://apt.dockerproject.org/repo ubuntu-xenial/main all Packages
Ign:6 https://apt.dockerproject.org/repo ubuntu-xenial/main Translation-en_US
Ign:7 https://apt.dockerproject.org/repo ubuntu-xenial/main Translation-en
Ign:8 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
Ign:4 https://apt.dockerproject.org/repo ubuntu-xenial/main i386 Packages
Ign:5 https://apt.dockerproject.org/repo ubuntu-xenial/main all Packages
Ign:6 https://apt.dockerproject.org/repo ubuntu-xenial/main Translation-en_US
Ign:7 https://apt.dockerproject.org/repo ubuntu-xenial/main Translation-en
Ign:8 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
Err:4 https://apt.dockerproject.org/repo ubuntu-xenial/main i386 Packages
  Received HTTP code 403 from proxy after CONNECT
Ign:5 https://apt.dockerproject.org/repo ubuntu-xenial/main all Packages
Ign:6 https://apt.dockerproject.org/repo ubuntu-xenial/main Translation-en_US
Ign:7 https://apt.dockerproject.org/repo ubuntu-xenial/main Translation-en
Ign:8 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
Ign:9 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:10 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:11 http://dl.google.com/linux/chrome/deb stable Release
Hit:13 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:14 http://archive.ubuntu.com/ubuntu xenial-security InRelease
Reading package lists... Done
W: The repository 'https://apt.dockerproject.org/repo ubuntu-xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://apt.dockerproject.org/repo/dists/ubuntu-xenial/main/binary-i386/Packages Received HTTP code 403 from proxy after CONNECT
E: Some index files failed to download. They have been ignored, or old ones used instead.

When apt is set to disable the proxy, using the following /etc/apt/apt.conf:
Acquire::http::Proxy "http://127.0.0.1:9999/";
Acquire::https::Proxy "DIRECT";

(note that the port has been changed from the default)

The response is:
$ sudo apt-get update
Hit:1 https://apt.dockerproject.org/repo ubuntu-xenial InRelease
Ign:2 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:3 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:4 http://dl.google.com/linux/chrome/deb stable Release
Hit:6 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:7 http://archive.ubuntu.com/ubuntu xenial-security InRelease
Reading package lists... Done

$lsb_release -rd
Description: Ubuntu 16.04.1 LTS
Release: 16.04
$ apt-cache policy apt-cacher-ng
apt-cacher-ng:
  Installed: 0.9.1-1ubuntu1
  Candidate: 0.9.1-1ubuntu1
  Version table:
 *** 0.9.1-1ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        100 /var/lib/dpkg/status

[Simon Déziel (sdeziel)]

@wiredfool, this is the expected behavior and I ran into it as well. Since apt-cacher-ng cannot cache the HTTPS connection so it's refusing to proxy it. A possible workaround is to let it pass through apt-cacher-ng using a regex like that in the config file:

# Required for HTTPS outbound
PassThroughPattern: ^apt\.dockerproject\.org:443$

FYI. if you need to pass through more than one HTTPS destination, use that syntax as there can only be one PassThroughPattern directive:

PassThroughPattern: ^(apt\.dockerproject\.org|registry\.npmjs\.org|packages\.graylog2\.org|graylog2-package-repository\.s3\.amazonaws\.com|artifacts\.elastic\.co):443$

[Eduard Bloch (edi-gmx)]

Please reopen this as a bug against APT. There is a detailed HTTP status message and not a mere 403 code. The application should display it.

[Rolf Leggewie (r0lf)]

Added a task against apt as requested.

Eduard, if I understan Simon correctly wouldn't it be possible to deal with this problem by changing the the defaults in acng for PassThroughPattern?

[Julian Andres Klode (juliank)]

It would be nice to have some more details, but I think it's a bit unrealistic to display a web page in apt's error handling system.

[Julian Andres Klode (juliank)]

Think about that: Your proxy returns 403 and you have 20 repositories, and now get 20 messages, each 100 lines long because your proxy decided to go fancy.

And of course: If reported as an actual error message, these things also become python exceptions.

Or did you really just mean the message after the status code (RFC calls it reason-phrase) and not the entire response body? That's usually reasonable short. That said, apt-cacher-ng gets very creative with these messages, other proxies stick to the standard messages. And the HTTP standard says a client should ignore the messages completely, as they are legacy (compare RFC 7230, 3.1.2).

So I'm not sure it's worthwhile to display the reason-phrase either: It's only used by apt-cacher-ng and the standard says we should ignore its contents.

[Julian Andres Klode (juliank)]

Marking this as Won't Fix now, as it's outside the scope for apt to display error messages from remote servers

[Rolf Leggewie (r0lf)]

Reopening against acng for further consideration and possible discussion.