Incompatible with full URL HTTP requests from recent apt versions

[James Dupin]

Binary package hint: apt-proxy

Analysis:
Ubuntu's apt now sends the full URL in the HTPP request instead of just the path. Apt-proxy isn't able to handle this. (does not appear to affect local copies of apt-proxy though)

Original report:
install apt-proxy 1.9.33, configure it, restart it.
change /etc/apt/sources.list accordingly.

apt-get update waits (forever) while there are some inputs in the log.

repeat anytime.

Also done apt-get update from dapper to apt-proxy 1.9.31 on breezy same result in the breezy apt-proxy logs (while apt-get update from a breezy machine to breezy apt-proxy 1.9.31 works fine)

[James Dupin (james.dupin)]

my conf

;; Server IP to listen on
address = 192.168.1.20

;; Server port to listen on
port = 9999

sources.list

deb http://192.168.1.20:9999/ubuntu/ dapper main restricted

[Robert Sander (gurubert)]

This happens because Ubuntu's apt now sends the full URL in the HTPP request instead of just the path:

Ubuntu Dapper Drake:

GET http://debian-hm:9999/ubuntu/dists/dapper/Release.gpg HTTP/1.1
Host: debian-hm:9999
Cache-Control: max-age=0
User-Agent: Debian APT-HTTP/1.3

Debian sarge:

GET /debian/dists/sarge/main/binary-i386/Packages.gz HTTP/1.1

Host: debian-hm:9999

Connection: keep-alive

If-Modified-Since: Tue, 18 Apr 2006 11:49:24 GMT

User-Agent: Debian APT-HTTP/1.3

apt-proxy 1.9.33 is not able to decode the path when the full URL is included.

[Tim Frost (timfrost)]

Because I have multiple machines, and was testing dapper under vmware, I installed apt-proxy under breezy. After upgrading to dapper, I get errors similar to
Failed to fetch http://marvin:9999/ubuntu/dists/dapper-updates/universe/binary-i386/Packages.gz 503 Service Unavailable

(marvin is the name of the machine that apt-proxy is running on.

apt-proxy.log shows:
2006/06/03 21:12 NZST [Channel,216,192.168.13.2] Starting factory <apt_proxy.apt_proxy.ClientFactory instance at 0xb6d1b12c>
2006/06/03 21:12 NZST [-] "[error] [ubuntu] Connection Failed: /dists/dapper/Release.gpg (DNS lookup failed: address 'ubuntu' not found: (-2, 'Name or service not known').)"
2006/06/03 21:12 NZST [-] Stopping factory <apt_proxy.apt_proxy.ClientFactory instance at 0xb6d1b12c>

/etc/apt-proxy/apt-proxy-v2.conf has the following entries for ubuntu:
[ubuntu]
;; Ubuntu archive
backends = http://archive.ubuntu.com/ubuntu

[ubuntu-security]
;; Ubuntu security updates
backends = http://security.ubuntu.com/ubuntu

The error indicates that the backend parsing is wrong - the path is being picked up as the host name

[Mark Howard (mh-tildemh)]

Confirmed here too.
I have an apt-proxy on 192.168.2.2:9999
Apt-updates work fine from the local machine, but fail when attempting to connect from a local machine. It looks like Robert's comment is correct - apt-proxy is failing to parse the url correctly.
Compare the following log segments:
Failing log (connecting from a remote machine).
Notice that a dynamic backend is created for ttp: (nonsense)
2006/06/03 12:16 BST [Channel,0,192.168.2.11] [debug] Headers: Host: 192.168.2.2:9999, Cache-Control: max-age=0, User-Agent: Debian APT-HTTP/1.3
2006/06/03 12:16 BST [Channel,0,192.168.2.11] [debug] Request: GET http://192.168.2.2:9999/ubuntu/dists/dapper/Release.gpg
2006/06/03 12:16 BST [Channel,0,192.168.2.11] [simplify_path] simplified path from http://192.168.2.2:9999/ubuntu/dists/dapper/Release.gpg to http:/192.168.2.2:9999/ubuntu/dists/dapper/Release.gpg
2006/06/03 12:16 BST [Channel,0,192.168.2.11] [debug] Request: GET http:/192.168.2.2:9999/ubuntu/dists/dapper/Release.gpg backend=ttp: local_file=/var/cache/apt-proxyhttp:/192.168.2.2:9999/ubuntu/dists/dapper/Release.gpg
2006/06/03 12:16 BST [Channel,0,192.168.2.11] [debug] Adding ttp: backend dynamicaly
2006/06/03 12:16 BST [Channel,0,192.168.2.11] [debug] Created new BackendServer: http://ttp:
2006/06/03 12:16 BST [Channel,0,192.168.2.11] [debug] backend: ttp: [<apt_proxy.apt_proxy.BackendServer instance at 0xb765c60c>]
2006/06/03 12:16 BST [Channel,0,192.168.2.11] [Fetcher.activate] (ttp:) servers:1http:/192.168.2.2:9999/ubuntu/dists/dapper/Release.gpg
2006/06/03 12:16 BST [Channel,0,192.168.2.11] [file_ok] check_cached: /var/cache/apt-proxyhttp:/192.168.2.2:9999/ubuntu/dists/dapper/Release.gpg
2006/06/03 12:16 BST [Channel,0,192.168.2.11] [debug] cached: 0
2006/06/03 12:16 BST [Channel,0,192.168.2.11] [fetch_real] Consulting server about /var/cache/apt-proxyhttp:/192.168.2.2:9999/ubuntu/dists/dapper/Release.gpg
2006/06/03 12:16 BST [Channel,0,192.168.2.11] [Fetcher.activate] (ttp:) servers:1http:/192.168.2.2:9999/ubuntu/dists/dapper/Release.gpg
2006/06/03 12:16 BST [Channel,0,192.168.2.11] Unhandled error in Deferred:
2006/06/03 12:16 BST [Channel,0,192.168.2.11] Traceback (most recent call last):
          File "/usr/lib/python2.4/site-packages/twisted/web/http.py", line 557, in requestReceived
            self.process()
          File "/usr/lib/python2.4/site-packages/apt_proxy/apt_proxy.py", line 1400, in process
            self.fetch()
          File "/usr/lib/python2.4/site-packages/apt_proxy/apt_proxy.py", line 1482, in fetch
            (dummyFetcher, 0, running,), None)
          File "/usr/lib/python2.4/site-packages/twisted/internet/defer.py", line 182, in addCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/lib/python2.4/site-packages/twisted/internet/defer.py", line 307, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/usr/lib/python2.4/site-packages/apt_proxy/apt_proxy.py", line 1451, in fetch_real
            fetcher = dummyFetcher.apEndTransfer(fetcher_class)
          File "/usr/lib/python2.4/site-packages/apt_proxy/apt_proxy.py", line 463,...

[Mark Howard (mh-tildemh)]

This simple patch strips out the newly added protocol/host:port section.
Tested and works both when the proto/host:port is and is not present.

[Tim Frost (timfrost)]

I can confirm that Mark's patch does work, both from the VMware images and from the local machine

[Alwin Garside (yogarine)]

Just applied the patch, works fine so far. :)

This bug was really driving me crazy the past few days.

[barroca (barroca)]

Hello all, i had the same error listed in this bug and the patch worked fine to me too...

thanks.

[R. Pereira Braga (rpereira)]

Same error.
Works for me too.

Thanks Mark.

[Daniel T Chen (crimsun)]

This debdiff contains Mark Howard's patch stripping the protocol/host:port to work with Ubuntu's apt, which sends full URL HTTP requests.

[Matt Zimmerman (mdz)]

I've checked just now, and Dapper's apt sends GET <path>, not GET <url>, same as Debian's. I'd be quite surprised if anyone made an explicit change like that anyway.

Do you have apt configured to talk to a proxy via apt.conf perhaps?

[Robert Sander (gurubert)]

apt is not configured to use a specific web-proxy but there is a transparent squid running in between the client and the apt-proxy server.

[Mark Howard (mh-tildemh)]

Matt,
How did you perform the check? Adding a simple print statement to the part of the code which is touched in the patch shows the full url here (no other proxies or apt.conf settings).
Note that this is only for cases where apt and apt-proxy are running on different machines. Local instances do seem to have just the path (and hence work unmodified).

[Matt Zimmerman (mdz)]

On Fri, Jul 14, 2006 at 11:03:56AM -0000, Robert Sander wrote:
> apt is not configured to use a specific web-proxy but there is a
> transparent squid running in between the client and the apt-proxy
> server.

In that case, the request is coming from squid, and not from apt at all.
However, I use squid as well (non-transparent and without apt-proxy), and I
still can't reproduce the behaviour described here. It still does a
standard GET with a path and not a URL. Which version of squid are you
using?

--
 - mdz

[Matt Zimmerman (mdz)]

On Fri, Jul 14, 2006 at 11:16:06AM -0000, Mark Howard wrote:
> How did you perform the check? Adding a simple print statement to the part
> of the code which is touched in the patch shows the full url here (no
> other proxies or apt.conf settings). Note that this is only for cases
> where apt and apt-proxy are running on different machines. Local
> instances do seem to have just the path (and hence work unmodified).

I used a packet sniffer to examine the HTTP session. As I mentioned, I
don't use apt-proxy; if this behaviour only appears when talking to
apt-proxy, it's probably an issue with that, rather than with apt.

At any rate, I'm happy for MOTU to make the call on this patch, since
apt-proxy is in universe.

--
 - mdz

[cvrse (cvrse)]

i had the same problem and all it took to fix was changing a line in /etc/apt/apt.conf back to the breezy default. ie.
Acquire::http::Proxy "false";
to
Acquire::::Proxy "false";

[SyBorg (stefano-bartaletti)]

Hi,

I tried the hint by cvrse first, it worked with an old Breezy installation (to make upgrade to dapper) but did not work on a fresh Dapper installation.

Had to apply the forementioned patch to apt_proxy.py to have it work correctly on Dapper too

[Matt Zimmerman (mdz)]

I'm a bit mystified by this problem; I'd like to hear an opinion from upstream (the bug has been forwarded to bugs.debian.org) before changing it in Dapper.

[Gary (gary-smithe)]

I noticed the problem as well, and changed my apt.conf to reflect the changes posed by cvrse.

This was on a fresh install of Dapper (server cd). Apt-proxy wasn't working even for the local machine.

Once I changed the apt.conf, all was good.

[Matt Zimmerman (mdz)]

The problem seems in fact to be with apt; it should be fixed there rather than hacked around in apt-proxy.

[Michael Vogt (mvo)]

Could you please check via "apt-config dump" if your http proxy is set to "false" ?

[James Dupin (james.dupin)]

Michael Vogt wrote:
> Could you please check via "apt-config dump" if your http proxy is set
> to "false" ?
>
>
it is. see attached.

APT "";
APT::Architecture "i386";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Periodic "";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
APT::Archives "";
APT::Archives::MaxAge "30";
APT::Archives::MinAge "2";
APT::Archives::MaxSize "500";
APT::Authentication "";
APT::Authentication::TrustCDROM "true";
Dir "/";
Dir::State "var/lib/apt/";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::userstatus "status.user";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt/";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt/";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::vendorlist "vendors.list";
Dir::Etc::vendorparts "vendors.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::dpkg "/usr/bin/dpkg";
aptitude "";
aptitude::Keep-Unused-Pattern "^linux-image.*$ | ^linux-restricted-modules.*$";
Unattended-Upgrade "";
Unattended-Upgrade::Allowed-Origins "";
Unattended-Upgrade::Allowed-Origins:: "Ubuntu dapper-security";
DPkg "";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Post-Invoke "";
DPkg::Post-Invoke:: "if [ -d /var/lib/update-notifier ]; then touch /var/lib/update-notifier/dpkg-run-stamp; fi";
Acquire "";
Acquire::http "";
Acquire::http::Proxy "false";

[Michael Vogt (mvo)]

IMHO it is apt-setup that is setting Acquire::http::Proxy to a incorrect value. If no proxy is used apt-setup should just not write this entry.

[Michael Vogt (mvo)]

I commited a fix that adds more careful checking for valid proxy settings to methods/http.cc. This should fix this problem in apt. I also re-assigned the debian bug to apt-setup because it writes out incorrect values.

[William Grant (wgrant)]

The fixed apt-proxy has been in Ubuntu for a while now.

[darren (darrenm)]

If anyone comes to this bug report trying to find out why it is still broken, check for your $http_proxy, $https_proxy and $ftp_proxy being set on the machine you are trying to do updates from, these need to be unset.