update-manager could detect transparent proxy to avoid "Hash Sum mismatch" errors

[James Hunt]

If you are behind a transparent proxy (such as squid) or using a local caching proxy such as apt-cacher-ng, you can end up with errors resulting from the proxy caching an old version of particular packages, or not honouring certain HTTP headers.

Errors such as this can occur when attempting to upgrade your system:

  W: Failed to fetch http://gb.archive.ubuntu.com/...some-pkg.i386.deb Hash Sum mismatch

We could consider enhancing update-manager et al to detect this scenario and warn the user that the update/upgrade *may* be problematic as a result of the proxy. Techniques to use include:

- Attempting a "GET /" on port 80 of an IP address on which it is guaranteed there is no web server listening.
  We could then check the response to look for common proxy info.

- Creating a CGI script on an ubuntu.com server which is guaranteed to return a small file with different content every time.
  Assuming the ubuntu.com web server is configured correctly, if calling this CGI script returns the same value twice,
  the result must be being cached by a proxy.

- Attempt to retrieve the HTTP headers for any valid package file in the archive.
  If the response comes back and includes a "Via:" header, a proxy is in use.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: update-manager 1:0.152.25.5
ProcVersionSignature: Ubuntu 3.0.0-14.23-generic-pae 3.0.9
Uname: Linux 3.0.0-14-generic-pae i686
NonfreeKernelModules: nvidia
ApportVersion: 1.23-0ubuntu4
Architecture: i386
Date: Thu Jan 12 09:59:41 2012
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Release i386 (20101007)
PackageArchitecture: all
ProcEnviron:
 PATH=(custom, user)
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
SourcePackage: update-manager
UpgradeStatus: Upgraded to precise on 2012-01-12 (0 days ago)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ralf Heiringhoff (frosty-geek)]

We see the same behaviour with 10.04.3 Lucid "Clients" (with backports & proposed updates enabled) using a "regular" squid3 proxy server

-------------------cut--------------
root@backup:~# dpkg -l apt
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-===============================================-===============================================-==============================================================================================================
ii apt 0.7.25.3ubuntu9.9 Advanced front-end for dpkg

root@backup:~# cat /etc/apt/apt.conf.d/99proxy
Acquire::http::Proxy "http://dhcp.office.XXX.de:3128";

root@dhcp:~# dpkg -l squid3
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-===============================================-===============================================-==============================================================================================================
ii squid3 3.0.STABLE19-1ubuntu0.1 A full featured Web Proxy cache (HTTP proxy)

root@dhcp:~# egrep -v '^(#|$)' /etc/squid3/squid.conf
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
acl localnet src 192.168.219.0/24 # RFC1918 possible internal network
acl localnet src 192.168.220.0/23 # RFC1918 possible internal network
acl localnet src 192.168.230.0/24 # RFC1918 possible internal network
acl localnet src 192.168.232.0/23 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny to_localhost
http_access allow localnet
http_access allow localhost
http_access deny all
icp_access deny all
htcp_access deny all
http_port 3128
hierarchy_stoplist cgi-bin ?
cache_mem 200 MB
maximum_object_size_in_memory 10240 KB
cache_replacement_policy heap LFUDA
cache_dir ufs /var/spool/squid3 8192 16 256
maximum_object_size 512 MB
access_log /var/log/squid3/access.log squid
refresh_pattern -i .udeb$ 129600 100% 129600
refresh_pattern -i .deb$ 129600 100% 129600
refresh_pattern -i .rpm$ 129600 100% 129600
refresh_pattern -i .tgz$ 129600 1...

[suokunlong (suokunlong)]

The "Hash Sum mismatch" issue may be a disaster in some circumstances, because of this my ubuntu system was broken once.

Just think about that when apt-get updating,
'http://mirrors.163.com/ubuntu/dists/trusty/main/binary-i386/Packages.bz2' was downloaded without a hash error, while
http://mirrors.163.com/ubuntu/dists/trusty/multiverse/binary-i386/Packages.bz2 was downloaded with a mismatch error

then, if a package in the first link should be upgraded in order for another package in the 2nd link to work properly, then it's a disaster, just like the one I had a year ago.

ISP cache/proxy is very commen, because it makes the network within a certain area very fast, especially for the most visited contents.

[Sasa Paporovic (melchiaros)]

Adding the tags lucid and trusty, refering to user comments and Duplicates.