Ubuntu archive server returning incorrect content-encoding - content-header incorrect type

[Bob Drzyzgula]

This appears to be related to bug 215694, but I've identified Ubuntu's web server as the culprit.

https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/215694

When trying to do an upgrade from 7.10 to 8.04, do-release-upgrade attempts to download the file hardy.tar.gz.gpg. On a machine directly connected to the Internet, this appears to work. However, behind a proxy server, this fails because archive.ubuntu.com is miscoding the content-encoding of the gpg file as x-gzip.

The error message from do-release-upgrade (run from behind a proxy server) looks like this:

-----------------------
# do-release-upgrade
Checking for a new ubuntu release
Failed Upgrade tool signature
Done Upgrade tool
Done downloading
extracting '/tmp/tmpArwlWD/hardy.tar.gz'
authenticate '/tmp/tmpArwlWD/hardy.tar.gz' against '/tmp/tmpArwlWD/hardy.tar.gz.gpg'
exception from gpg: GnuPG exited non-zero, with code 131072
Debug information:

gpg: WARNING: unsafe permissions on homedir `/tmp/tmpArwlWD'

gpg: can't open `/tmp/tmpArwlWD/hardy.tar.gz.gpg'
gpg: verify signatures failed: file open error

Authentication failed
Authenticating the upgrade failed. There may be a problem with the network or with the server.
-----------------------

Using wireshark to trace this conversation, I find that the utility is attempting to download two files, one of which is:
http://archive.ubuntu.com/ubuntu/dists/hardy-proposed/main/dist-upgrader-all/0.87.27/hardy.tar.gz.gpg

Doing a wget of this file (again via a proxy server) also fails:

-----------------------
# wget --no-cache http://archive.ubuntu.com/ubuntu/dists/hardy-proposed/main/dist-upgrader-all/0.87.27/hardy.tar.gz.gpg
--11:04:13-- http://archive.ubuntu.com/ubuntu/dists/hardy-proposed/main/dist-upgrader-all/0.87.27/hardy.tar.gz.gpg
           => `hardy.tar.gz.gpg'
Resolving <proxy server>
Connecting to <proxy server>:8080... connected.
Proxy request sent, awaiting response... 502 Bad Gateway
11:04:13 ERROR 502: Bad Gateway.
-----------------------

However, if I run this wget from an external machine with a direct connection, it works correctly:

-----------------------
$ wget --no-cache http://archive.ubuntu.com/ubuntu/dists/hardy-proposed/main/dist-upgrader-all/0.87.27/hardy.tar.gz.gpg
--11:09:42-- http://archive.ubuntu.com/ubuntu/dists/hardy-proposed/main/dist-upgrader-all/0.87.27/hardy.tar.gz.gpg
           => `hardy.tar.gz.gpg'
Resolving archive.ubuntu.com... 91.189.88.45, 91.189.88.46, 91.189.88.31
Connecting to archive.ubuntu.com|91.189.88.45|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 191 [application/x-tar]

100%[================================================================================================================>] 191 --.--K/s

11:09:42 (5.30 MB/s) - `hardy.tar.gz.gpg' saved [191/191]
-----------------------

Using tcpdump to capture this conversation from the external system, and wireshark's "Follow TCP Stream" utility to interpret it, one sees that, in the successful direct connection, the server archive.ubuntu.com is identifying the download as having content-encoding of x-gzip, while the packet itself contains a text gpg signature:

-----------------------
GET /ubuntu/dists/hardy-proposed/main/dist-upgrader-all/0.87.27/hardy.tar.gz.gpg HTTP/1.0
Pragma: no-cache
User-Agent: Wget/1.10.2
Accept: */*
Host: archive.ubuntu.com
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Thu, 03 Jul 2008 14:29:48 GMT
Server: Apache/2.0.55 (Ubuntu)
Last-Modified: Fri, 09 May 2008 16:46:50 GMT
ETag: "4074034-bf-44ccef1c47280"
Accept-Ranges: bytes
Content-Length: 191
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: application/x-tar
Content-Encoding: x-gzip

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQBIJH/6QJdur0N9BbURAgyxAJ4mn9rGKONYm5J4OiKvDvhGB8ypLQCgrSyh
0ZUEkk2K32jQ47tO32hN2Uo=
=jJYy
-----END PGP SIGNATURE-----
-----------------------

Looking at the detailed response from the proxy server to the internal system confirms that this is the problem; the proxy server reports:

-----------------------
Server response could not be decoded using encoding type returned by server. This is typically caused by a Web Site presenting a content encoding header of one type, and then encoding the data differently.
-----------------------

[Antti Kaihola (akaihola)]

I am seeing the same symptom trying to upgrade from Gutsy to Hardy using apt-cacher. Is there a work-around or do I have to connect directly (re-downloading all packages)?

[PooyaPlus (pooya-zarei)]

Hi, I have a similar problem and reported it to ubuntuforums installation and upgrade section but still could not find a workaround.

I am trying to upgarde gutsy to hardy behind a proxy server that requires authentication username and password. I can update the packages using the global proxy setting of gnome and even can wget in the terminal with that setting. Even the synaptic is working fine with its own proxy configuration. But when attempted the upgarde process it says Failed to fetch some packages -- proxy authorization problem.

[Charles Curley (charlescurley)]

I see the symptoms while trying to upgrade from 8.04 to 8.10 beta.

root@white:~# update-manager --devel-release &
[1] 6790
root@white:~# extracting 'intrepid.tar.gz'
authenticate 'intrepid.tar.gz' against 'intrepid.tar.gz.gpg'
exception from gpg: GnuPG exited non-zero, with code 131072
Debug information:

gpg: WARNING: unsafe permissions on homedir `/tmp/tmp83J8IK'

gpg: can't open `/tmp/tmp83J8IK/intrepid.tar.gz.gpg'
gpg: verify signatures failed: file open error

[1]+ Done update-manager --devel-release
root@white:~#

On a successful run, I do not get the permissions message.

update-manager-1:0.87.30--all
update-manager-core-1:0.87.30--i386

I removed a local apt-cacher-ng proxy from my configuration, and got past the problem.

Acquire::http { Proxy "http://192.168.1.30:3142"; };

I have not verified the wireshark analysis.

[don hardaway (don-hardaway)]

How do you remove local apt-cacher-ng proxy from my configuration?

[xteejx (xteejx-deactivatedaccount)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. You reported this bug a while ago and there hasn't been any activity in it recently. We were wondering is this still an issue for you? Can you try with latest Ubuntu release? Thanks in advance.

[M. Vefa Bicakci (mvb)]

Dear Teej,

This problem still exists because the problem is with the configuration of Ubuntu's
servers (such as "archive.ubuntu.com"), not the distribution it produces. To summarize,
Ubuntu's archive servers return wrong headers for gpg files, and this causes errors
when there is a proxy between the computer running Ubuntu's upgrade program and
Ubuntu's archive servers.

I encountered the same problem when I tried to upgrade from 6.06 to 8.04 a few
days ago. When I disabled the proxy configuration for apt by modifying apt's
configuration, the upgrade worked. This is expected, as can be understood from
the description of the bug.

I would appreciate it if the configuration of Ubuntu's archive servers could be fixed.
Please read the description of this bug to learn about the problem in the configuration
of Ubuntu's archive servers. (I did not actually let the upgrade begin, so I can still test
the server if the configuration of the server is eventually changed.)

Regards,

M. Vefa Bicakci

[xteejx (xteejx-deactivatedaccount)]

Sorry I never got back to you on this. I don't suppose this is still a problem in Karmic 9.10? Thank you.

[Antti Kaihola (akaihola)]

Teej, as M. Vefa pointed out, the problem is not in the distribution but on the Ubuntu HTTP servers.

The Content-Encoding header mentioned by Bob Drzyzgula in the original report seems to be gone now.

However, Content-Type is now application/x-gzip which is incorrect for a PGP signature. I haven't tested upgrading through a proxy recently, but it's still possible that this header could break that.

It looks like mod_mime matches extensions in the middle of a file name if the final extension is unknown. This explains why for files with a .tar.gz.gpg extension, the Ubuntu servers used to indicate a gzip encoding and a tar content type, and why they now return a gzip content type.

Here is what the Ubuntu servers currently reply:

$ wget -q --no-cache --save-headers http://archive.ubuntu.com/ubuntu/dists/hardy-proposed/main/dist-upgrader-all/0.87.31/hardy.tar.gz.gpg
$ cat hardy.tar.gz.gpg
HTTP/1.1 200 OK
Date: Sun, 17 Jan 2010 13:00:31 GMT
Server: Apache/2.2.8 (Ubuntu)
Last-Modified: Thu, 29 Jan 2009 15:38:41 GMT
ETag: "bd-461a0e1fda240"
Accept-Ranges: bytes
Content-Length: 189
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: application/x-gzip

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBJgc2BQJdur0N9BbURAnhDAJwNh53DLg9jF8c3+dcjTKM+CnpK6gCggG9e
2Fi2CguhyGmEktTiVEQT4Hk=
=5Ug/
-----END PGP SIGNATURE-----

$ wget -q --no-cache --save-headers http://archive.ubuntu.com/ubuntu/dists/lucid/main/dist-upgrader-all/0.131.3/lucid.tar.gz.gpg
akaihola@morris:/tmp$ cat lucid.tar.gz.gpg
HTTP/1.1 200 OK
Date: Sun, 17 Jan 2010 13:04:35 GMT
Server: Apache/2.2.8 (Ubuntu)
Last-Modified: Tue, 12 Jan 2010 19:27:03 GMT
ETag: "bd-47cfca3780fc0"
Accept-Ranges: bytes
Content-Length: 189
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: application/x-gzip

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBLTM0HQJdur0N9BbURAgoQAJ9jwi/0EBi+Hd5VWivJ90SYhRFwsQCggxR/
Y2Z4yunLdiqmR9x2DrOUxM4=
=tv7O
-----END PGP SIGNATURE-----

[Antti Kaihola (akaihola)]

I think this should be added to Apache's /etc/apache2/mods-available/mime.conf:
AddType application/pgp-signature .gpg

I tried this on my Debian Squeeze -based server and sure enough, .tar.gz.gpg files are now served with the application/pgp-signature content type. This needs to be tested though by creating an Ubuntu repository mirror and upgrading from it through a proxy.

On the other hand, maybe a wrong Content-Type doesn't matter and the removal of the Content-Encoding header already fixed the issue.

[xteejx (xteejx-deactivatedaccount)]

Thanks for updating this. Marking this Triaged, Medium, as there is more than enough information for someone to see what is going on here.

[Antti Kaihola (akaihola)]

I also notified Debian's apache2 bug tracker about the issue:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=565626

[xteejx (xteejx-deactivatedaccount)]

Brilliant! Thank you. I will add a watch for this. For future reference you can do this easily by choosing "Also affects project" and pasting the link to the other Bug Tracking System. :)

[xteejx (xteejx-deactivatedaccount)]

"Also affect distribution", sorry.

[xteejx (xteejx-deactivatedaccount)]

No, I was right the first time, don't worry, it's set now, and Launchpad will keep track of it. Remember if this affects you to add yourself as affected, currently there is only 1 affected user, and if you can provide any more relevant information to this bug report, or the Debian bug, please add it, perhaps on both to make it easier for developers. Thank you.

[Colin Watson (cjwatson)]

Unassigning ubuntu-archive - we manage the contents of the archive but in general have no control over the configuration of the servers that host it, so there isn't much point in us being assigned to this.

[Garcez Diomedes da Silva (garcezcont)]

OK

[Andreas Hasenack (ahasenack)]

Very old bug. I just tried wgetting http://archive.ubuntu.com/ubuntu/dists/disco-proposed/main/dist-upgrader-all/current/disco.tar.gz.gpg with and without a proxy, and it worked just fine, even though it's still being identified as x-gzip:
andreas@nsnx:~/x$ http_proxy=http://squid-ds216.lxd:3128/ wget --no-cache http://archive.ubuntu.com/ubuntu/dists/disco-proposed/main/dist-upgrader-all/current/disco.tar.gz.gpg
--2019-02-06 14:57:16-- http://archive.ubuntu.com/ubuntu/dists/disco-proposed/main/dist-upgrader-all/current/disco.tar.gz.gpg
Resolving squid-ds216.lxd (squid-ds216.lxd)... 10.0.100.206
Connecting to squid-ds216.lxd (squid-ds216.lxd)|10.0.100.206|:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1554 (1,5K) [application/x-gzip]
Saving to: ‘disco.tar.gz.gpg’

disco.tar.gz.gpg 100%[================================================================================================================================>] 1,52K --.-KB/s in 0s

2019-02-06 14:57:17 (75,9 MB/s) - ‘disco.tar.gz.gpg’ saved [1554/1554]

andreas@nsnx:~/x$ file disco.tar.gz.gpg
disco.tar.gz.gpg: PGP signature Signature (old)

andreas@nsnx:~/x$ cat disco.tar.gz.gpg
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAABCgAGBQJcRf7TAAoJEDtP5qzAsh8yFaMQANW6mTi+bCHp08NeFC7wcSal
z2xtiNxm2FYIJ9BkZZGvS1YI5rQsS8X0x1auU21mCus39anOUp5XaWUDPVMfjuZO
RxajHohB1YUAiLMi7zY86LkpEMSYOMknwhH57ZZSMlHELyqSb/hKzawPfrBKTnWo
N0b0+3hbpwi7tRZu0Y3Y3cooeH6C6lgFJEAPkFXcgdFw72OtGs1ByOIkRlr4AUb7
...

I'm going to mark this bug as incomplete, and it will expire automatically unless someone has something to add.

[Bob Drzyzgula (bob-drzyzgula)]

I'm real sorry to waste bandwidth like this, but I just want to let y'all know that I'm LOLing at this point. :-)