apt 403 Forbidden on some packages

[radle]

When I run "apt-get install ntp" it fails with

Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  ntp-doc
The following NEW packages will be installed:
  ntp
0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
Need to get 517kB of archives.
After this operation, 1,323kB of additional disk space will be used.
Err http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ lucid-updates/main ntp 1:4.2.4p8+dfsg-1ubuntu2.1
  403 Forbidden
Failed to fetch http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/n/ntp/ntp_4.2.4p8+dfsg-1ubuntu2.1_i386.deb 403 Forbidden
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

[radle (0-ra)]

Works good with us-west-1 though.

[Ben Howard (darkmuggle-deactivatedaccount)]

This is a know problem with S3.

Populate a new file named /etc/apt/apt.conf.d/90-fix-s3 with:
Acquire::http::Pipeline-Depth "0";

This problem with go away.

[Ben Howard (darkmuggle-deactivatedaccount)]

Confirmed:
Suggested packages:
  ntp-doc
The following NEW packages will be installed:
  ntp
0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
Need to get 559kB of archives.
After this operation, 1,450kB of additional disk space will be used.
Err http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ lucid-updates/main ntp 1:4.2.4p8+dfsg-1ubuntu2.1
  403 Forbidden
Failed to fetch http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/n/ntp/ntp_4.2.4p8+dfsg-1ubuntu2.1_amd64.deb 403 Forbidden

[Ben Howard (darkmuggle-deactivatedaccount)]

Sigh...looks like a problem with S3...

root@domU-12-31-39-05-49-DC:/etc/apt# curl https://s3-us-west-1.amazonaws.com/us-west-1.ec2.archive.ubuntu.com/ubuntu/pool/main/n/ntp/ntp_4.2.4p8+dfsg-1ubuntu2.1_amd64.deb -o /dev/null
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 546k 100 546k 0 0 414k 0 0:00:01 0:00:01 --:--:-- 561k
root@domU-12-31-39-05-49-DC:/etc/apt# curl http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/n/ntp/ntp_4.2.4p8+dfsg-1ubuntu2.1_amd64.deb -o /dev/null
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
115 231 0 231 0 0 834 0 --:--:-- --:--:-- --:--:-- 2179
root@domU-12-31-39-05-49-DC:/etc/apt# curl http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/n/ntp/ntp_4.2.4p8+dfsg-1ubuntu2.1_amd64.deb
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>D382C6B38E86A602</RequestId><HostId>p4SZkVYnZvkV95GRKXRTqWR6QQ0YSLFsG2w04sjGeTJL780SoPwhgjViUd7S+CQW</HostId></Error>root@domU-12-31-39-05-49-DC:/etc/apt#

[Ben Howard (darkmuggle-deactivatedaccount)]

So the problem that I am seeing is that if you access the bucket via:
http://s3-us-west-1.amazonaws.com/us-west-1.ec2.archive.ubuntu.com/.... it works

while
http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/... fails

The difference is that the one that is failing is caused by accessing the bucket the new way, while the first way is the old way.

[Eric Hammond (esh)]

You need to specify the correct S3 region (s3-us-west-1) when using the "new way":

  http://us-west-1.ec2.archive.ubuntu.com.s3-us-west-1.amazonaws.com/...

For example:

  http://us-west-1.ec2.archive.ubuntu.com.s3-us-west-1.amazonaws.com/ubuntu/pool/main/n/ntp/ntp_4.2.4p8+dfsg-1ubuntu2.1_amd64.deb

Note, however, that you cannot use "https" URLs when specifying the bucket name in the hostname as it will not match the SSL certificate hostname.

[Ben Howard (darkmuggle-deactivatedaccount)]

Pending Amazon.

[Ben Howard (darkmuggle-deactivatedaccount)]

Yeah, so I was completely wrong here. I was comparing us-west-1 to us-west-2.

However, Amazon reports that this is a problem with object ownership. They are working on resolving this.

[radle (0-ra)]

Yes, the bucket owner should make the file accessible by public.
Maybe it concerns not only ntp package, I just got it by accident.
So - bucket owner should make all the package files accessible by public.

[Ben Howard (darkmuggle-deactivatedaccount)]

The real problem is "+"'s that S3 deals with very, very poorly. You upload keys with one encoding and request them with another encoding. Which means you have to upload any file with "+"'s three times -- once with "+"'s, then replacing the "+" with " " and "%2B". If any of those three are missing, it blows up.

I've put a fix in the code that looks for those files and uploads them. Hopefully this fixes the problems.

[Shane Meyers (shanemeyers)]

I'm having trouble today with pulling a package from the s3 package repositories and this error doesn't appear to be transient. I'm consistently getting a "403 Forbidden" response for the following package. This one does have a "+" in the filename, so I think my issue may be related.

http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/libx/libxml-sax-perl/libxml-sax-perl_0.96+dfsg-2_all.deb

This prevented apt from completing a package install run, which stopped a number of instances from completing their boot process (we install a number of packages upon first boot with ec2).

Relevant line in my /etc/apt/sources.list file (to make sure I'm not incorrectly referencing the bucket; this has been working for many months until today):
deb http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ oneiric main

[alexius ludeman (lexinator)]

Just to add additional datapoint it's also happening in us-west-1.

Failed to fetch http://us-west-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/universe/n/nsca/nsca_2.7.2+nmu2_amd64.deb 403 Forbidden
Failed to fetch http://us-west-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/n/ntp/ntp_4.2.6.p2+dfsg-1ubuntu12_amd64.deb 403 Forbidden

[Scott Moser (smoser)]

Ben (utlemming) is aware of this and is working on getting it fixed.
Thank you for making sure we were aware, and your continued testing.

[Ben Howard (darkmuggle-deactivatedaccount)]

For the S3 mirrors, there were three distinct issues at play.

Background: S3 is deals with "+" in backwards ways. The short version is that if you have a "+" in the name, you have to upload it three times to deal with character encoding problems -- once with "+", and then once with "%2B" and " " replacing any "+".

1. Amazon is now enforcing exact URL fetching. It used to be that if you requested "foo%2Bbar" and there was a file of "foo+bar", as long as your request was using utf-8 encoding you would get the file "foo+bar". Now, if you request "foo%2Bbar", even though the request is interpetted as "foo+bar" in ascii, you must have "foo%2Bbar" in the bucket.

2. My code had a bug where it was not properly syncronizing and checking the different encodings of "+". The result was that some "+" files were not getting pushed up.

3. My code had a second bug which was caused by archive.ubuntu.com not returning content-type in the http headers. Previously I was using the content-type header from archive.ubuntu.com to set the content type header for S3. However, recently that changed, which was causing syncronization problems.

The issue was fixed late last on 2012-05-23. After looking at the logs, the problems seems to have been fixed.

Marking fixed released.

[Dave Myron (therealdave-myron)]

This appears to have cropped up again. `apt-get source imagemagick` fails with a 403 from us-east-1.ec2.archive.ubuntu.com ("west" fails in the same way). `apt-get clean` and `apt-get update` doesn't resolve the issue. I have the pipelining fix in place on 12.04.1 LTS.