apt 403 Forbidden on some packages

Bug #987182 reported by radle on 2012-04-23
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
S3AptMirror
High
Unassigned
Ubuntu
High
Unassigned

Bug Description

When I run "apt-get install ntp" it fails with

Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  ntp-doc
The following NEW packages will be installed:
  ntp
0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
Need to get 517kB of archives.
After this operation, 1,323kB of additional disk space will be used.
Err http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ lucid-updates/main ntp 1:4.2.4p8+dfsg-1ubuntu2.1
  403 Forbidden
Failed to fetch http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/n/ntp/ntp_4.2.4p8+dfsg-1ubuntu2.1_i386.deb 403 Forbidden
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

radle (0-ra) wrote :

Works good with us-west-1 though.

This is a know problem with S3.

Populate a new file named /etc/apt/apt.conf.d/90-fix-s3 with:
Acquire::http::Pipeline-Depth "0";

This problem with go away.

Changed in ubuntu-on-ec2:
status: New → Won't Fix
assignee: nobody → Ben Howard (utlemming)
importance: Undecided → Low

Confirmed:
Suggested packages:
  ntp-doc
The following NEW packages will be installed:
  ntp
0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
Need to get 559kB of archives.
After this operation, 1,450kB of additional disk space will be used.
Err http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ lucid-updates/main ntp 1:4.2.4p8+dfsg-1ubuntu2.1
  403 Forbidden
Failed to fetch http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/n/ntp/ntp_4.2.4p8+dfsg-1ubuntu2.1_amd64.deb 403 Forbidden

Changed in ubuntu-on-ec2:
status: Won't Fix → Confirmed
importance: Low → High

Sigh...looks like a problem with S3...

root@domU-12-31-39-05-49-DC:/etc/apt# curl https://s3-us-west-1.amazonaws.com/us-west-1.ec2.archive.ubuntu.com/ubuntu/pool/main/n/ntp/ntp_4.2.4p8+dfsg-1ubuntu2.1_amd64.deb -o /dev/null
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 546k 100 546k 0 0 414k 0 0:00:01 0:00:01 --:--:-- 561k
root@domU-12-31-39-05-49-DC:/etc/apt# curl http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/n/ntp/ntp_4.2.4p8+dfsg-1ubuntu2.1_amd64.deb -o /dev/null
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
115 231 0 231 0 0 834 0 --:--:-- --:--:-- --:--:-- 2179
root@domU-12-31-39-05-49-DC:/etc/apt# curl http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/n/ntp/ntp_4.2.4p8+dfsg-1ubuntu2.1_amd64.deb
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>D382C6B38E86A602</RequestId><HostId>p4SZkVYnZvkV95GRKXRTqWR6QQ0YSLFsG2w04sjGeTJL780SoPwhgjViUd7S+CQW</HostId></Error>root@domU-12-31-39-05-49-DC:/etc/apt#

So the problem that I am seeing is that if you access the bucket via:
http://s3-us-west-1.amazonaws.com/us-west-1.ec2.archive.ubuntu.com/.... it works

while
http://us-west-2.ec2.archive.ubuntu.com.s3.amazonaws.com/... fails

The difference is that the one that is failing is caused by accessing the bucket the new way, while the first way is the old way.

Eric Hammond (esh) wrote :

You need to specify the correct S3 region (s3-us-west-1) when using the "new way":

  http://us-west-1.ec2.archive.ubuntu.com.s3-us-west-1.amazonaws.com/...

For example:

  http://us-west-1.ec2.archive.ubuntu.com.s3-us-west-1.amazonaws.com/ubuntu/pool/main/n/ntp/ntp_4.2.4p8+dfsg-1ubuntu2.1_amd64.deb

Note, however, that you cannot use "https" URLs when specifying the bucket name in the hostname as it will not match the SSL certificate hostname.

Pending Amazon.

Yeah, so I was completely wrong here. I was comparing us-west-1 to us-west-2.

However, Amazon reports that this is a problem with object ownership. They are working on resolving this.

radle (0-ra) wrote :

Yes, the bucket owner should make the file accessible by public.
Maybe it concerns not only ntp package, I just got it by accident.
So - bucket owner should make all the package files accessible by public.

The real problem is "+"'s that S3 deals with very, very poorly. You upload keys with one encoding and request them with another encoding. Which means you have to upload any file with "+"'s three times -- once with "+"'s, then replacing the "+" with " " and "%2B". If any of those three are missing, it blows up.

I've put a fix in the code that looks for those files and uploads them. Hopefully this fixes the problems.

Shane Meyers (shanemeyers) wrote :

I'm having trouble today with pulling a package from the s3 package repositories and this error doesn't appear to be transient. I'm consistently getting a "403 Forbidden" response for the following package. This one does have a "+" in the filename, so I think my issue may be related.

http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/libx/libxml-sax-perl/libxml-sax-perl_0.96+dfsg-2_all.deb

This prevented apt from completing a package install run, which stopped a number of instances from completing their boot process (we install a number of packages upon first boot with ec2).

Relevant line in my /etc/apt/sources.list file (to make sure I'm not incorrectly referencing the bucket; this has been working for many months until today):
deb http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ oneiric main

Scott Moser (smoser) wrote :

Ben (utlemming) is aware of this and is working on getting it fixed.
Thank you for making sure we were aware, and your continued testing.

summary: - 403 Forbidden on some packages
+ apt 403 Forbidden on some packages
affects: ubuntu-on-ec2 → ubuntu
Changed in ubuntu:
status: Confirmed → Fix Released

For the S3 mirrors, there were three distinct issues at play.

Background: S3 is deals with "+" in backwards ways. The short version is that if you have a "+" in the name, you have to upload it three times to deal with character encoding problems -- once with "+", and then once with "%2B" and " " replacing any "+".

1. Amazon is now enforcing exact URL fetching. It used to be that if you requested "foo%2Bbar" and there was a file of "foo+bar", as long as your request was using utf-8 encoding you would get the file "foo+bar". Now, if you request "foo%2Bbar", even though the request is interpetted as "foo+bar" in ascii, you must have "foo%2Bbar" in the bucket.

2. My code had a bug where it was not properly syncronizing and checking the different encodings of "+". The result was that some "+" files were not getting pushed up.

3. My code had a second bug which was caused by archive.ubuntu.com not returning content-type in the http headers. Previously I was using the content-type header from archive.ubuntu.com to set the content type header for S3. However, recently that changed, which was causing syncronization problems.

The issue was fixed late last on 2012-05-23. After looking at the logs, the problems seems to have been fixed.

Marking fixed released.

Changed in s3aptmirror:
assignee: nobody → Ben Howard (utlemming)
importance: Undecided → High
status: New → Fix Released
Dave Myron (therealdave-myron) wrote :

This appears to have cropped up again. `apt-get source imagemagick` fails with a 403 from us-east-1.ec2.archive.ubuntu.com ("west" fails in the same way). `apt-get clean` and `apt-get update` doesn't resolve the issue. I have the pipelining fix in place on 12.04.1 LTS.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers