EC2 apt repository DNS resolution on VPC instances

[Eric Hammond]

DNS names like eu-west-1.ec2.archive.ubuntu.com (apt repository for eu-west-1 on EC2) are currently resolving to private IP addresses (e.g., "10.").

An EC2 instance running in VPC cannot access these repositories.

More details and possible fixes at:

  https://forums.aws.amazon.com/thread.jspa?threadID=73379

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: cloud-init 0.6.1-0ubuntu8
ProcVersionSignature: User Name 2.6.38-8.42-virtual 2.6.38.2
Uname: Linux 2.6.38-8-virtual i686
Architecture: i386
Date: Fri Aug 12 03:19:39 2011
Ec2AMI: ami-06ad526f
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1a
Ec2InstanceType: m1.small
Ec2Kernel: aki-407d9529
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)

[Clint Byrum (clint-fewbar)]

cloud-init should probably detect that VPC is in use, and not assume that these regional archives are accessible. Marking as importance High (since it affects all VPC users), and Confirmed.

[Eric Hammond (esh)]

Amazon recommends fixing this through DNS instead of through software on the instance.

Instead of resolving eu-west-1.ec2.archive.ubuntu.com directly to an A record of the internal IP address starting with "10.", Canonical should change it to resolve to a CNAME of the external elastic IP address hostname (e.g., ec2-NNN-NNN-NNN-NNN.compute-1.amazonaws.com)

This will resolve to the internal "10." IP address for normal EC2 instances saving performance and cost, and will resolve to the external elastic IP address for VPC EC2 instances.

Making this change not only clears up the issue with VPC, but any other future situation where an EC2 instance cannot access "10." IP addresses and EC2 DNS points it to the external IP address of the apt repository.

This approach also makes it easier for Canonical when the apt repository instance gets a new internal IP address (e.g., stop/start, failure). Canonical would simply reassociate the elastic IP address with the new/restarted instance and all DNS would resolve to the correct new IP address without Canonical making any changes to their DNS servers.

If Canonical is concerned about the EC2 apt repositories being accessed from outside of EC2 (I wouldn't be, but it's your choice), Amazon recommends the following:

"To protect the rep from being accessed outside of AWS, lockdown the security group rules to allow only traffic from the public AWS IP ranges (https://forums.aws.amazon.com/ann.jspa?annID=1097) and to the 10. network."

Here is a github repository that keeps up to date lists of the EC2 IP address ranges in a format that is easy to parse:

  https://github.com/garnaat/missingcloud

[Clint Byrum (clint-fewbar)]

Excerpts from Eric Hammond's message of Fri Aug 12 23:42:37 UTC 2011:
> Amazon recommends fixing this through DNS instead of through software on
> the instance.
>
> Instead of resolving eu-west-1.ec2.archive.ubuntu.com directly to an A
> record of the internal IP address starting with "10.", Canonical should
> change it to resolve to a CNAME of the external elastic IP address
> hostname (e.g., ec2-NNN-NNN-NNN-NNN.compute-1.amazonaws.com)
>
> This will resolve to the internal "10." IP address for normal EC2
> instances saving performance and cost, and will resolve to the external
> elastic IP address for VPC EC2 instances.

OH! I didn't realize that this was the case.

I'll open a case with our ops team to look into this, thanks for the
extra info!

>
> Making this change not only clears up the issue with VPC, but any other
> future situation where an EC2 instance cannot access "10." IP addresses
> and EC2 DNS points it to the external IP address of the apt repository.
>
> This approach also makes it easier for Canonical when the apt repository
> instance gets a new internal IP address (e.g., stop/start, failure).
> Canonical would simply reassociate the elastic IP address with the
> new/restarted instance and all DNS would resolve to the correct new IP
> address without Canonical making any changes to their DNS servers.
>
> If Canonical is concerned about the EC2 apt repositories being accessed
> from outside of EC2 (I wouldn't be, but it's your choice), Amazon
> recommends the following:
>
> "To protect the rep from being accessed outside of AWS, lockdown the
> security group rules to allow only traffic from the public AWS IP ranges
> (https://forums.aws.amazon.com/ann.jspa?annID=1097) and to the 10.
> network."
>
> Here is a github repository that keeps up to date lists of the EC2 IP
> address ranges in a format that is easy to parse:
>
> https://github.com/garnaat/missingcloud
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/824947
>
> Title:
> EC2 apt repository DNS resolution on VPC instances
>
> Status in “cloud-init” package in Ubuntu:
> Confirmed
>
> Bug description:
> DNS names like eu-west-1.ec2.archive.ubuntu.com (apt repository for
> eu-west-1 on EC2) are currently resolving to private IP addresses
> (e.g., "10.").
>
> An EC2 instance running in VPC cannot access these repositories.
>
> More details and possible fixes at:
>
> https://forums.aws.amazon.com/thread.jspa?threadID=73379
>
> ProblemType: Bug
> DistroRelease: Ubuntu 11.04
> Package: cloud-init 0.6.1-0ubuntu8
> ProcVersionSignature: User Name 2.6.38-8.42-virtual 2.6.38.2
> Uname: Linux 2.6.38-8-virtual i686
> Architecture: i386
> Date: Fri Aug 12 03:19:39 2011
> Ec2AMI: ami-06ad526f
> Ec2AMIManifest: (unknown)
> Ec2AvailabilityZone: us-east-1a
> Ec2InstanceType: m1.small
> Ec2Kernel: aki-407d9529
> Ec2Ramdisk: unavailable
> PackageArchitecture: all
> ProcEnviron:
> LANG=en_US.UTF-8
> SHELL=/bin/bash
> SourcePackage: cloud-init
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications a...

[Scott Moser (smoser)]

This is fix-released in maverick and later (bug 615545).
The only supported release that it would be present on would be 10.04.

You reported this using ubuntu-bug on a 11.04 instance. Are you actually stating that you saw this bug with that ami?

Eric's suggestion does seem reasonable. The only change is that Canonical would have to either:
a.) make the mirrors available from outside a region
b.) manually track announcements like https://forums.aws.amazon.com/ann.jspa?annID=1097 and update security groups.

[Scott Moser (smoser)]

To be clear, I marked this fix-releases as 10.10 and later it should not be an issue.
Eric, if you did see this issue on 11.04, please let me know.

[Eric Hammond (esh)]

Sorry, I should have been clear in the original bug report that I was submitting it on behalf of Amazon and another customer and did not experience it myself on that particular instance or AMI.

Also, I'm not sure that "lack of a public IP" address as described in #615545 is sufficient to determine if you are in VPC now-a-days. When VPC was launched, all instances were entirely private, but Amazon later released the ability for a VPC instance to have a public IP address with direct Internet access as an optional feature depending on the customer's security policies.

Modifying Canonical's DNS seems like the best approach to support current and future AWS services and features. Using a CNAME to an Elastic IP Address transfers the burden to Amazon for determining how the instance should access the apt repository (internal or external).

[Scott Moser (smoser)]

Can you please verify that maverick (or later) images have this issue?
Also, could you provide the output of:
python -c 'import boto.utils, pprint; pprint.pprint(boto.utils.get_instance_metadata())'

[Mitchell Hashimoto (mitchell-hashimoto)]

Note that I just launched the latest 10.04 64-bit instance-store AMI and this issue is still around. "ami-fbbf7892"

Will this fix be backported to the LTS release? As we move our infrastructure into VPC, this is becoming very important.

[Scott Moser (smoser)]

Just to make tracking easier, I'm marking this as a duplicat of bug 615545.