support adding region/availability to mirror selection

Bug #1037727 reported by Scott Moser on 2012-08-16
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
High
Unassigned
Precise
High
Unassigned

Bug Description

=== Begin SRU Information ===
[Impact]
 * Currently, local mirror selection is done specifically for EC2 instances
   in the DataSource code. cloud-init will select
  'us-east-1a.ec2.archive.ubuntu.com' as a mirror in EC2. However, in other
  clouds, there is no way to get a local mirror selected.

  The change here is to allow mirrors to be specified in configuration based
  on templates that can include the "availability-zone". This makes the EC2
  code generic and allows a cloud provider to have cloud-init select a local
  mirror simply by presenting 'availability-zone' in the metadata.

  With this change, if a cloud provider puts 'region-1.mycloud' into their
  datasource metadata as the availability-zone, cloud-init will select
  a mirror at 'region-1.mycloud.clouds.archive.ubuntu.com'. By default,
  Canonical's dns will return that as a wildcard for archive.ubuntu.com,
  but the information is availble now in the dns request to specify
  the address of a local mirror.

[Test Case]
 * This is really a feature backport. There is no specific bug here.
   However, the behavior change is easily seen by running an instance on a
   openstack cloud. Previously, the mirror selected would be
   'archive.ubuntu.com'. With the change in place, the mirror will be:
   'nova.clouds.archive.ubuntu.com' as by default openstack provides the
   string 'nova' as the availability-zone.

[Regression Potential]
 * Mirror selection will change. It is possible that users relying on
   the old behavior will have to modify something.

[Other Information]
 * We want this behavior brought back to 12.04 so that LTS images can "just work"
   to select a local mirror simply by a change in DNS records. It will enable
   large clouds to provide local mirrors easily.

=== End SRU Information ===

currently, the ec2 datasource has some code that selects the mirror location by looking at the availability-zone.
I think this is generally a pretty good idea. I'd like to extend that to other datasources, and not limit to ".ec2.archive.ubuntu.com".

The goal would be to basically check for a mirror at:
  <availability-zone>[.<cloud>][dns-subdomain]

dns subdomain would default to '.cloud.archive.ubuntu.com'.

if the datasource provided the 'cloud' value, we'd use that, but otherwise just use availability-zone.

This would fit in with what we're currently doing on ec2, but would extend to others.
I'd just treat the "cc-direction-[0-9]" availability-zones specially, to know that this is an ec2 one. Then basically append ".ec2" in that case.

Related branches

Scott Moser (smoser) on 2012-08-16
Changed in cloud-init (Ubuntu):
status: New → Triaged
importance: Undecided → High
Eric Hammond (esh) wrote :

I think you mean <region> and not <availability-zone> since availability zone names mean different things for different AWS accounts.

Scott Moser (smoser) wrote :

you're right. for amazon, 'availability-zone' there is 'region'.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.0~bzr633-0ubuntu1

---------------
cloud-init (0.7.0~bzr633-0ubuntu1) quantal; urgency=low

  * New upstream snapshot.
    * support creating users on boot. remove requirement for a 'ubuntu'
      user to be previously present in image. (LP: #1028503)
    * add experimental apt_reboot_if_required flag to reboot if necessary
      after upgrade or package install (LP: #1038108)
    * improve mirror selection for a distro:
      * support arm mirrors (LP: #1028501)
      * support seeding security mirror (LP: #1006963)
      * support dns mirrors including availability-zone reference
        (LP: #1037727)
    * include a "None" datasource so items like ssh host key generation
      occur if there is no other metadata service. (LP: #906669)
    * print authorized_keys for users to the console (LP: #1010582)
    * Add RHEVm and vSphere support as datasource AltCloud [Joseph VLcek]
 -- Scott Moser <email address hidden> Thu, 23 Aug 2012 01:06:34 -0400

Changed in cloud-init (Ubuntu):
status: Triaged → Fix Released
Scott Moser (smoser) on 2012-10-05
Changed in cloud-init (Ubuntu Precise):
status: New → Triaged
importance: Undecided → High
Scott Moser (smoser) on 2012-11-08
description: updated

Hello Scott, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Precise):
status: Triaged → Fix Committed
tags: added: verification-needed
Scott Moser (smoser) wrote :

I verified by getting an image (amd64 20121210), mounting it loopback, modifying /etc/resolv.conf then:

export DEBIAN_FRONTEND=noninteractive
echo "deb $mirror $rel-proposed main universe" > "/etc/apt/sources.list.d/$rel-proposed.list"
{
apt-get update --assume-yes --quiet &&
apt-get install cloud-init --assume-yes --quiet --option Dpkg::Options::=--force-confold
} </dev/null

Launched an instance of that.
Then:
$ ec2metadata --availability-zone
nova
$ grep "^deb .*ubuntu.*$(lsb_release -sc) main" /etc/apt/sources.list
deb http://nova.clouds.archive.ubuntu.com/ubuntu/ precise main
$ sed -n '/system_info/,$p' /etc/cloud/cloud.cfg
system_info:
   package_mirrors:
     - arches: [i386, amd64]
       failsafe:
         primary: http://archive.ubuntu.com/ubuntu
         security: http://security.ubuntu.com/ubuntu
       search:
         primary:
           - http://%(ec2_region)s.ec2.archive.ubuntu.com/ubuntu/
           - http://%(availability_zone)s.clouds.archive.ubuntu.com/ubuntu/
         security: []
     - arches: [armhf, armel, default]
       failsafe:
         primary: http://ports.ubuntu.com/ubuntu-ports
         security: http://ports.ubuntu.com/ubuntu-ports

tags: added: verification-done
removed: verification-needed
Clint Byrum (clint-fewbar) wrote :

Hello Scott, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: removed: verification-done
tags: added: verification-needed
Scott Moser (smoser) on 2013-01-09
tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.3

---------------
cloud-init (0.6.3-0ubuntu1.3) precise-proposed; urgency=low

  * debian/patches/lp-1070345-landscape-restart-after-change.patch,
    debian/patches/lp-1066115-landscape-install-fix-perms.patch:
    fix missing or incorrect imports (LP: #1070345, LP: #1066115).

cloud-init (0.6.3-0ubuntu1.2) precise-proposed; urgency=low

  * debian/patches/lp-978127-maas-oauth-fix-bad-clock.patch: fix usage of
    oauth in maas data source if local system has a bad clock (LP: #978127)
  * debian/cloud-init.preinst: fix bug where user data scripts re-ran on
    upgrade from 10.04 versions (LP: #1049146)
  * debian/patches/lp-974509-detect-dns-server-redirection.patch: detect dns
    server redirection and disable searching dns for a mirror named
    'ubuntu-mirror' (LP: #974509)
  * debian/patches/lp-1018554-shutdown-message-to-console.patch: write a
    message to the console on system shutdown. (LP: #1018554)
  * debian/patches/lp-1066115-landscape-install-fix-perms.patch: install
    landscape package if needed which will ensure proper permissions on config
    file (LP: #1066115).
  * debian/patches/lp-1070345-landscape-restart-after-change.patch: restart
    landscape after modifying config (LP: #1070345)
  * debian/patches/lp-1073077-zsh-workaround-for-locale_warn.patch: avoid
    warning when user's shell is zsh (LP: #1073077)
  * debian/patches/rework-mirror-selection.patch: improve mirror selection by:
    * allowing region/availability-zone to be part of mirror (LP: #1037727)
    * making mirror selection arch aware (LP: #1028501)
    * allow specification of a security mirror (LP: #1006963)
 -- Scott Moser <email address hidden> Thu, 13 Dec 2012 12:16:56 -0500

Changed in cloud-init (Ubuntu Precise):
status: Fix Committed → Fix Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers