cloud-init selects wrong mirrors for arm

[Scott Moser]

[Impact]
 * The official Ubuntu mirrors for armhf and armel arch are on
   ports.ubuntu.com where as i386 and amd64 are on archive.ubuntu.com.
   Cloud-init has code that selects a local mirror, but that code in 12.04
   did not take arch into into consideration at all.

   As a result, if you're using cloud-init on arm with 12.04, you need to
   either
   a.) tell cloud-init to "preserve_sources_list"
   b.) provide the correct 'mirror' in cloud-config syntax always when you
       launch the instance, and this isn't really even sufficient as
       cloud-init's template for sources.list references security.ubuntu.com
       which is also not valid for arm.

   The fix here is to make the mirror selection code architecture specific
   and to also make it handle security.ubuntu.com separately. Looking at
   the default configuration probably makes it obvious how it works:

    | package_mirrors:
    | - arches: [i386, amd64]
    | failsafe:
    | primary: http://archive.ubuntu.com/ubuntu
    | security: http://security.ubuntu.com/ubuntu
    | search:
    | primary:
    | - http://%(ec2_region)s.ec2.archive.ubuntu.com/ubuntu/
    | - http://%(availability_zone)s.clouds.archive.ubuntu.com/ubuntu/
    | security: []
    | - arches: [armhf, armel, default]
    | failsafe:
    | primary: http://ports.ubuntu.com/ubuntu
    | security: http://ports.ubuntu.com/ubuntu

[Test Case]
 Simply letting cloud-init take its default behavior will show the
 brokenness on arm. The code is not arch specific at all so launching
 an EC2 instance (of i386 or amd64) will select the primary mirror
 of http://us-east-1.ec2.archive.ubuntu.com and security.ubuntu.com for
 security. This is clearly broken.

 Showing the correct working path requires actually running cloud-init
 within arm. This can be done using an arm image from
 cloud-images.ubuntu.com.

[Regression Potential]
 * Selection of the wrong mirror is really the potential failure path
   here, and on arm in 12.04, cloud-init already selects the wrong mirror
   so it is assured to not make the situation worse.

=== End SRU Information ===

cloud-inits' mirror selection does not include any logic to account for arch.
the arm arch are not available on ec2 mirrors nor on *.archive.ubuntu.com .

The best work around at the moment is to disable mirror selection via:
 apt_preserve_sources_list: True

For juju, that can be done by modifying juju/providers/common/utils.py to contain:
 cloud_config = {
     "apt-update": True,
     "apt-upgrade": True,
     "apt_preserve_sources_list": True,
     "ssh_authorized_keys": authorized_keys,
     "packages": [],
     "output": {"all": "| tee -a /var/log/cloud-init-output.log"}}

Related bugs:
 * bug 1028501 sources.list configuration does not cover security

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.0~bzr633-0ubuntu1

---------------
cloud-init (0.7.0~bzr633-0ubuntu1) quantal; urgency=low

  * New upstream snapshot.
    * support creating users on boot. remove requirement for a 'ubuntu'
      user to be previously present in image. (LP: #1028503)
    * add experimental apt_reboot_if_required flag to reboot if necessary
      after upgrade or package install (LP: #1038108)
    * improve mirror selection for a distro:
      * support arm mirrors (LP: #1028501)
      * support seeding security mirror (LP: #1006963)
      * support dns mirrors including availability-zone reference
        (LP: #1037727)
    * include a "None" datasource so items like ssh host key generation
      occur if there is no other metadata service. (LP: #906669)
    * print authorized_keys for users to the console (LP: #1010582)
    * Add RHEVm and vSphere support as datasource AltCloud [Joseph VLcek]
 -- Scott Moser <email address hidden> Thu, 23 Aug 2012 01:06:34 -0400

[Robie Basak (racb)]

Nominating for Precise. This is one way to allow MAAS to work on ARM deploying to Precise.

[Scott Moser (smoser)]

This was fixed in maas trunk with:
 https://code.launchpad.net/~smoser/maas/preserve-sources-list/+merge/127825

[Steve Langasek (vorlon)]

Hello Scott, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

I verified this (close to it) on a precise instance:
* enable proposed
  $ m=http://archive.ubuntu.com/ubuntu
  $ r=$(lsb_release -sc)
  $ echo "deb $m $r-proposed main" |
     sudo tee /etc/apt/sources.list.d/$r-proposed.list
* update cloud-init
  $ sudo apt-get install cloud-init
* hack dpkg so '--print-architecture' lies
  $ sudo mv /usr/bin/dpkg /usr/bin/dpkg.real
  $ sudo tee /usr/bin/dpkg <<"EOF"
  #!/bin/sh
  [ "$1" = "--print-architecture" -a $# -eq 1 ] && { echo "armhf" && exit; }
  exec "$0.real" "$@"
  EOF
  $ sudo chmod 755 /usr/bin/dpkg
* re-run the apt-update-upgrade config module
  $ sudo cp -a /etc/apt/sources.list /etc/apt/sources.list.save
  $ sudo cloud-init-cfg apt-update-upgrade always
* now, /etc/apt/sources.list has 'ports.ubuntu.com' in it
  and /var/log/cloud-init.log mentions it used it:

  $ grep mirror.info /var/log/cloud-init.log
  Dec 12 21:58:33 ip-10-117-77-88 [CLOUDINIT] cc_apt_update_upgrade.py[DEBUG]: mirror info: {'security': 'http://ports.ubuntu.com/ubuntu-ports', 'primary': 'http://ports.ubuntu.com/ubuntu-ports', 'mirror': 'http://ports.ubuntu.com/ubuntu-ports'}
  $ grep "^deb " /etc/apt/sources.list | head -n 3
  deb http://ports.ubuntu.com/ubuntu-ports precise main
  deb http://ports.ubuntu.com/ubuntu-ports precise-updates main
  deb http://ports.ubuntu.com/ubuntu-ports precise universe

[Clint Byrum (clint-fewbar)]

Hello Scott, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.3

---------------
cloud-init (0.6.3-0ubuntu1.3) precise-proposed; urgency=low

  * debian/patches/lp-1070345-landscape-restart-after-change.patch,
    debian/patches/lp-1066115-landscape-install-fix-perms.patch:
    fix missing or incorrect imports (LP: #1070345, LP: #1066115).

cloud-init (0.6.3-0ubuntu1.2) precise-proposed; urgency=low

  * debian/patches/lp-978127-maas-oauth-fix-bad-clock.patch: fix usage of
    oauth in maas data source if local system has a bad clock (LP: #978127)
  * debian/cloud-init.preinst: fix bug where user data scripts re-ran on
    upgrade from 10.04 versions (LP: #1049146)
  * debian/patches/lp-974509-detect-dns-server-redirection.patch: detect dns
    server redirection and disable searching dns for a mirror named
    'ubuntu-mirror' (LP: #974509)
  * debian/patches/lp-1018554-shutdown-message-to-console.patch: write a
    message to the console on system shutdown. (LP: #1018554)
  * debian/patches/lp-1066115-landscape-install-fix-perms.patch: install
    landscape package if needed which will ensure proper permissions on config
    file (LP: #1066115).
  * debian/patches/lp-1070345-landscape-restart-after-change.patch: restart
    landscape after modifying config (LP: #1070345)
  * debian/patches/lp-1073077-zsh-workaround-for-locale_warn.patch: avoid
    warning when user's shell is zsh (LP: #1073077)
  * debian/patches/rework-mirror-selection.patch: improve mirror selection by:
    * allowing region/availability-zone to be part of mirror (LP: #1037727)
    * making mirror selection arch aware (LP: #1028501)
    * allow specification of a security mirror (LP: #1006963)
 -- Scott Moser <email address hidden> Thu, 13 Dec 2012 12:16:56 -0500

[Colin Watson (cjwatson)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2276