cloud-init selects wrong mirror with dns server redirection

[Chuck Short]

=== Begin SRU Information ===
[Impact]
 * If a user launches an cloud-image in an environment where the DNS
   server does DNS redirection (also known as DNS hijacking), then the
   system will configure itself to use a mirror at
   http://ubuntu-mirror/ubuntu .

   This behavior was by design in cloud-init. It was intended to allow
   a cloud provider to set up a mirror at 'ubuntu-mirror' and have
   cloud-init select the mirror transparently. However, this causes
   failure if dns hijacking ins being used.

 * The fix is two fold:
   a.) cloud-init's code that checks for DNS entries is now protected
       by logic that detects the dns hijacking and does not consider
       such entries as valid.
   b.) the selection of the "search dns for 'ubuntu-mirror'" behavior
       has been disabled by default.

[Test Case]
 * download cloud image from cloud-images.ubuntu.com, and convert for use
   $ url="http://cloud-images.ubuntu.com/server/releases/precise/release-20121026.1/"
   $ wget "$url/ubuntu-12.04-server-cloudimg-i386-disk1.img" -O disk.img.orig
   $ qemu-img convert -O raw disk.img.orig disk.raw.dist

 * have *some* way to add 'ubuntu-mirror' to the dns for kvm guests (or
   just have a service provider that uses dns hijacking)

   I used dnsmasq on a server system, and can control this by adding entries
   to /etc/hosts. You need to be able to configure your system such
   that 'host ubuntu-mirror' returns something:
   $ host ubuntu-mirror
   ubuntu-mirror has address 192.168.1.1

 * boot kvm guest (cloud-localds from 12.10 cloud-utils)
   $ qemu-img create -f qcow2 -b disk.raw.dist disk.img
   # this user-data just sets password so you can log in
   $ cat user-data.txt
   #cloud-config
   password: passw0rd
   chpasswd: { expire: False }
   ssh_pwauth: True

   $ cloud-localds seed.img user-data.txt
   $ kvm -m 512 -curses -drive file=seed.img,if=virtio \
      -drive file=disk.img,if=virtio

 * login and see problem.
   looking at sources.list will show 'ubuntu-mirror' entry

[Regression Potential]
 * A regression is possible due to this designed change in behavior. If
   someone was expecting the 'ubuntu-mirror' mirror to be automatically
   located they will subsequently have to take different means to
   accomplish this. That can be either:
    a.) modifying the image to set 'apt_mirror_search_dns: true'
    b.) doing 'a' through user-data user-data
 * The change made in quantal was tested for regression as described in
   comment 5 below.

[Other Info]
 * The changes here also enable 2 other fixes
    * allowing region/availability-zone to be part of mirror (bug 1037727)
    * making mirror selection arch aware (bug #1028501)

=== End SRU Information ===

=== original bug report ===
Hi,

I have Rogers as an ISP in the great white north, and use their DNS servers. However they run DNS redirectors so that when you get a bad domain then it does bogus things to the hostname. Anyways this resolves in unresovalble hosts in my /etc/apt/sources.list when Im running an openstack instance.

ubuntu@server-5:/var/log$ host nov.ec2.archive.ubuntu.com
nov.ec2.archive.ubuntu.com has address 8.15.7.107
nov.ec2.archive.ubuntu.com has address 63.251.179.17
Host nov.ec2.archive.ubuntu.com not found: 3(NXDOMAIN)
Host nov.ec2.archive.ubuntu.com not found: 3(NXDOMAIN)

The console output is the following:

http://paste.ubuntu.com/916324/

If you have any questions please let me know.

Regards
chuck

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Scott Moser (smoser)]

The clear solution here is to use google dns (8.8.8.8) or opendns or some other dns server that does not provide bogus answers for non-existing domains.

That said, I think we'll likely be finding a better solution for discovery of a local mirror by an instance.

[Scott Moser (smoser)]

This was fixed in trunk in revno 612.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.0~bzr614-0ubuntu1

---------------
cloud-init (0.7.0~bzr614-0ubuntu1) quantal; urgency=low

  * New upstream snapshot.
    * disable searching for 'ubuntu-mirror' in local dns to find a
      local mirror (LP: #974509)
    * emit the cloud-config event (LP: #1028674)
    * write timestamps to console on reboot and shutdown (LP: #1018554)
 -- Scott Moser <email address hidden> Fri, 03 Aug 2012 14:55:37 -0400

[Scott Moser (smoser)]

This is fixed in cloud-init trunk and in the ubuntu quantal package listed in comment 4.
I've tested this by:

 * setup:
   * have *some* way to add 'ubuntu-mirror' to the dns for kvm guests
     I used dnsmasq on a server system, and can control this by adding entries
     to /etc/hosts. You need to be able to configure your system such
     that 'host ubuntu-mirror' returns something:
     $ host ubuntu-mirror
     ubuntu-mirror has address 192.168.1.1
   * use make-seed-disk [1] to create a seed disk for cloud-init.
     $ cat user-data.txt
     #cloud-config
     password: passw0rd
     chpasswd: { expire: False }
     ssh_pwauth: True
     $ make-seed-disk seed.img user-data.txt

 * verify issue in released version of precise or quantal alpha3:
   * add 'ubuntu-mirror' entry to dns
   * boot kvm guest
   * check mirror in /etc/apt/sources by:
     $ grep "^deb .*main$" /etc/apt/sources.list
   * quantal-alpha3: ubuntu-mirror is selected
   * precise 20120728: ubuntu-mirror is selected

 * verify fix in quantal 20120804 or later:
   * add 'ubuntu-mirror' entry to dns
   * boot kvm guest of quantal 20120804 or later.
   * check mirror in /etc/apt/sources by:
     $ grep "^deb .*main$" /etc/apt/sources.list
 * verify no regression:
   * remove 'ubuntu-mirror' entry from dns, boot, check /etc/apt/sources.list
 * verify explicit enablement works:
   * add ubuntu-mirror
   * create seed disk with 'apt_mirror_search_dns: True' in user-data
   * verify that /etc/apt/sources.list shows ubuntu-mirror

 * Notes, boot kvm with:
    kvm -m 512 -curses -drive file=seed.img,if=virtio \
      -drive file=disk.img,if=virtio

--
[1] http://smoser.brickies.net/git/?p=tildabin.git;a=blob;f=make-seed-disk;hb=HEAD

[Steve Langasek (vorlon)]

Hello Chuck, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

following steps in description, i booted a cloud-image with new cloud-init and verified that it did not have 'ubuntu-mirror' in the /etc/apt/sources.list entries.

[Clint Byrum (clint-fewbar)]

Hello Chuck, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Colin Watson (cjwatson)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.3

---------------
cloud-init (0.6.3-0ubuntu1.3) precise-proposed; urgency=low

  * debian/patches/lp-1070345-landscape-restart-after-change.patch,
    debian/patches/lp-1066115-landscape-install-fix-perms.patch:
    fix missing or incorrect imports (LP: #1070345, LP: #1066115).

cloud-init (0.6.3-0ubuntu1.2) precise-proposed; urgency=low

  * debian/patches/lp-978127-maas-oauth-fix-bad-clock.patch: fix usage of
    oauth in maas data source if local system has a bad clock (LP: #978127)
  * debian/cloud-init.preinst: fix bug where user data scripts re-ran on
    upgrade from 10.04 versions (LP: #1049146)
  * debian/patches/lp-974509-detect-dns-server-redirection.patch: detect dns
    server redirection and disable searching dns for a mirror named
    'ubuntu-mirror' (LP: #974509)
  * debian/patches/lp-1018554-shutdown-message-to-console.patch: write a
    message to the console on system shutdown. (LP: #1018554)
  * debian/patches/lp-1066115-landscape-install-fix-perms.patch: install
    landscape package if needed which will ensure proper permissions on config
    file (LP: #1066115).
  * debian/patches/lp-1070345-landscape-restart-after-change.patch: restart
    landscape after modifying config (LP: #1070345)
  * debian/patches/lp-1073077-zsh-workaround-for-locale_warn.patch: avoid
    warning when user's shell is zsh (LP: #1073077)
  * debian/patches/rework-mirror-selection.patch: improve mirror selection by:
    * allowing region/availability-zone to be part of mirror (LP: #1037727)
    * making mirror selection arch aware (LP: #1028501)
    * allow specification of a security mirror (LP: #1006963)
 -- Scott Moser <email address hidden> Thu, 13 Dec 2012 12:16:56 -0500

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2254