Ubuntu

cloud-init selects wrong mirror with dns server redirection

Reported by Chuck Short on 2012-04-05
32
This bug affects 4 people
Affects Status Importance Assigned to Milestone
cloud-init
Medium
Unassigned
cloud-init (Ubuntu)
Low
Unassigned
Precise
Medium
Scott Moser

Bug Description

=== Begin SRU Information ===
[Impact]
 * If a user launches an cloud-image in an environment where the DNS
   server does DNS redirection (also known as DNS hijacking), then the
   system will configure itself to use a mirror at
   http://ubuntu-mirror/ubuntu .

   This behavior was by design in cloud-init. It was intended to allow
   a cloud provider to set up a mirror at 'ubuntu-mirror' and have
   cloud-init select the mirror transparently. However, this causes
   failure if dns hijacking ins being used.

 * The fix is two fold:
   a.) cloud-init's code that checks for DNS entries is now protected
       by logic that detects the dns hijacking and does not consider
       such entries as valid.
   b.) the selection of the "search dns for 'ubuntu-mirror'" behavior
       has been disabled by default.

[Test Case]
 * download cloud image from cloud-images.ubuntu.com, and convert for use
   $ url="http://cloud-images.ubuntu.com/server/releases/precise/release-20121026.1/"
   $ wget "$url/ubuntu-12.04-server-cloudimg-i386-disk1.img" -O disk.img.orig
   $ qemu-img convert -O raw disk.img.orig disk.raw.dist

 * have *some* way to add 'ubuntu-mirror' to the dns for kvm guests (or
   just have a service provider that uses dns hijacking)

   I used dnsmasq on a server system, and can control this by adding entries
   to /etc/hosts. You need to be able to configure your system such
   that 'host ubuntu-mirror' returns something:
   $ host ubuntu-mirror
   ubuntu-mirror has address 192.168.1.1

 * boot kvm guest (cloud-localds from 12.10 cloud-utils)
   $ qemu-img create -f qcow2 -b disk.raw.dist disk.img
   # this user-data just sets password so you can log in
   $ cat user-data.txt
   #cloud-config
   password: passw0rd
   chpasswd: { expire: False }
   ssh_pwauth: True

   $ cloud-localds seed.img user-data.txt
   $ kvm -m 512 -curses -drive file=seed.img,if=virtio \
      -drive file=disk.img,if=virtio

 * login and see problem.
   looking at sources.list will show 'ubuntu-mirror' entry

[Regression Potential]
 * A regression is possible due to this designed change in behavior. If
   someone was expecting the 'ubuntu-mirror' mirror to be automatically
   located they will subsequently have to take different means to
   accomplish this. That can be either:
    a.) modifying the image to set 'apt_mirror_search_dns: true'
    b.) doing 'a' through user-data user-data
 * The change made in quantal was tested for regression as described in
   comment 5 below.

[Other Info]
 * The changes here also enable 2 other fixes
    * allowing region/availability-zone to be part of mirror (bug 1037727)
    * making mirror selection arch aware (bug #1028501)

=== End SRU Information ===

=== original bug report ===
Hi,

I have Rogers as an ISP in the great white north, and use their DNS servers. However they run DNS redirectors so that when you get a bad domain then it does bogus things to the hostname. Anyways this resolves in unresovalble hosts in my /etc/apt/sources.list when Im running an openstack instance.

ubuntu@server-5:/var/log$ host nov.ec2.archive.ubuntu.com
nov.ec2.archive.ubuntu.com has address 8.15.7.107
nov.ec2.archive.ubuntu.com has address 63.251.179.17
Host nov.ec2.archive.ubuntu.com not found: 3(NXDOMAIN)
Host nov.ec2.archive.ubuntu.com not found: 3(NXDOMAIN)

The console output is the following:

http://paste.ubuntu.com/916324/

If you have any questions please let me know.

Regards
chuck

James Page (james-page) on 2012-04-12
Changed in cloud-init (Ubuntu):
importance: Undecided → Low
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cloud-init (Ubuntu):
status: New → Confirmed
Scott Moser (smoser) wrote :

The clear solution here is to use google dns (8.8.8.8) or opendns or some other dns server that does not provide bogus answers for non-existing domains.

That said, I think we'll likely be finding a better solution for discovery of a local mirror by an instance.

Scott Moser (smoser) on 2012-06-25
summary: - Issues resolving ubuntu archives.
+ cloud-init selects wrong mirror with dns server redirection
Scott Moser (smoser) wrote :

This was fixed in trunk in revno 612.

Changed in cloud-init:
status: New → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.0~bzr614-0ubuntu1

---------------
cloud-init (0.7.0~bzr614-0ubuntu1) quantal; urgency=low

  * New upstream snapshot.
    * disable searching for 'ubuntu-mirror' in local dns to find a
      local mirror (LP: #974509)
    * emit the cloud-config event (LP: #1028674)
    * write timestamps to console on reboot and shutdown (LP: #1018554)
 -- Scott Moser <email address hidden> Fri, 03 Aug 2012 14:55:37 -0400

Changed in cloud-init (Ubuntu):
status: Confirmed → Fix Released
Scott Moser (smoser) on 2012-08-06
Changed in cloud-init (Ubuntu Precise):
assignee: nobody → Scott Moser (smoser)
importance: Undecided → Medium
status: New → Triaged
Scott Moser (smoser) wrote :

This is fixed in cloud-init trunk and in the ubuntu quantal package listed in comment 4.
I've tested this by:

 * setup:
   * have *some* way to add 'ubuntu-mirror' to the dns for kvm guests
     I used dnsmasq on a server system, and can control this by adding entries
     to /etc/hosts. You need to be able to configure your system such
     that 'host ubuntu-mirror' returns something:
     $ host ubuntu-mirror
     ubuntu-mirror has address 192.168.1.1
   * use make-seed-disk [1] to create a seed disk for cloud-init.
     $ cat user-data.txt
     #cloud-config
     password: passw0rd
     chpasswd: { expire: False }
     ssh_pwauth: True
     $ make-seed-disk seed.img user-data.txt

 * verify issue in released version of precise or quantal alpha3:
   * add 'ubuntu-mirror' entry to dns
   * boot kvm guest
   * check mirror in /etc/apt/sources by:
     $ grep "^deb .*main$" /etc/apt/sources.list
   * quantal-alpha3: ubuntu-mirror is selected
   * precise 20120728: ubuntu-mirror is selected

 * verify fix in quantal 20120804 or later:
   * add 'ubuntu-mirror' entry to dns
   * boot kvm guest of quantal 20120804 or later.
   * check mirror in /etc/apt/sources by:
     $ grep "^deb .*main$" /etc/apt/sources.list
 * verify no regression:
   * remove 'ubuntu-mirror' entry from dns, boot, check /etc/apt/sources.list
 * verify explicit enablement works:
   * add ubuntu-mirror
   * create seed disk with 'apt_mirror_search_dns: True' in user-data
   * verify that /etc/apt/sources.list shows ubuntu-mirror

 * Notes, boot kvm with:
    kvm -m 512 -curses -drive file=seed.img,if=virtio \
      -drive file=disk.img,if=virtio

--
[1] http://smoser.brickies.net/git/?p=tildabin.git;a=blob;f=make-seed-disk;hb=HEAD

Scott Moser (smoser) on 2012-08-16
Changed in cloud-init:
importance: Undecided → Medium
Scott Moser (smoser) on 2012-10-01
Changed in cloud-init:
status: Fix Committed → Fix Released
Scott Moser (smoser) on 2012-11-08
description: updated

Hello Chuck, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Precise):
status: Triaged → Fix Committed
tags: added: verification-needed
Scott Moser (smoser) on 2012-12-12
description: updated
Scott Moser (smoser) wrote :

following steps in description, i booted a cloud-image with new cloud-init and verified that it did not have 'ubuntu-mirror' in the /etc/apt/sources.list entries.

tags: added: verification-done
removed: verification-needed
Clint Byrum (clint-fewbar) wrote :

Hello Chuck, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: removed: verification-done
tags: added: verification-needed
Scott Moser (smoser) on 2013-01-08
tags: added: verification-done
removed: verification-needed

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.3

---------------
cloud-init (0.6.3-0ubuntu1.3) precise-proposed; urgency=low

  * debian/patches/lp-1070345-landscape-restart-after-change.patch,
    debian/patches/lp-1066115-landscape-install-fix-perms.patch:
    fix missing or incorrect imports (LP: #1070345, LP: #1066115).

cloud-init (0.6.3-0ubuntu1.2) precise-proposed; urgency=low

  * debian/patches/lp-978127-maas-oauth-fix-bad-clock.patch: fix usage of
    oauth in maas data source if local system has a bad clock (LP: #978127)
  * debian/cloud-init.preinst: fix bug where user data scripts re-ran on
    upgrade from 10.04 versions (LP: #1049146)
  * debian/patches/lp-974509-detect-dns-server-redirection.patch: detect dns
    server redirection and disable searching dns for a mirror named
    'ubuntu-mirror' (LP: #974509)
  * debian/patches/lp-1018554-shutdown-message-to-console.patch: write a
    message to the console on system shutdown. (LP: #1018554)
  * debian/patches/lp-1066115-landscape-install-fix-perms.patch: install
    landscape package if needed which will ensure proper permissions on config
    file (LP: #1066115).
  * debian/patches/lp-1070345-landscape-restart-after-change.patch: restart
    landscape after modifying config (LP: #1070345)
  * debian/patches/lp-1073077-zsh-workaround-for-locale_warn.patch: avoid
    warning when user's shell is zsh (LP: #1073077)
  * debian/patches/rework-mirror-selection.patch: improve mirror selection by:
    * allowing region/availability-zone to be part of mirror (LP: #1037727)
    * making mirror selection arch aware (LP: #1028501)
    * allow specification of a security mirror (LP: #1006963)
 -- Scott Moser <email address hidden> Thu, 13 Dec 2012 12:16:56 -0500

Changed in cloud-init (Ubuntu Precise):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers