using random hostnames to detect dns proxies allows for false positives

Bug #1088611 reported by Steve Langasek
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
Medium
Unassigned
cloud-init (Ubuntu)
Fix Released
Wishlist
Unassigned
Xenial
Fix Released
Low
Unassigned
Zesty
Fix Released
Low
Unassigned

Bug Description

=== Begin SRU Template ===
[Impact]
Prior to this fix, cloud-init attempts to detect dns redirection by doing
dns queries for a random hostname and two invalid hostnames. Then, if
the result returned for the input value was the same as the response for
the invalid query cloud-init would assume that result was also invalid.

The change was to replace the random string with
  __cloud_init_expected_not_found__
This is a valid hostname and resolution will use the 'search' path in
resolv.conf where the other invalid domain names would not.

[Test Case]
The test case for this consists of excercising the the 'is_resolvable_url'
method in cloudinit.util and watching dns queries. To do this, see the
following steps:
a.) start an lxc container
   $ release=xenial
   $ name=$release-1088611
   $ lxc launch ubuntu-daily:$release $name
b.) start a dnsmasq server
   $ ./run-dnsmasq lxdbr0
   ...
   === listening on 10.75.205.2/24 ===

   # run-dnsmasq is attached and at
   # https://git.launchpad.net/~smoser/cloud-init/+git/sru-info/tree/bugs/lp-1088611/run-dnsmasq

c.) point /etc/resolv.conf at your server ip
   $ lxc exec $name -- sh -c 'exec >/etc/resolv.conf;
       echo nameserver 10.75.205.2; echo search foo;'

d.) perform query via is_resolvable_url watch dnsmasq output, expect
    to see the random query.
   $ lxc exec $name -- python3 -c 'import sys;
from cloudinit.util import is_resolvable_url;
print(is_resolvable_url(sys.argv[1]))' http://ubuntu.com

e.) upgrade to -proposed version
f.) perform query via is_resolvable_url, expect to *not* see random query.

[Regression Potential]
Immediate regression seems unlikely. Effectively the change in cloud-init
code path was simply to change a dns lookup attempt from rand() to a defined
string.

We chose a random string initially to make it difficult for a dns server to
circumvent cloud-init's attempt to identify dns redirection. The regression
path really then seems to involve a dns redirection service specifically
provding a response for '__cloud_init_expected_not_found__' that differs
from does-not-exist.example.com. Cloud-init could then be tricked into
believing that a apt mirror was valid where it previously would have
identified the dns redirection. The failure would be seen as errors
in package installation or 'apt-get update'.

[Other Info]
Upstream commit at
  https://git.launchpad.net/cloud-init/commit/?id=42a7b34a12

Original upstream commit at
  https://git.launchpad.net/cloud-init/commit/?id=1bb67be5bd

=== End SRU Template ===

The fix that's been applied for bug #974509 checks for the presence of a redirector by looking of three hostnames, and treating as invalid any results pointing to a matching address:

 - does-not-exist.example.com.
 - example.invalid.
 - a random, unqualified 32-character alphanumeric hostname.

The last of these carries a small but non-zero risk of colliding with a real hostname, and there's a small but non-zero risk that this host points to the same address as something we care about. If possible, it would be better to not include this random-host lookup in the algorithm, as somewhere, some day, chances are there will eventually be a collision, causing an incomprehensible and unreproducible failure for a user.

Related branches

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Instead of random request, you can check the checksum of:

http://start.ubuntu.com/connectivity-check.html

Which is highly available connectivity check hosted for installer purposes currently and is used by ubiquity.

Changed in cloud-init (Ubuntu):
status: New → Confirmed
importance: Undecided → Wishlist
Revision history for this message
Bob Aman (sporkmonger) wrote :

For anyone who's looking at DNS logs for security purposes, that "random, unqualified 32-character alphanumeric hostname" is going to cause significant issues for anyone trying to figure out what initiated that DNS request. It's also going to look suspiciously like a botnet's DGA. It would be better if these requests made it explicitly clear a) that it's intentionally invalid b) what triggered the request.

Scott Moser (smoser)
Changed in cloud-init:
status: New → Confirmed
importance: Undecided → Medium
Scott Moser (smoser)
Changed in cloud-init:
status: Confirmed → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.9-231-g80bf98b9-0ubuntu1

---------------
cloud-init (0.7.9-231-g80bf98b9-0ubuntu1) artful; urgency=medium

  * New upstream snapshot.
    - tests: remove 'yakkety' from releases as it is EOL.
    - systemd: make systemd-fsck run after cloud-init.service (LP: #1691489)
    - tests: Add initial tests for EC2 and improve a docstring.
    - locale: Do not re-run locale-gen if provided locale is system default.
    - archlinux: fix set hostname usage of write_file.
      [Joshua Powers] (LP: #1705306)
    - sysconfig: support subnet type of 'manual'.
    - Drop rand_str() usage in DNS redirection detection
      [Bob Aman] (LP: #1088611)

 -- Scott Moser <email address hidden> Mon, 31 Jul 2017 09:47:34 -0400

Changed in cloud-init (Ubuntu):
status: Confirmed → Fix Released
Scott Moser (smoser)
Changed in cloud-init (Ubuntu Xenial):
status: New → Confirmed
Changed in cloud-init (Ubuntu Zesty):
status: New → Confirmed
Changed in cloud-init (Ubuntu Xenial):
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Zesty):
importance: Undecided → Low
Changed in cloud-init (Ubuntu Xenial):
importance: Medium → Low
Revision history for this message
Scott Moser (smoser) wrote :
description: updated
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Steve, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-233-ge586fe35-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Xenial):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-xenial
Changed in cloud-init (Ubuntu Zesty):
status: Confirmed → Fix Committed
tags: added: verification-needed-zesty
Revision history for this message
Chris J Arges (arges) wrote :

Hello Steve, or anyone else affected,

Accepted cloud-init into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-233-ge586fe35-0ubuntu1~17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Chad Smith (chad.smith) wrote :
Download full text (5.6 KiB)

Validated xenial queries:

# Pre-release
$ lxc exec test-xenial -- dpkg-query --show cloud-init
cloud-init 0.7.9-153-g16a7302f-0ubuntu1~16.04.2
$ lxc exec test-xenial -- python3 -c 'import sys;
from cloudinit.util import is_resolvable_url;
print(is_resolvable_url(sys.argv[1]))' http://ubuntu.com

# from dns-masq
dnsmasq[27789]: query[A] does-not-exist.example.com from 10.249.77.13
dnsmasq[27789]: forwarded does-not-exist.example.com to 127.0.0.53
dnsmasq[27789]: query[AAAA] does-not-exist.example.com from 10.249.77.13
dnsmasq[27789]: forwarded does-not-exist.example.com to 127.0.0.53
dnsmasq[27789]: query[A] example.invalid from 10.249.77.13
dnsmasq[27789]: forwarded example.invalid to 127.0.0.53
dnsmasq[27789]: query[AAAA] example.invalid from 10.249.77.13
dnsmasq[27789]: forwarded example.invalid to 127.0.0.53
dnsmasq[27789]: query[A] example.invalid from 10.249.77.13
dnsmasq[27789]: forwarded example.invalid to 127.0.0.53
dnsmasq[27789]: query[AAAA] example.invalid from 10.249.77.13
dnsmasq[27789]: forwarded example.invalid to 127.0.0.53
dnsmasq[27789]: query[A] T77S9LM0QtKKC9WnG96MbWLQlRWgt5hj.foo from 10.249.77.13
dnsmasq[27789]: forwarded T77S9LM0QtKKC9WnG96MbWLQlRWgt5hj.foo to 127.0.0.53
dnsmasq[27789]: query[AAAA] T77S9LM0QtKKC9WnG96MbWLQlRWgt5hj.foo from 10.249.77.13
dnsmasq[27789]: forwarded T77S9LM0QtKKC9WnG96MbWLQlRWgt5hj.foo to 127.0.0.53
dnsmasq[27789]: query[A] T77S9LM0QtKKC9WnG96MbWLQlRWgt5hj from 10.249.77.13
dnsmasq[27789]: forwarded T77S9LM0QtKKC9WnG96MbWLQlRWgt5hj to 127.0.0.53
dnsmasq[27789]: query[AAAA] T77S9LM0QtKKC9WnG96MbWLQlRWgt5hj from 10.249.77.13
dnsmasq[27789]: forwarded T77S9LM0QtKKC9WnG96MbWLQlRWgt5hj to 127.0.0.53
dnsmasq[27789]: query[A] T77S9LM0QtKKC9WnG96MbWLQlRWgt5hj from 10.249.77.13
dnsmasq[27789]: forwarded T77S9LM0QtKKC9WnG96MbWLQlRWgt5hj to 127.0.0.53
dnsmasq[27789]: query[AAAA] T77S9LM0QtKKC9WnG96MbWLQlRWgt5hj from 10.249.77.13
dnsmasq[27789]: forwarded T77S9LM0QtKKC9WnG96MbWLQlRWgt5hj to 127.0.0.53
dnsmasq[27789]: query[A] ubuntu.com from 10.249.77.13
dnsmasq[27789]: cached ubuntu.com is 91.189.94.40
dnsmasq[27789]: query[AAAA] ubuntu.com from 10.249.77.13
dnsmasq[27789]: forwarded ubuntu.com to 127.0.0.53
dnsmasq[27789]: query[A] test-xenial.foo from 10.249.77.13
dnsmasq[27789]: forwarded test-xenial.foo to 127.0.0.53
dnsmasq[27789]: query[AAAA] test-xenial.foo from 10.249.77.13
dnsmasq[27789]: forwarded test-xenial.foo to 127.0.0.53
dnsmasq[27789]: query[A] test-xenial from 10.249.77.13
dnsmasq[27789]: forwarded test-xenial to 127.0.0.53
dnsmasq[27789]: query[AAAA] test-xenial from 10.249.77.13
dnsmasq[27789]: forwarded test-xenial to 127.0.0.53
dnsmasq[27789]: query[A] test-xenial from 10.249.77.13
dnsmasq[27789]: forwarded test-xenial to 127.0.0.53
dnsmasq[27789]: query[AAAA] test-xenial from 10.249.77.13
dnsmasq[27789]: forwarded test-xenial to 127.0.0.53
dnsmasq[27789]: query[A] test-xenial.foo from 10.249.77.13
dnsmasq[27789]: forwarded test-xenial.foo to 127.0.0.53
dnsmasq[27789]: query[AAAA] test-xenial.foo from 10.249.77.13
dnsmasq[27789]: forwarded test-xenial.foo to 127.0.0.53
dnsmasq[27789]: query[A] test-xenial from 10.249.77.13
dnsmasq[27789]: forwarded test-xenial to 127.0.0.53
dns...

Read more...

tags: added: verification-done-xenial
removed: verification-needed-xenial
Revision history for this message
Chad Smith (chad.smith) wrote :

--- zesty validation
# old version
$ lxc exec test-zesty -- dpkg-query --show cloud-init
cloud-init 0.7.9-153-g16a7302f-0ubuntu1~17.04.2

$ lxc exec test-zesty -- python3 -c 'import sys;
> from cloudinit.util import is_resolvable_url; print(is_resolvable_url(sys.argv[1]))' http://ubuntu.com

# dns-masq
dnsmasq[27789]: query[A] does-not-exist.example.com from 10.249.77.96
dnsmasq[27789]: forwarded does-not-exist.example.com to 127.0.0.53
dnsmasq[27789]: query[AAAA] does-not-exist.example.com from 10.249.77.96
dnsmasq[27789]: forwarded does-not-exist.example.com to 127.0.0.53
dnsmasq[27789]: query[A] rDKpshqoOwleYJvm7ByvpsJxQSUWqMh0.foo from 10.249.77.96
dnsmasq[27789]: forwarded rDKpshqoOwleYJvm7ByvpsJxQSUWqMh0.foo to 127.0.0.53
dnsmasq[27789]: query[AAAA] rDKpshqoOwleYJvm7ByvpsJxQSUWqMh0.foo from 10.249.77.96
dnsmasq[27789]: forwarded rDKpshqoOwleYJvm7ByvpsJxQSUWqMh0.foo to 127.0.0.53
dnsmasq[27789]: query[AAAA] ubuntu.com from 10.249.77.96
dnsmasq[27789]: forwarded ubuntu.com to 127.0.0.53
dnsmasq[27789]: query[A] ubuntu.com from 10.249.77.96
dnsmasq[27789]: forwarded ubuntu.com to 127.0.0.53
dnsmasq[27789]: reply ubuntu.com is 91.189.94.40

# proposed cloud-init version

$ lxc exec test-zesty -- dpkg-query --show cloud-init
cloud-init 0.7.9-233-ge586fe35-0ubuntu1~17.04.1dnsmasq[27789]: query[AAAA] does-not-exist.example.com from 10.249.77.96
dnsmasq[27789]: forwarded does-not-exist.example.com to 127.0.0.53
dnsmasq[27789]: query[A] does-not-exist.example.com from 10.249.77.96
dnsmasq[27789]: forwarded does-not-exist.example.com to 127.0.0.53
dnsmasq[27789]: query[A] __cloud_init_expected_not_found__.foo from 10.249.77.96
dnsmasq[27789]: forwarded __cloud_init_expected_not_found__.foo to 127.0.0.53
dnsmasq[27789]: query[AAAA] __cloud_init_expected_not_found__.foo from 10.249.77.96
dnsmasq[27789]: forwarded __cloud_init_expected_not_found__.foo to 127.0.0.53
dnsmasq[27789]: query[AAAA] ubuntu.com from 10.249.77.96
dnsmasq[27789]: forwarded ubuntu.com to 127.0.0.53

tags: added: verification-done-zesty
removed: verification-needed verification-needed-zesty
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (6.4 KiB)

This bug was fixed in the package cloud-init - 0.7.9-233-ge586fe35-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-233-ge586fe35-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/cloud-init.templates: enable Scaleway cloud.
  * debian/cloud-init.templates: enable Aliyun cloud.
  * drop the following cherry picks, now incorporated in snapshot.
    + debian/patches/cpick-5fb49bac-azure-identify-platform...
    + debian/patches/cpick-003c6678-net-remove-systemd-link...
    + debian/patches/cpick-1cd4323b-azure-remove-accidental...
    + debian/patches/cpick-ebc9ecbc-Azure-Add-network-config...
    + debian/patches/cpick-11121fe4-systemd-make-cloud-final...
  * debian/patches/stable-release-no-jsonschema-dep.patch:
    add patch to remove optional dependency on jsonschema.
  * New upstream snapshot.
    - cloudinit.net: add initialize_network_device function and tests
      [Chad Smith]
    - makefile: fix ci-deps-ubuntu target [Chad Smith]
    - tests: adjust locale integration test to parse default locale.
    - tests: remove 'yakkety' from releases as it is EOL.
    - centos: do not package systemd-fsck drop-in.
    - systemd: make systemd-fsck run after cloud-init.service (LP: #1691489)
    - tests: Add initial tests for EC2 and improve a docstring.
    - locale: Do not re-run locale-gen if provided locale is system default.
    - archlinux: fix set hostname usage of write_file. [Joshua Powers]
    - sysconfig: support subnet type of 'manual'.
    - tools/run-centos: make running with no argument show help.
    - Drop rand_str() usage in DNS redirection detection
      [Bob Aman] (LP: #1088611)
    - sysconfig: use MACADDR on bonds/bridges to configure mac_address
      [Ryan Harper]
    - net: eni route rendering missed ipv6 default route config
      [Ryan Harper] (LP: #1701097)
    - sysconfig: enable mtu set per subnet, including ipv6 mtu
      [Ryan Harper]
    - sysconfig: handle manual type subnets [Ryan Harper]
    - sysconfig: fix ipv6 gateway routes [Ryan Harper]
    - sysconfig: fix rendering of bond, bridge and vlan types.
      [Ryan Harper]
    - Templatize systemd unit files for cross distro deltas. [Ryan Harper]
    - sysconfig: ipv6 and default gateway fixes. [Ryan Harper]
    - net: fix renaming of nics to support mac addresses written in upper
      case. (LP: #1705147)
    - tests: fixes for issues uncovered when moving to python 3.6.
    - sysconfig: include GATEWAY value if set in subnet
      [Ryan Harper]
    - Scaleway: add datasource with user and vendor data for Scaleway.
      [Julien Castets]
    - Support comments in content read by load_shell_content.
    - cloudinitlocal fail to run during boot [Hongjiang Zhang]
    - doc: fix disk setup example table_type options [Sandor Zeestraten]
    - tools: Fix exception handling. [Joonas Kylmälä]
    - tests: fix usage of mock in GCE test.
    - test_gce: Fix invalid mock of platform_reports_gce to return False
      [Chad Smith]
    - test: fix incorrect keyid for apt repository. [Joshua Powers]
    - tests: Update version of pylxd [Joshua Powers]
    - write_files: Remove log from helper function signatures.
      [Andrew Jorgensen]
    - doc: document...

Read more...

Changed in cloud-init (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (6.4 KiB)

This bug was fixed in the package cloud-init - 0.7.9-233-ge586fe35-0ubuntu1~17.04.1

---------------
cloud-init (0.7.9-233-ge586fe35-0ubuntu1~17.04.1) zesty; urgency=medium

  * debian/cloud-init.templates: enable Scaleway cloud.
  * debian/cloud-init.templates: enable Aliyun cloud.
  * drop the following cherry picks, now incorporated in snapshot.
    + debian/patches/cpick-5fb49bac-azure-identify-platform...
    + debian/patches/cpick-003c6678-net-remove-systemd-link...
    + debian/patches/cpick-1cd4323b-azure-remove-accidental...
    + debian/patches/cpick-ebc9ecbc-Azure-Add-network-config...
    + debian/patches/cpick-11121fe4-systemd-make-cloud-final...
  * debian/patches/stable-release-no-jsonschema-dep.patch:
    add patch to remove optional dependency on jsonschema.
  * New upstream snapshot.
    - cloudinit.net: add initialize_network_device function and tests
      [Chad Smith]
    - makefile: fix ci-deps-ubuntu target [Chad Smith]
    - tests: adjust locale integration test to parse default locale.
    - tests: remove 'yakkety' from releases as it is EOL.
    - centos: do not package systemd-fsck drop-in.
    - systemd: make systemd-fsck run after cloud-init.service (LP: #1691489)
    - tests: Add initial tests for EC2 and improve a docstring.
    - locale: Do not re-run locale-gen if provided locale is system default.
    - archlinux: fix set hostname usage of write_file. [Joshua Powers]
    - sysconfig: support subnet type of 'manual'.
    - tools/run-centos: make running with no argument show help.
    - Drop rand_str() usage in DNS redirection detection
      [Bob Aman] (LP: #1088611)
    - sysconfig: use MACADDR on bonds/bridges to configure mac_address
      [Ryan Harper]
    - net: eni route rendering missed ipv6 default route config
      [Ryan Harper] (LP: #1701097)
    - sysconfig: enable mtu set per subnet, including ipv6 mtu
      [Ryan Harper]
    - sysconfig: handle manual type subnets [Ryan Harper]
    - sysconfig: fix ipv6 gateway routes [Ryan Harper]
    - sysconfig: fix rendering of bond, bridge and vlan types.
      [Ryan Harper]
    - Templatize systemd unit files for cross distro deltas. [Ryan Harper]
    - sysconfig: ipv6 and default gateway fixes. [Ryan Harper]
    - net: fix renaming of nics to support mac addresses written in upper
      case. (LP: #1705147)
    - tests: fixes for issues uncovered when moving to python 3.6.
    - sysconfig: include GATEWAY value if set in subnet
      [Ryan Harper]
    - Scaleway: add datasource with user and vendor data for Scaleway.
      [Julien Castets]
    - Support comments in content read by load_shell_content.
    - cloudinitlocal fail to run during boot [Hongjiang Zhang]
    - doc: fix disk setup example table_type options [Sandor Zeestraten]
    - tools: Fix exception handling. [Joonas Kylmälä]
    - tests: fix usage of mock in GCE test.
    - test_gce: Fix invalid mock of platform_reports_gce to return False
      [Chad Smith]
    - test: fix incorrect keyid for apt repository. [Joshua Powers]
    - tests: Update version of pylxd [Joshua Powers]
    - write_files: Remove log from helper function signatures.
      [Andrew Jorgensen]
    - doc: document the cmdli...

Read more...

Changed in cloud-init (Ubuntu Zesty):
status: Fix Committed → Fix Released
Revision history for this message
Scott Moser (smoser) wrote : Fixed in Cloud-init 17.1

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
Revision history for this message
James Falcon (falcojr) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.