[OSSA 2014-039] Maliciously crafted dns_nameservers will crash neutron (CVE-2014-7821)

[Jason Meridth]

The following request body will crash neutron nodes.

{"subnet": {"network_id": "2aeb163a-a415-4568-bb9e-9c0ac93d54e4", "ip_version": 4,
"cidr": "192.168.1.3/16",
"dns_nameservers": ["111111111111111111111111111111111111111111111111111111111111"]}}

Even strace stops logging.

[Tristan Cacqueray (tristan-cacqueray)]

Thanks for the report, the OSSA task is set to incomplete, pending additional details from security reviewer.

[Jeremy Stanley (fungi)]

Jason, in what version(s) of Neutron was this observed? Just in the master/release candidate branches or Icehouse as well?

[Jason Meridth (jmeridth)]

Jeremy, the last commit from upstream that we have is https://github.com/openstack/neutron/commit/3b46f2ca28fedc3de6769a0a5dd48b19ab900ce0

which also has the 2014.2.rc1

[Jeremy Stanley (fungi)]

A bit of spelunking shows that the HOSTNAME_PATTERN expression was introduced to Quantum (now Neutron) during the Grizzly development cycle with change https://review.openstack.org/14219 and backported to Folsom (in time its first stable point release) with https://review.openstack.org/16085 in an effort to fix bug 1062046. It has never been adjusted since, so this vulnerability potentially stretches back to 2012.2.1.

Looks like we probably need a stable/icehouse series task for this bug.

[Jeremy Stanley (fungi)]

I've also subscribed Michael Xin, as he just reported an identical bug I've marked as a duplicate.

[Jason Meridth (jmeridth)]

Jeremy,
Also can you add/change that the bug was discovered by Henry Yamauchi, Charles Neill and Michael Xin (our security QE team). I was just the one filling the bug and attaching patch from our Neutron developers. Hope it is possible. Thank you for any help.

[Michael Xin (michael-xin)]

Jason, Thanks for your quick action.

[Jeremy Stanley (fungi)]

Jason, absolutely. I'll make sure we credit "Henry Yamauchi, Charles Neill and Michael Xin from Rackspace" as the researchers responsible for its discovery on any advisory we publish for this issue. Thanks for the additional detail.

[Jason Meridth (jmeridth)]

Jeremy,
Awesome. Thank you

---
Jason
On Oct 8, 2014 1:45 PM, "Jeremy Stanley" <email address hidden> wrote:

> Jason, absolutely. I'll make sure we credit "Henry Yamauchi, Charles
> Neill and Michael Xin from Rackspace" as the researchers responsible for
> its discovery on any advisory we publish for this issue. Thanks for the
> additional detail.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1378450
>
> Title:
> Maliciously crafted dns_nameservers will crash neutron
>
> Status in OpenStack Neutron (virtual network service):
> New
> Status in OpenStack Security Advisories:
> Incomplete
>
> Bug description:
> The following request body will crash neutron nodes.
>
> {"subnet": {"network_id": "2aeb163a-a415-4568-bb9e-9c0ac93d54e4",
> "ip_version": 4,
> "cidr": "192.168.1.3/16",
> "dns_nameservers":
> ["111111111111111111111111111111111111111111111111111111111111"]}}
>
> Even strace stops logging.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/neutron/+bug/1378450/+subscriptions
>

[Kyle Mestery (mestery)]

I can confirm that this issue exists, and also that the supplied patch fixes the issue.

[Jason Meridth (jmeridth)]

Kyle:
What is the process from here? Does someone else apply our suggested patch or do we need to create a gerrit review? Thank you guys for helping us learn this process.

[Jeremy Stanley (fungi)]

Neutron core security reviewers need to review the patches attached to this bug and/or propose some new attachments. At the same time the vulnerability management team drafts an impact description, requests a CVE with that and schedules a disclosure date with sufficient time to warn downstream consumers and provide advance copies of the accepted patches. On the disclosure date the patches get pushed into Gerrit, rapidly approved and at the same time an advisory is published detailing the vulnerability. https://wiki.openstack.org/wiki/Vulnerability_Management

[Tristan Cacqueray (tristan-cacqueray)]

Here is the impact description draft #1:

Title: Neutron DoS through invalid DNS configuration
Reporter: Henry Yamauchi, Charles Neill and Michael Xin (Rackspace)
Products: Neutron
Versions: up to 2014.2

Description:
Henry Yamauchi, Charles Neill and Michael Xin from Rackspace reported a vulnerability in Neutron. By configuring a maliciously crafted dns_nameservers an authenticated user may crash Neutron service resulting in a denial of service attack. All Neutron setups are affected.

[Tristan Cacqueray (tristan-cacqueray)]

@neutron-coresec Can someone please review the proposed fix ?

[Tristan Cacqueray (tristan-cacqueray)]

Doing a run_tests.sh here on a freshly installed "Ubuntu 14.04.1 LTS" with a stable/juno devstack, it fails with:

neutron.tests.unit.db.test_l3_ha_db.L3HAGetSyncDataTestCase.test_l3_agent_routers_query_interface
neutron.tests.unit.db.test_l3_ha_db.L3HAGetSyncDataTestCase.test_l3_agent_routers_query_interface ... FAIL
neutron.tests.unit.db.test_l3_ha_db.L3HATestCase.test_create_ha_interfaces_binding_failure_rolls_back_ports
neutron.tests.unit.db.test_l3_ha_db.L3HATestCase.test_create_ha_interfaces_binding_failure_rolls_back_ports ... FAIL

======================================================================
FAIL: process-returncode
process-returncode
----------------------------------------------------------------------
_StringException: returncode 1

======================================================================
FAIL: process-returncode
process-returncode
----------------------------------------------------------------------
_StringException: returncode 1

======================================================================
FAIL: neutron.tests.unit.db.test_l3_ha_db.L3HAGetSyncDataTestCase.test_l3_agent_routers_query_interface
neutron.tests.unit.db.test_l3_ha_db.L3HAGetSyncDataTestCase.test_l3_agent_routers_query_interface
----------------------------------------------------------------------
_StringException

======================================================================
FAIL: neutron.tests.unit.db.test_l3_ha_db.L3HATestCase.test_create_ha_interfaces_binding_failure_rolls_back_ports
neutron.tests.unit.db.test_l3_ha_db.L3HATestCase.test_create_ha_interfaces_binding_failure_rolls_back_ports
----------------------------------------------------------------------
_StringException

----------------------------------------------------------------------
Ran 1709 tests in 512.654s

FAILED (failures=4, skipped=75)

[Tristan Cacqueray (tristan-cacqueray)]

Oups nevermind, those errors also happen without the patch...

[Grant Murphy (gmurphy)]

Impact description in #14 lgtm

+1

[Kyle Mestery (mestery)]

Impact description in #14 LGTM to me as well. And I confirmed the attached patches fix the issue.

[Tristan Cacqueray (tristan-cacqueray)]

@jmeridth && @mestery: Can you check if the patch is also effective on Icehouse (it applies and tests seems ok).

[Salvatore Orlando (salvatore-orlando)]

I am happy with the proposed patch and impact description.

I have verified the attached patch fixed the patch in Icehouse as well.

[Thierry Carrez (ttx)]

Proposed public disclosure date/time:
2014-11-19, 1500UTC

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/135616

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/135617

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/135618

[OpenStack Infra (hudson-openstack)]

Change abandoned by Tristan Cacqueray (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/135618
Reason: Will resubmit with correct changeId...

[OpenStack Infra (hudson-openstack)]

Change abandoned by Tristan Cacqueray (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/135617
Reason: Will resubmit with correct changeId...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/135623

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/135624

[Dave Walker (davewalker)]

It would seem to me to be a good idea to include the CVE and OSSA reference in the commit message, at least for stable/*.

Thanks

[Jeremy Stanley (fungi)]

In the past the VMT has merely relied on the closes-bug header in the commit message, but I'll engage the others on the possibility of amending this. Note however that the VMT often is not the source of stable backport uploads, so this would be a requirement the stable branch managers would need to impose on backport change authors. The VMT can document it as a recommended guideline but can't do much to enforce it directly. Also, it could have the potential to further delay landing stable branch security fixes if encumbered with too much additional process beyond that which is strictly necessary to accomplish the intended task.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/135616
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1681f62ec91b6c3705a14393815542dc1746de71
Submitter: Jenkins
Branch: master

commit 1681f62ec91b6c3705a14393815542dc1746de71
Author: John Perkins <email address hidden>
Date: Mon Oct 6 16:24:57 2014 -0500

    Fix hostname regex pattern

    Current hostname_pattern regex complexity grows exponentially
    when given a string of just digits, which can be exploited to
    cause neutron-server to freeze.

    Change-Id: I886c6d883a9cb0acd9908495eec50bf0411d8ba8
    Closes-bug: #1378450

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/135623
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ad6fefcb4d4068b46b69284e277df6ab2ee30105
Submitter: Jenkins
Branch: stable/juno

commit ad6fefcb4d4068b46b69284e277df6ab2ee30105
Author: John Perkins <email address hidden>
Date: Mon Oct 6 16:24:57 2014 -0500

    Fix hostname regex pattern

    Current hostname_pattern regex complexity grows exponentially
    when given a string of just digits, which can be exploited to
    cause neutron-server to freeze.

    Change-Id: I886c6d883a9cb0acd9908495eec50bf0411d8ba8
    Closes-bug: #1378450

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/136238

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/135624
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ab7ea069de5cecf1c26af50996a26e1a7f86def4
Submitter: Jenkins
Branch: stable/icehouse

commit ab7ea069de5cecf1c26af50996a26e1a7f86def4
Author: John Perkins <email address hidden>
Date: Mon Oct 6 16:24:57 2014 -0500

    Fix hostname regex pattern

    Current hostname_pattern regex complexity grows exponentially
    when given a string of just digits, which can be exploited to
    cause neutron-server to freeze.

    Change-Id: I886c6d883a9cb0acd9908495eec50bf0411d8ba8
    Closes-bug: #1378450

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140153
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=7686508a2b03e20f6a032c29d3bfc1f6df107ec6
Submitter: Jenkins
Branch: master

commit 7686508a2b03e20f6a032c29d3bfc1f6df107ec6
Author: Tristan Cacqueray <email address hidden>
Date: Thu Jan 15 20:40:22 2015 +0000

    Amend OSSA-2014-039 for ERRATA1

    Related-Bug: #1378450
    Change-Id: I4732ac1431ba122e8f2f330711156e06ee0a8047

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/136238
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2794bb89d664355ae1194a0b1f8346c1538caef8
Submitter: Jenkins
Branch: master

commit 2794bb89d664355ae1194a0b1f8346c1538caef8
Author: YAMAMOTO Takashi <email address hidden>
Date: Fri Nov 21 14:16:03 2014 +0900

    attributes: Additional IP address validation

    Introduce an additional IP address validation instead of assuming
    that netaddr provides it. Namely, it ensures that an address
    either has ':' (IPv6) or 3 periods like 'xx.xx.xx.xx'. (IPv4)

    The "'1' * 59" test case recently introduced by
    commit 1681f62ec91b6c3705a14393815542dc1746de71 fails on
    some platforms because it's considered a valid address by
    their inet_aton. Examples of such platforms: NetBSD, OS X

    While one might argue it's a fault of the platforms, this is
    a historical behavior which is probably too late to change there.

    (The breakage has been hidden by later UT changes in
    commit 35662d07628452d14306f5197871ad64f6396ff3 .
    This commit includes a UT change to uncover the problem again.)

    Closes-Bug: #1394867
    Related-Bug: #1378450
    Change-Id: Ibe02f8b7c4d437bf7abbe7304ca138bdcf4bfdb9

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/juno
Review: https://review.openstack.org/161930

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/161930
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d329c221c37714f1ce306918de8367dd948ec220
Submitter: Jenkins
Branch: stable/juno

commit d329c221c37714f1ce306918de8367dd948ec220
Author: YAMAMOTO Takashi <email address hidden>
Date: Fri Nov 21 14:16:03 2014 +0900

    attributes: Additional IP address validation

    Introduce an additional IP address validation instead of assuming
    that netaddr provides it. Namely, it ensures that an address
    either has ':' (IPv6) or 3 periods like 'xx.xx.xx.xx'. (IPv4)

    The "'1' * 59" test case recently introduced by
    commit 1681f62ec91b6c3705a14393815542dc1746de71 fails on
    some platforms because it's considered a valid address by
    their inet_aton. Examples of such platforms: NetBSD, OS X

    While one might argue it's a fault of the platforms, this is
    a historical behavior which is probably too late to change there.

    (The breakage has been hidden by later UT changes in
    commit 35662d07628452d14306f5197871ad64f6396ff3 .
    This commit includes a UT change to uncover the problem again.)

    (cherry-picked from 2794bb89d664355ae1194a0b1f8346c1538caef8)
    Closes-Bug: #1394867
    Related-Bug: #1378450
    Change-Id: Ibe02f8b7c4d437bf7abbe7304ca138bdcf4bfdb9