[OSSA 2014-014] neutron allows security group rules with invalid cidrs, resulting in broken iptables rules (breaking iptables-restore) (CVE-2014-0187)

Bug #1300785 reported by Christoph Thiel on 2014-04-01
260
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Security Advisory
High
Tristan Cacqueray
neutron
Undecided
Aaron Rosen
Havana
Undecided
Unassigned
Icehouse
Undecided
Unassigned

Bug Description

This bug is already reported in https://bugs.launchpad.net/neutron/+bug/1255338, but no security impact is discussed in that bug so far. We have been hitting the same issue in our cloud recently, and found that it is basically breaking quantum-plugin-openvswitch-agent / neutron-plugin-openvswitch-agent with errors like this:

security group was created with:

quantum security-group-rule-create default --direction egress --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix /32

    ERROR [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Error in agent event loop
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py", line 700, in rpc_loop
      sync = self.process_network_ports(port_info)
    File "/usr/lib/python2.7/dist-packages/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py", line 655, in process_network_ports
      resync_a = self.treat_devices_added(port_info['added'])
    File "/usr/lib/python2.7/dist-packages/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py", line 601, in treat_devices_added
      self.sg_agent.prepare_devices_filter(devices)
    File "/usr/lib/python2.7/dist-packages/quantum/agent/securitygroups_rpc.py", line 114, in prepare_devices_filter
      self.firewall.prepare_port_filter(device)
    File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
      self.gen.next()
    File "/usr/lib/python2.7/dist-packages/quantum/agent/firewall.py", line 107, in defer_apply
      self.filter_defer_apply_off()
    File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_firewall.py", line 284, in filter_defer_apply_off
      self.iptables.defer_apply_off()
    File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_manager.py", line 304, in defer_apply_off
      self._apply()
    File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/lockutils.py", line 228, in inner
      retval = f(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_manager.py", line 340, in _apply
      root_helper=self.root_helper)
    File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py", line 61, in execute
      raise RuntimeError(m)
  RuntimeError:
  Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'iptables-restore']
  Exit code: 2
  Stdout: ''
  Stderr: "iptables-restore v1.4.12: host/network `' not found\nError occurred at line: 391\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n"

Our operations team is telling me, that just removing the broken role didn't help, but "restart quantum-plugin-openvswitch-agent" was needed on all affected nodes.

IMHO, this is an issue the vulnerability management team should consider. (We have been seeing this on stable/grizzly, but havana and upcoming icehouse are also affected.)

Thierry Carrez (ttx) on 2014-04-01
Changed in ossa:
status: New → Incomplete
Thierry Carrez (ttx) wrote :

I think that's a valid DoS vector, due to its efficiency.

Not the first time an invalid iptables rule can be passed and wreck things -- maybe they should consider testing them in a sandbox before applying them, rather than try to sanitize them and then apply them blindly...

Changed in ossa:
importance: Undecided → High
status: Incomplete → Confirmed
Thierry Carrez (ttx) wrote :

Question: does it just break VMs that apply the rule, or the whole host ?

Christoph Thiel (cthiel-suse) wrote :

IIUC, it depends on your configuration if it affects the whole host, ore just a namespace / tenant. Someone with deeper understanding of the agent/linux/iptables_manager.py should give his view on the topic :)

The scope of the potential damage can vary as suggested by Cristoph.
It's not really a DoS, more an issue where security groups could be de-enforced without a user knowing it, since no information would be reflected in ports' statuses.

I agree about the security concern, but since the original bug is open, a security embargo is hardly enforceable now.

Aaron Rosen (arosen) wrote :

I agree this is a security concern. It prevents security group rules from being applied on a host after one installs that rule. I just reproduced this issue and tested this patch: https://review.openstack.org/#/c/59212/16 which fixes the issue. I think we should focus on getting that merged asap. I'm wondering if there should also be a security CVE released though so people are aware of the issue downstream.

Changed in neutron:
status: New → Confirmed
assignee: nobody → Aaron Rosen (arosen)
Thierry Carrez (ttx) wrote :

Even if the security aspect is not clearly described in the patch, we should assume that the patch is public and issue a CVE/OSSA on this ASAP. We can keep this bug private a bit more as we work out the details.

We'll need backports for this on stable/icehouse. Is Havana also affected ?

Changed in neutron:
status: Confirmed → Fix Committed
Christoph Thiel (cthiel-suse) wrote :

Yes, havana is also affected. (As is grizzly, which is no longer an issue.)

I guess the OSSA draft is going to be put into this bug for review?

@Christoph Yes, here it is.

I have a question, is the iptables-restore issue will also prevent former rules from being re-applied or will it break at the
 time the invalid rule is actually being applied ?
In such a case, the impact desc would need a note about that in case of a host reboot.

Impact description draft #1:

Title: Neutron security group DoS through invalid CIDR
Reporter: Stephen Ma (HP)
Products: Neutron
Versions: 2013.1 to 2013.2.3, and 2014.1 versions

Description:
Stephen Ma from Hewlett Packard reported a vulnerability in Neutron security group. By creating a security group rule with an invalid CIDR, an authenticated user may break openvswitch-agent process, preventing further rules from being applied on the host. Note: removal of the faulty rule is not enough, the openvswitch-agent must be restarted.
All Neutron setups are affected.

Changed in ossa:
assignee: nobody → Tristan Cacqueray (tristan-cacqueray)
Thierry Carrez (ttx) wrote :

Comments:
* would just say "Versions: 2013.1 to 2013.2.3, and 2014.1" (since 2014.1 is the only "2014.1 version" that will be affected)
* reported a vulnerability in Neutron security *groups*
* I would say "bypass" rather than DoS, since it allows to evade security groups rather than deny service.

Christoph Thiel (cthiel-suse) wrote :

I understand Stephan Ma first reported it, but didn't point out security issue. I wouldn't mind being mentioned as well, i.e. Christoph Thiel (Deutsche Telekom) ;)

In addition to that I would recommend noting, that only Neutron setups are affected that are using the OpenVswitch.

Thanks for your comments!

Impact description draft #2:

Title: Neutron security groups bypass through invalid CIDR
Reporters: Stephen Ma (HP) and Christoph Thiel (Deutsche Telekom)
Products: Neutron
Versions: 2013.1 to 2013.2.3, and 2014.1

Description:
Stephen Ma from Hewlett Packard and Christoph Thiel from Deutsche Telekom reported a vulnerability in Neutron security groups. By creating a security group rule with an invalid CIDR, an authenticated user may break openvswitch-agent process, preventing further rules from being applied on the host. Note: removal of the faulty rule is not enough, the openvswitch-agent must be restarted. All Neutron setups using Open vSwitch are affected.

Thierry Carrez (ttx) wrote :

Looks good to me

summary: neutron allows security group rules with invalid cidrs, resulting in
- broken iptables rules (breaking iptables-restore)
+ broken iptables rules (breaking iptables-restore) (CVE-2014-0187)
Thierry Carrez (ttx) on 2014-04-22
Changed in ossa:
status: Confirmed → In Progress
summary: - neutron allows security group rules with invalid cidrs, resulting in
- broken iptables rules (breaking iptables-restore) (CVE-2014-0187)
+ [OSSA 2014-014] neutron allows security group rules with invalid cidrs,
+ resulting in broken iptables rules (breaking iptables-restore)
+ (CVE-2014-0187)
information type: Private Security → Public Security
Thierry Carrez (ttx) on 2014-04-22
Changed in ossa:
status: In Progress → Fix Committed
Thierry Carrez (ttx) wrote :

Still needs to be merged in stable/havana: https://review.openstack.org/#/c/88057/

Jeremy Stanley (fungi) wrote :

stable/havana change merged April 30 at 02:36 UTC.

Thierry Carrez (ttx) on 2014-05-06
Changed in ossa:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-06-12
Changed in neutron:
milestone: none → juno-1
status: Fix Committed → Fix Released
Jamie Strandboge (jdstrand) wrote :

FYI, I tried to backport the havana patch to Ubuntu 13.10 and have 3 new test failures as can be see in:
https://launchpadlibrarian.net/177272442/buildlog_ubuntu-saucy-i386.neutron_1%3A2013.2.3-0ubuntu1.3_UPLOADING.txt.gz

as opposed to:
https://launchpadlibrarian.net/174614850/buildlog_ubuntu-saucy-i386.neutron_1%3A2013.2.3-0ubuntu1.1_UPLOADING.txt.gz

Specifically:
FAIL: neutron.tests.unit.linuxbridge.test_lb_security_group.TestLinuxBridgeSecurityGroups.test_create_security_group_rule_invalid_ip_prefix
FAIL: neutron.tests.unit.linuxbridge.test_lb_security_group.TestLinuxBridgeSecurityGroupsXML.test_create_security_group_rule_invalid_ip_prefix
FAIL: neutron.tests.unit.midonet.test_midonet_plugin.TestMidonetSecurityGroup.test_create_security_group_rule_invalid_ip_prefix

I applied patchset 10 from https://review.openstack.org/#/c/88057/. Am I missing another patchset?

(There are some different failures in neutron.tests.unit.ml2.drivers.test_cisco_mech.* but I think this may be a problem in the testsuite.)

Jamie Strandboge (jdstrand) wrote :

Nevermind, I just now found https://review.openstack.org/#/c/88058/.

Thierry Carrez (ttx) on 2014-10-16
Changed in neutron:
milestone: juno-1 → 2014.2
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers