non-IP ethertypes are permitted with iptables_hybrid firewall driver

[Nate Johnston]

Background of the Issue
=======================
Customers expect that when they set Neutron security group rules such that all traffic is blocked that all traffic is in fact blocked by the Neutron firewall driver. Neutron has multiple firewall drivers available, but in certain distributions (RHOSP for example) the default is the iptables_hybrid driver.

The iptables_hybrid driver was implemented using the iptables toolset. This means that it is very effective at filtering IP traffic. But IPv4 and IPv6 are only 2 of the available ethernet types (ethertypes). There are other types of traffic that ride over ethernet that are not IP [1], and the iptables_hybrid firewall driver does not inspect or control them.

First contact with this issue: A customer alerted us to an issue because they were transitioning from the iptables_hybrid firewall driver to the OpenVSwitch firewall driver (ovsfw). When they made the switch they noticed that all of their InfiniBand traffic was being blocked, because it uses a non-IP ethertype (0x4008). The fact that with iptables_hybrid the traffic passed without any specific control enabling it is the actual issue, we realized as we evaluated the case.

Analysis of the Vulnerability
=============================
Non-standard ethertypes are layer 3 protocols that are transmitted within the layer 2 ethernet frame. This means that non-IP ethertypes cannot use IP-based routing to gain egress from the layer 2 domain. When the iptables_hybrid firewall is in use:

- In clouds using self-service networking, instances on tenant networks depend on either Floating IPs or Neutron Routers for egress from their networks. Since both of those are IP-specific concepts, traffic with custom ethertypes would have no native means to egress the network.
- In clouds using provider networking, instances depend on the provider network hardware to route traffic outside of the layer 2 domain. Some ethertypes may be routable by network hardware - some routers understand InfiniBand or FCoE natively, for example, and will serve the routing implementation of those protocols. We are not knowdgeable about the variety of routers out there and what they may support; it is entirely possible that legacy support for older protocols that have different ethertypes like IPX or PPPoE may exist in Cisco, Juniper, or other code..
- In ether case, if unauthorized access to a host is obtained by an intruder, then other hosts on the same L2 domain may be succeptible to the use of traffic travelling over other ethertypes to bypass hostbased firewalling. Exfiltration from that network from the compromised host is still subject to the restrictions in the above 2 bullet points.

To succinctly summarize the above: because non-IP based ethertypes bypass all hostbased controls, only network-based controls restrict them. The best network-based control is the lack of support for routing out of the L2 domain, which is mostly but not always true.

Proposed strategy
=================
We propose that a response to this issue has two parts.

1. The iptables_hybrid firewall is changed to deny ethertypes other than IPv4, IPv6, and ARP by default. Those 3 protocols are not covered because they are handled through other mechanisms in the Neutron security group system. We propose that the iptables_hybrid control to drop other ethertypes by default be enabled by default but configurable, so that an operator could disable it either within the upgrade context to maintain compatibility with conditions pre-upgrade, as an issue triage step, or as part of a looser security policy.

2. The Neutron security group system must be enhanced with a control that allows specific ethertypes to be permitted. In upstream Neutron master development this should be an addition to the existing Security Groups API; for older Neutron versions this will be a configuration file option. This configuration file option should be manageable through Director/TripleO.

This issue has been discussed in the upstream Neutron community [2] and there is consensus on proposed points #1 and #2.

[1] https://en.wikipedia.org/wiki/EtherType#Examples
[2] http://eavesdrop.openstack.org/meetings/neutron_drivers/2019/neutron_drivers.2019-06-28-14.00.log.html

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/673692
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=24aef1398b5b69967696aa8cb6598ab1be37c626
Submitter: Zuul
Branch: master

commit 24aef1398b5b69967696aa8cb6598ab1be37c626
Author: Nate Johnston <email address hidden>
Date: Tue Jul 30 17:15:24 2019 -0400

    Ensure that extended ethertypes still work by name

    When the change to add support for extended ethertypes was introduced,
    one scenario for backwards compatibility was not addressed in the
    validator: using the canonical string names for ethertypes. This was
    previously supported in the original implementation, using 'IPv4' or
    'IPv6' for example.

    Since the list of canonical ethertype names is available from os-ken we
    consume it from there, which adds an additional dependency for
    neutron-lib.

    Related-Bug: #1838473
    Change-Id: I53c6538dfbeea9691d95c6c555f0c56ae13d1a33

[Summer Long (slong-g)]

We are treating this as a security issue at Red Hat. Please let us know if you will not be assigning a CVE so that we can.

[Jeremy Stanley (fungi)]

The OpenStack VMT generally requests a CVE if we're going to be issuing an advisory, which we only do if there are backportable fixes available. So far there is no fix under review, so it's impossible to know at this point.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/670203
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/670203
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Tobias Urdin (tobias-urdin)]

Can this be closed as solved by [1]?

[1] https://review.opendev.org/673692

[Jeremy Stanley (fungi)]

The report asserts that allowing Ethertypes other than IPv4, IPv6 and ARP by default when iptables_hybrid is used is a security risk. It's not immediately clear to me that https://review.opendev.org/673692 blocks them. It looks like the hope was that https://review.opendev.org/670203 would do so, but it hasn't been finished for several years.

[Jeremy Stanley (fungi)]

Based on discussion between members of the VMT and others in the OpenStack Security SIG during the 2023.1 PTG, it appears that any fixes will depend on non-backportable default behavior or configuration changes. An OSSN might be warranted, but we wouldn't likely issue a security advisory about this, class B1 in our report taxonomy: https://security.openstack.org/vmt-process.html#report-taxonomy