[RFE] Allow/deny custom ethertypes in security groups

[Dan Sneddon]

Some operators need to allow/deny custom Ethertypes for applications which use their own non-IP traffic (such as for clustering applications). The Security Group API only handles specifying behavior within the IP protocol. With the firewall reference implementation (OVS Firewall) anything other than IPv4 and IPv6 is subject to the default deny. This means OpenStack customers have no options to use OpenStack to permit protocols that use separate ethertypes like InfiniBand and FCoE.

We propose adding to the Security Group API the capability to specify standard security group behaviors (allow, deny) for custom ethertypes, with the aim of implementing these controls in the OVS firewall.

[Miguel Lavalle (minsel)]

If I look at the security group rules API attributes (https://developer.openstack.org/api-ref/network/v2/index.html?expanded=create-security-group-rule-detail#id416), clearly a several of them exist under the assumption of IP protocols: protocol, port ranges, remote ip prefix. If we allow other ethertypes,

1) Won't many of them require additional specific attributes in the API to allow the user to specify the behavior she is looking for?

2) Or can we come up with a standard set of attributes that will apply to all those ethertypes?

3) Should this even be approached from the security groups API perspective? What you are talking about here is link layer, whereas the security groups API is at the IP level, at least as it is today

Also you indicate that the aim is "implementing these controls in the OVS firewall". Shouldn't we be discussing in this RFE the entire thing and not just the addition of ethertypes?

[Nate Johnston (nate-johnston)]

I'm not sure that there are a set of attributes that would properly cover all the variations in the various ethertypes that are possible options here, and I am not possessed of sufficient expertise in them to comment authoritatively. The real objective is to establish a set of ethertypes that are permitted to pass through ingress/egress filtering. It might be as uncomplicated as that: a whitelist of ethertypes would work perfectly well.

The iptables_hybrid firewall takes the approach you indicate. Custom ethertypes operate at a lower level than IP and as such no firewall inspection or control is done to them. If they were to be handled it would need to be ebtables that would be configured to control them. But the OVS firewall does operate at the link level and does block ethertypes other than those allowed by security groups.

Use case: Customer is running an application using InfiniBand (ethertype 0x4008) in OpenStack, and that OpenStack transitions from iptables_hybrid to ovs firewall. The Infiniband traffic is blocked by the ovs firewall, and at present the Neutron API offers no methodology to unblock it.

[Miguel Lavalle (minsel)]

Let's talk about it in the drivers meeting

[YAMAMOTO Takashi (yamamoto)]

some relevant bugs:
https://bugs.launchpad.net/neutron/+bug/1622753
https://bugs.launchpad.net/neutron/+bug/1508155

[YAMAMOTO Takashi (yamamoto)]

wrt Nate's point in comment #2, i'm inclined to consider it as a bug in ovs-fw.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/668224

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/668242

[Miguel Lavalle (minsel)]

This RFE is approved. The implementation will include the following:

1) Custom ethertypes traffic will be not allowed by default. User has to add rules to the security group to enable them
2) There will be a new API extension
3) A backportable lightweight approach will be implemented using a config option as outlined here: https://review.opendev.org/#/c/668224/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668224
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9ea6a616658268b833ccb787c181962898863058
Submitter: Zuul
Branch: master

commit 9ea6a616658268b833ccb787c181962898863058
Author: Nate Johnston <email address hidden>
Date: Fri Jun 28 14:03:28 2019 -0400

    Add custom ethertype processing

    The OVS Firewall blocks traffic that does not have either the IPv4 or
    IPv6 ethertypes at present. This is a behavior change compared to the
    iptables_hybrid firewall, which only operates on IP packets and thus
    does not address other ethertypes.

    This is a lightweight change that sets a configuration option in the
    neutron openvswitch agent configuration file for permitted ethertypes
    and then ensures that the requested ethertypes are permitted on
    initialization. This addresses the security and usability concerns on
    both master and stable branches while a full-fledged extension to the
    security groups API is considered.

    Change-Id: Ide78b0b90cf6d6069ce3787fc60766be52062da0
    Related-Bug: #1832758

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/668423

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/668424

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/668425

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668423
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0be471486bf957971394990ac3b57f83aadba674
Submitter: Zuul
Branch: stable/stein

commit 0be471486bf957971394990ac3b57f83aadba674
Author: Nate Johnston <email address hidden>
Date: Fri Jun 28 14:03:28 2019 -0400

    Add custom ethertype processing

    The OVS Firewall blocks traffic that does not have either the IPv4 or
    IPv6 ethertypes at present. This is a behavior change compared to the
    iptables_hybrid firewall, which only operates on IP packets and thus
    does not address other ethertypes.

    This is a lightweight change that sets a configuration option in the
    neutron openvswitch agent configuration file for permitted ethertypes
    and then ensures that the requested ethertypes are permitted on
    initialization. This addresses the security and usability concerns on
    both master and stable branches while a full-fledged extension to the
    security groups API is considered.

    Change-Id: Ide78b0b90cf6d6069ce3787fc60766be52062da0
    Related-Bug: #1832758
    (cherry picked from commit 9ea6a616658268b833ccb787c181962898863058)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668425
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=51d71097c99ad9ad67fca1530be69101a70e411e
Submitter: Zuul
Branch: stable/queens

commit 51d71097c99ad9ad67fca1530be69101a70e411e
Author: Nate Johnston <email address hidden>
Date: Fri Jun 28 14:03:28 2019 -0400

    Add custom ethertype processing

    The OVS Firewall blocks traffic that does not have either the IPv4 or
    IPv6 ethertypes at present. This is a behavior change compared to the
    iptables_hybrid firewall, which only operates on IP packets and thus
    does not address other ethertypes.

    This is a lightweight change that sets a configuration option in the
    neutron openvswitch agent configuration file for permitted ethertypes
    and then ensures that the requested ethertypes are permitted on
    initialization. This addresses the security and usability concerns on
    both master and stable branches while a full-fledged extension to the
    security groups API is considered.

    Change-Id: Ide78b0b90cf6d6069ce3787fc60766be52062da0
    Related-Bug: #1832758
    (cherry picked from commit 9ea6a616658268b833ccb787c181962898863058)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668424
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=55a503b4c96181625e60bf3964cd2f10f3af68e8
Submitter: Zuul
Branch: stable/rocky

commit 55a503b4c96181625e60bf3964cd2f10f3af68e8
Author: Nate Johnston <email address hidden>
Date: Fri Jun 28 14:03:28 2019 -0400

    Add custom ethertype processing

    The OVS Firewall blocks traffic that does not have either the IPv4 or
    IPv6 ethertypes at present. This is a behavior change compared to the
    iptables_hybrid firewall, which only operates on IP packets and thus
    does not address other ethertypes.

    This is a lightweight change that sets a configuration option in the
    neutron openvswitch agent configuration file for permitted ethertypes
    and then ensures that the requested ethertypes are permitted on
    initialization. This addresses the security and usability concerns on
    both master and stable branches while a full-fledged extension to the
    security groups API is considered.

    Change-Id: Ide78b0b90cf6d6069ce3787fc60766be52062da0
    Related-Bug: #1832758
    (cherry picked from commit 9ea6a616658268b833ccb787c181962898863058)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/670203

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668242
Committed: https://git.openstack.org/cgit/openstack/neutron-specs/commit/?id=2bbb80d049e27bcbfde5a16bb99e48ab2e89d883
Submitter: Zuul
Branch: master

commit 2bbb80d049e27bcbfde5a16bb99e48ab2e89d883
Author: Nate Johnston <email address hidden>
Date: Fri Jun 28 16:17:17 2019 -0400

    Add spec for custom ethertype feature

    This change adds a specification for the change to add control for
    custom ethertypes.

    Change-Id: I158b1be16fba7b3eb8d53a51625eb3d9b2a26773
    Related-Bug: #1832758

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/678021
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fb859966f79fc0f4fbb2c2ea3c0e6b4c17a9704f
Submitter: Zuul
Branch: master

commit fb859966f79fc0f4fbb2c2ea3c0e6b4c17a9704f
Author: Nate Johnston <email address hidden>
Date: Thu Aug 22 10:13:00 2019 -0400

    OVS flows for custom ethertypes must be on EGRESS

    Some traffic does not work if the OVS flows to permit custom ethertypes
    are not set on the base egress table. If the rule is added to the base
    egress table then both ingress and egress work properly. Also move
    initialization code to the function to initialize egress.

    Related-Bug: #1832758
    Change-Id: Ia312fe75df58723bf41804eec4bd918d223bd60c

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/686215

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/686218

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/686219

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/686215
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9d04e38e8679c3f87c364bc6c9e39b43dfb959b9
Submitter: Zuul
Branch: stable/stein

commit 9d04e38e8679c3f87c364bc6c9e39b43dfb959b9
Author: Nate Johnston <email address hidden>
Date: Thu Aug 22 10:13:00 2019 -0400

    OVS flows for custom ethertypes must be on EGRESS

    Some traffic does not work if the OVS flows to permit custom ethertypes
    are not set on the base egress table. If the rule is added to the base
    egress table then both ingress and egress work properly. Also move
    initialization code to the function to initialize egress.

    Related-Bug: #1832758
    Change-Id: Ia312fe75df58723bf41804eec4bd918d223bd60c
    (cherry picked from commit fb859966f79fc0f4fbb2c2ea3c0e6b4c17a9704f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/686218
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f88d703efe6a29993f0543f229e92201a4d5f743
Submitter: Zuul
Branch: stable/rocky

commit f88d703efe6a29993f0543f229e92201a4d5f743
Author: Nate Johnston <email address hidden>
Date: Thu Aug 22 10:13:00 2019 -0400

    OVS flows for custom ethertypes must be on EGRESS

    Some traffic does not work if the OVS flows to permit custom ethertypes
    are not set on the base egress table. If the rule is added to the base
    egress table then both ingress and egress work properly. Also move
    initialization code to the function to initialize egress.

    Related-Bug: #1832758
    Change-Id: Ia312fe75df58723bf41804eec4bd918d223bd60c
    (cherry picked from commit fb859966f79fc0f4fbb2c2ea3c0e6b4c17a9704f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/686219
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=99eaf3eeee415e7862ce31d0c41668f45e5e406e
Submitter: Zuul
Branch: stable/queens

commit 99eaf3eeee415e7862ce31d0c41668f45e5e406e
Author: Nate Johnston <email address hidden>
Date: Thu Aug 22 10:13:00 2019 -0400

    OVS flows for custom ethertypes must be on EGRESS

    Some traffic does not work if the OVS flows to permit custom ethertypes
    are not set on the base egress table. If the rule is added to the base
    egress table then both ingress and egress work properly. Also move
    initialization code to the function to initialize egress.

    Related-Bug: #1832758
    Change-Id: Ia312fe75df58723bf41804eec4bd918d223bd60c
    (cherry picked from commit fb859966f79fc0f4fbb2c2ea3c0e6b4c17a9704f)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/876563

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/876563
Committed: https://opendev.org/openstack/neutron/commit/008277b8c12d99438951a308b278203fa7a7c3ef
Submitter: "Zuul (22348)"
Branch: master

commit 008277b8c12d99438951a308b278203fa7a7c3ef
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sun Mar 5 22:12:55 2023 +0100

    [OVS] Allow custom ethertype traffic in the ingress table

    This patch is a partial revert of [1], reinstantiating the code merged
    in [2]. This patch is the complementary to [1]: the traffic with
    custom ethertypes is allowed in the ingress processing tables, same
    as [1] is allowing all traffic from the virtual machine ports in this
    host to leave the node. Both, this patch and [1], are bypassing the
    OVS firewall just for the traffic with the configured allowed
    ethertypes and just for/to the local ports and MAC addresses.

    Any other traffic not coming from a local port or with destination
    a local port, will be blocked as is now.

    [1]https://review.opendev.org/c/openstack/neutron/+/678021
    [2]https://review.opendev.org/c/openstack/neutron/+/668224/

    Closes-Bug: #2009221
    Related-Bug: #1832758
    Change-Id: Ib8340d9430b946a446edf80886c49fbac729073c

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/877585

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/877586

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/877605

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/877607

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/877608

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/877585
Committed: https://opendev.org/openstack/neutron/commit/17faa288cedcdb7a365be3105f281cfd957a1464
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 17faa288cedcdb7a365be3105f281cfd957a1464
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sun Mar 5 22:12:55 2023 +0100

    [OVS] Allow custom ethertype traffic in the ingress table

    This patch is a partial revert of [1], reinstantiating the code merged
    in [2]. This patch is the complementary to [1]: the traffic with
    custom ethertypes is allowed in the ingress processing tables, same
    as [1] is allowing all traffic from the virtual machine ports in this
    host to leave the node. Both, this patch and [1], are bypassing the
    OVS firewall just for the traffic with the configured allowed
    ethertypes and just for/to the local ports and MAC addresses.

    Any other traffic not coming from a local port or with destination
    a local port, will be blocked as is now.

    [1]https://review.opendev.org/c/openstack/neutron/+/678021
    [2]https://review.opendev.org/c/openstack/neutron/+/668224/

    Closes-Bug: #2009221
    Related-Bug: #1832758
    Change-Id: Ib8340d9430b946a446edf80886c49fbac729073c
    (cherry picked from commit 008277b8c12d99438951a308b278203fa7a7c3ef)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/877586
Committed: https://opendev.org/openstack/neutron/commit/138a47bfd62252ddea8ff7ccdedff265e99cfb0e
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 138a47bfd62252ddea8ff7ccdedff265e99cfb0e
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sun Mar 5 22:12:55 2023 +0100

    [OVS] Allow custom ethertype traffic in the ingress table

    This patch is a partial revert of [1], reinstantiating the code merged
    in [2]. This patch is the complementary to [1]: the traffic with
    custom ethertypes is allowed in the ingress processing tables, same
    as [1] is allowing all traffic from the virtual machine ports in this
    host to leave the node. Both, this patch and [1], are bypassing the
    OVS firewall just for the traffic with the configured allowed
    ethertypes and just for/to the local ports and MAC addresses.

    Any other traffic not coming from a local port or with destination
    a local port, will be blocked as is now.

    [1]https://review.opendev.org/c/openstack/neutron/+/678021
    [2]https://review.opendev.org/c/openstack/neutron/+/668224/

    Closes-Bug: #2009221
    Related-Bug: #1832758
    Change-Id: Ib8340d9430b946a446edf80886c49fbac729073c
    (cherry picked from commit 008277b8c12d99438951a308b278203fa7a7c3ef)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/877608
Committed: https://opendev.org/openstack/neutron/commit/8c7f3b61f75368f05369785f7931b5134a7e93fa
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 8c7f3b61f75368f05369785f7931b5134a7e93fa
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sun Mar 5 22:12:55 2023 +0100

    [OVS] Allow custom ethertype traffic in the ingress table

    This patch is a partial revert of [1], reinstantiating the code merged
    in [2]. This patch is the complementary to [1]: the traffic with
    custom ethertypes is allowed in the ingress processing tables, same
    as [1] is allowing all traffic from the virtual machine ports in this
    host to leave the node. Both, this patch and [1], are bypassing the
    OVS firewall just for the traffic with the configured allowed
    ethertypes and just for/to the local ports and MAC addresses.

    Any other traffic not coming from a local port or with destination
    a local port, will be blocked as is now.

    [1]https://review.opendev.org/c/openstack/neutron/+/678021
    [2]https://review.opendev.org/c/openstack/neutron/+/668224/

    Conflicts:
           doc/source/admin/config-ovsfwdriver.rst
           neutron/tests/unit/agent/linux/openvswitch_firewall/test_firewall.py

    Closes-Bug: #2009221
    Related-Bug: #1832758
    Change-Id: Ib8340d9430b946a446edf80886c49fbac729073c
    (cherry picked from commit 008277b8c12d99438951a308b278203fa7a7c3ef)
    (cherry picked from commit 5026d805fe01aaf237081c606f1d1bf87bbff6d4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/877605
Committed: https://opendev.org/openstack/neutron/commit/1e244c57c51f02ac9e10e91740da67fb88bdd9ec
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 1e244c57c51f02ac9e10e91740da67fb88bdd9ec
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sun Mar 5 22:12:55 2023 +0100

    [OVS] Allow custom ethertype traffic in the ingress table

    This patch is a partial revert of [1], reinstantiating the code merged
    in [2]. This patch is the complementary to [1]: the traffic with
    custom ethertypes is allowed in the ingress processing tables, same
    as [1] is allowing all traffic from the virtual machine ports in this
    host to leave the node. Both, this patch and [1], are bypassing the
    OVS firewall just for the traffic with the configured allowed
    ethertypes and just for/to the local ports and MAC addresses.

    Any other traffic not coming from a local port or with destination
    a local port, will be blocked as is now.

    [1]https://review.opendev.org/c/openstack/neutron/+/678021
    [2]https://review.opendev.org/c/openstack/neutron/+/668224/

    Conflicts:
           doc/source/admin/config-ovsfwdriver.rst
           neutron/tests/unit/agent/linux/openvswitch_firewall/test_firewall.py

    Closes-Bug: #2009221
    Related-Bug: #1832758
    Change-Id: Ib8340d9430b946a446edf80886c49fbac729073c
    (cherry picked from commit 008277b8c12d99438951a308b278203fa7a7c3ef)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/877607
Committed: https://opendev.org/openstack/neutron/commit/f5b6c2afd8c4d4c47d7f0efd91a23d3cc550aabd
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit f5b6c2afd8c4d4c47d7f0efd91a23d3cc550aabd
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sun Mar 5 22:12:55 2023 +0100

    [OVS] Allow custom ethertype traffic in the ingress table

    This patch is a partial revert of [1], reinstantiating the code merged
    in [2]. This patch is the complementary to [1]: the traffic with
    custom ethertypes is allowed in the ingress processing tables, same
    as [1] is allowing all traffic from the virtual machine ports in this
    host to leave the node. Both, this patch and [1], are bypassing the
    OVS firewall just for the traffic with the configured allowed
    ethertypes and just for/to the local ports and MAC addresses.

    Any other traffic not coming from a local port or with destination
    a local port, will be blocked as is now.

    [1]https://review.opendev.org/c/openstack/neutron/+/678021
    [2]https://review.opendev.org/c/openstack/neutron/+/668224/

    Conflicts:
           doc/source/admin/config-ovsfwdriver.rst
           neutron/tests/unit/agent/linux/openvswitch_firewall/test_firewall.py

    Closes-Bug: #2009221
    Related-Bug: #1832758
    Change-Id: Ib8340d9430b946a446edf80886c49fbac729073c
    (cherry picked from commit 008277b8c12d99438951a308b278203fa7a7c3ef)
    (cherry picked from commit 5026d805fe01aaf237081c606f1d1bf87bbff6d4)