[RFE] NFTables Firewall Driver

[Thiago Martins]

Nowadays, when using openvswitch-agent with security groups we must use hybrid bridging, i.e. per instance we have both openvswitch bridge and linux bridge. The rationale behind this approach is to set filtering rules matching on given linux bridge.

The hybrid bridging looks like a workaround, it is slow, harder to debug and make things very complicated to work NFV (specially instances running L2 Bridge applications (DPDK for example)...

If you do not use OpenvSwitch, and stick with Linux Bridges only, you have a much simpler setup, no hybrid bridges and easier to debug, nevertheless, you still make use of tons of solutions, like for example:

* iptables;
* ip6tables;
* artables;
* ebtables;
* ipset...

On the other hand, if we manage to create the nftables-firewall-driver for Neutron, we'll end up with a much consistent, simpler and faster solution for the project. Basically, we just need 1 binary:

* nft.

Also, NFTables might work, as-is, for both Linux Bridges and OpenvSwitch! Without involving hybrid bridging!!! One single and elegant solution, to solve all problems at once.

http://people.netfilter.org/2014/wiki/index.php/List_of_presentations

http://people.netfilter.org/2014/wiki/images/0/04/NFWS2014-OVS%2Bconntrack.pdf

Not to mention that with NFTables. we can have native NAT66 (which I dislike very much but, it is there, because NAT is a ugly workaround to deal with IPv4 exhaustion, it have nothing to do with IPv6), that might help us to create Floating IPv6 using the very same approach for Floating IPv4. Please, keep in mind that a Floating IPv6 using NAT66 is very, very bad, and it should be disabled by default. A better Floating IPv6 can be designed without any kind of NAT.

Plus, with NFTables, we might create something like Suricata-as-a-service (or "IDSaaS"), because NFTables is much more powerful than iptables, that it improves IDE systems... For example:

https://home.regit.org/2014/02/suricata-and-nftables/

So, a Neutron nftables-firewall-driver will deprecated everything related to old tools (listed above) and it will bring a much more consistent solution for the project.

Time to move to NFTables!

Cheers!
Thiago

[Kevin Benton (kevinbenton)]

This sounds good. What's the distribution support like for NFT?

[Thiago Martins (martinx)]

Cool!

Ubuntu have NFTables since 14.10 (super easy to backport to 14.04
(Linux 3.13 support it as well), if needed - I have it on my home
firewall now).

Next Ubuntu 16.04 will be the first LTS version of Ubuntu, with NFTables.

I believe that Debian 9 will also include NFTables.

Sorry about typos grammar errors... English isn't my native language...

Best,
Thiago

On 20 October 2015 at 16:36, Kevin Benton <email address hidden> wrote:
> This sounds good. What's the distribution support like for NFT?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1508155
>
> Title:
> [rfe] NFTables Firewall Driver
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/neutron/+bug/1508155/+subscriptions

[Armando Migliaccio (armando-migliaccio)]

To be discussed during the drivers meeting.

[Miguel Angel Ajo (mangelajo)]

For some reason the hybrid solution performs equal or sightly better in some conditions (don't ask me why, I don't know). It works, and I agree it has a lot of complexity.

NFT could be a good thing for linuxbridge or the virtual routers. But I'd prefer to see an openflow+CT firewall for OVS. (to avoid mixing too many technologies at once, like we have now)

We also have the OVS/CT initiative I know @jlibova was working on it already [1]. Btw, we have the old spec for this, but I'm not unsure if he filled any rfe.

[1] https://blueprints.launchpad.net/neutron/+spec/ovs-firewall-driver

[Armando Migliaccio (armando-migliaccio)]

So are we saying that going forward we could have the following options?

LinuxBridge -> NFTables
Open vSwitch -> Openflow+CT?

Let's reach consensus.

[Akihiro Motoki (amotoki)]

NFTables for Linux bridge sounds good.

For OVS, I tend to agree with ajo.
we use OVS tables for various purposes and NFtables is a wrapper for OVS/CT.
It is better to use OVS table in a native way as long as it becomes too complicated and/or reinvents an existing things.

One concern on ovs-firewall-driver ([1] above) is that security group API change was proposed as a part of the work. This blocked subsequent effort. Changing SG API is not a good idea. The situation might have changed since then. If the situation changed, I think OVS firewall is a good direction.

[Miguel Angel Ajo (mangelajo)]

@amotoki, yes, the situation changed, there are no changes suggested to the sg api now,
although the blueprint could be outdated (checking).

[Armando Migliaccio (armando-migliaccio)]

As discussed in [1], there was consensus on strategy as outlined in #5. However we need a sponsor for this initiative.

[1] http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-01-12-15.00.log.html

[Armando Migliaccio (armando-migliaccio)]

Any volunteers to spec and work on this?

[Armando Migliaccio (armando-migliaccio)]

http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-02-04-22.01.log.html#l-137

[Armando Migliaccio (armando-migliaccio)]

Great request in principle, but no-one seems willing to work on it.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Junien F (axino)]

I think this should be revisited, nftables may hinder throughput much less than iptables.

[Thiago Martins (martinx)]

Hey guys,

It's quite nice to see that more people thinking about this!

Things changed since 2015 and it might be fairly easy to start using nftables with Neutron today, the hybrid mode! As follows:

https://wiki.debian.org/nftables

https://www.redhat.com/en/blog/using-iptables-nft-hybrid-linux-firewall

This way, no changes are required on Neutron. However, I honestly don't know the limitations of this approach. I'm planning to test this soon!

---

Another thing that is on my radar, is a Load Balancer based on nftables! It seems to be really awesome! So, what about replacing Octavia with something a lot more simpler? Here is it:

https://www.zevenet.com/knowledge-base/nftlb/what-is-nftlb/

---

Time flies and now, Linux has the so-called BPFilter!

BPF comes to firewalls:
https://lwn.net/Articles/747551/

https://news.ycombinator.com/item?id=16421168

What in the hell? The nftables is already obsolete?

No idea... lol

Look, at, this:

BPF at Facebook (and beyond):
https://lwn.net/Articles/801871/

---

What do you guys think?!

:-D

[Ian Kumlien (pomac)]

I talked about this at the Ireland snowstorm, I mean, meeting - and now we're hitting the xlock limitations... hard...

I know both iptables and nft but... It could be hard to have the time to figure this out...

[Arturo Borrero Gonzalez (arturoborrero)]

It seems other folks in the virt community already switched to native nftables (see libvirt, LXD, firewalld, some ongoing efforts in k8s, etc).

I'm running neutron rocky with debian buster just fine with iptables-nft and ebtables-nft (requires iptables 1.8.5 at least), using linuxbridge.

Basically, neutron is using nftables just fine, without even noticing. It would be awesome if the native driver is implemented to actually gain from the benefits of nftables: native sets, dictionaries and maps, native support for all the address families in a single tool, and more.

[Brian Haley (brian-haley)]

Arturo - I don't see Neutron going to a native nftables implementation soon, mostly because the default install uses ML2/OVS with the OVS firewall, which doesn't require iptables for security groups. And with us moving more towards OVN it's not an issue, except when using linuxbridge, which isn't exactly a priority.

[Thiago Martins (martinx)]

Hey guys! Long time!

I'm using in production, nftables via Ubuntu's etc/alternative subsystem, so, my iptables points to iptables-nft and I'm actually using nftables without Neutron realizing it!

It's cool but, what about the ipset? I have it configured on Neutron but, apparently, nftables has its own sets and there is no `ipset` etc/alternative that uses nftables as a backend.

https://wiki.nftables.org/wiki-nftables/index.php/Moving_from_ipset_to_nftables

So, is this a problem? My environment is working okay...

I disagree with Brian, the Linux Bridges are the default in big deployments out there, for example, with OpenStack Ansible. I have deployed a handful of OpenStack clouds in the last 4 years, all based on Linux Bridges.

The OVN is highly experimental and adding OpenvSwitch to the mix only makes things way more complicated.

It would be awesome to see Neutron with a native nftables driver! But I'm happy with the etc/alternative option from Debian/Ubuntu BUT, what about the `ipset`?

Cheers!

[Brian Haley (brian-haley)]

We can agree to disagree on using the Linux Bridge driver, but we (Red Hat) have almost no customers using it, and our default is now OVN, so it's not experimental at all IMO.

And yes, most distros have iptables-nft under the hood, so you're using nftables indirectly. It should just continue to work. If someone committed to writing an nftables driver I'm sure the community would take a look at it, but no one has proposed doing it at our planning meetings.

[Slawek Kaplonski (slaweq)]

Actually we were discussing that on last PTG and Rodolfo already started slowly some work on that. Please see his early WIP patch https://review.opendev.org/c/openstack/neutron/+/759874

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/785177
Committed: https://opendev.org/openstack/neutron/commit/0a931391d8990f3e654b4bfda24ae4119c609bbf
Submitter: "Zuul (22348)"
Branch: master

commit 0a931391d8990f3e654b4bfda24ae4119c609bbf
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Apr 7 13:16:21 2021 +0000

    Make ARP protection commands compatible with "ebtables-nft"

    "nftables" compatible binary, "ebtables-nft", is not 100% compatible
    with the legacy API, as reported in LP#1922892.

    This patch fixes the following issues when using "ebtables-nft" (while
    keeping compatibility with legacy binary):
    - When a new chain is created, a default DROP rule is added at the end
      of the chain (append). This will prevent the error code 4 when the
      chain is listed.
    - The chain rules are added at the begining of the chain (insert),
      before the default DROP rule. This will prioritize the port rules.
    - The MAC rules are cleaned before the new ones are added. That will
      prevent the deletion of any new needed rule, now added after the
      deletion.
    - The "ebtables" command will retry on error code 4. This is the
      error returned when the chains are listed and no rule is present
      in a new created chain (reporeted in LP#1922892).

    This code is backwards compatible, that means it works with the legacy
    "ebtables" binary; this is currently installed in the Neutron CI [1].
    In order to test with the new binary, "ebtables-nft", two new CI jobs
    are added to the periodic queue [2].

    [1]https://github.com/openstack/neutron/blob/1ad9ca56b07ffdc9f7e0bc6a62af61961b9128eb/roles/legacy_ebtables/tasks/main.yaml
    [2]https://review.opendev.org/c/openstack/neutron/+/785144

    Closes-Bug: #1922892
    Related-Bug: #1508155

    Change-Id: I9463b000f6f63e65aaf91d60b30f6c92c01e3baf

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/785917
Committed: https://opendev.org/openstack/neutron/commit/dd40bac8b54a7f1c05a573d915c705b0539f0d8f
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit dd40bac8b54a7f1c05a573d915c705b0539f0d8f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Apr 7 13:16:21 2021 +0000

    Make ARP protection commands compatible with "ebtables-nft"

    "nftables" compatible binary, "ebtables-nft", is not 100% compatible
    with the legacy API, as reported in LP#1922892.

    This patch fixes the following issues when using "ebtables-nft" (while
    keeping compatibility with legacy binary):
    - When a new chain is created, a default DROP rule is added at the end
      of the chain (append). This will prevent the error code 4 when the
      chain is listed.
    - The chain rules are added at the begining of the chain (insert),
      before the default DROP rule. This will prioritize the port rules.
    - The MAC rules are cleaned before the new ones are added. That will
      prevent the deletion of any new needed rule, now added after the
      deletion.
    - The "ebtables" command will retry on error code 4. This is the
      error returned when the chains are listed and no rule is present
      in a new created chain (reporeted in LP#1922892).

    This code is backwards compatible, that means it works with the legacy
    "ebtables" binary; this is currently installed in the Neutron CI [1].
    In order to test with the new binary, "ebtables-nft", two new CI jobs
    are added to the periodic queue [2].

    [1]https://github.com/openstack/neutron/blob/1ad9ca56b07ffdc9f7e0bc6a62af61961b9128eb/roles/legacy_ebtables/tasks/main.yaml
    [2]https://review.opendev.org/c/openstack/neutron/+/785144

    Closes-Bug: #1922892
    Related-Bug: #1508155

    Change-Id: I9463b000f6f63e65aaf91d60b30f6c92c01e3baf
    (cherry picked from commit 0a931391d8990f3e654b4bfda24ae4119c609bbf)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/785144
Committed: https://opendev.org/openstack/neutron/commit/f7d2c3608d1fa6cbc8c562f9121ec0f0d80c3962
Submitter: "Zuul (22348)"
Branch: master

commit f7d2c3608d1fa6cbc8c562f9121ec0f0d80c3962
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Apr 7 10:56:09 2021 +0000

    Add periodic jobs to test "nftables" binaries

    In order to check how the "nftables" binaries work with Neutron, two
    new jobs have been added to the periodic queue:
    - neutron-tempest-plugin-scenario-linuxbridge-nftables
    - neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-nftables

    In those two jobs, the binaries for "iptables", "ip6tables",
    "arptables" and "ebtables" are replaced with the "nftables"
    counterparts; by default, newer operating systems use the "nftables"
    versions, providing the legacy API to the user but executing the
    new packet handling in Netfilter.

    Change-Id: Idec6d480886298f6d71b1dd649c9255ee6b7bebb
    Related-Bug: #1508155
    Related-Bug: #1922892

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/775413
Reason: tested patches already merged

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/804056

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/804057

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/804058

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/804056
Committed: https://opendev.org/openstack/neutron/commit/fafa5dacd5057120562184a734e7345e7c0e9639
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit fafa5dacd5057120562184a734e7345e7c0e9639
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Apr 7 13:16:21 2021 +0000

    Make ARP protection commands compatible with "ebtables-nft"

    "nftables" compatible binary, "ebtables-nft", is not 100% compatible
    with the legacy API, as reported in LP#1922892.

    This patch fixes the following issues when using "ebtables-nft" (while
    keeping compatibility with legacy binary):
    - When a new chain is created, a default DROP rule is added at the end
      of the chain (append). This will prevent the error code 4 when the
      chain is listed.
    - The chain rules are added at the begining of the chain (insert),
      before the default DROP rule. This will prioritize the port rules.
    - The MAC rules are cleaned before the new ones are added. That will
      prevent the deletion of any new needed rule, now added after the
      deletion.
    - The "ebtables" command will retry on error code 4. This is the
      error returned when the chains are listed and no rule is present
      in a new created chain (reporeted in LP#1922892).

    This code is backwards compatible, that means it works with the legacy
    "ebtables" binary; this is currently installed in the Neutron CI [1].
    In order to test with the new binary, "ebtables-nft", two new CI jobs
    are added to the periodic queue [2].

    [1]https://github.com/openstack/neutron/blob/1ad9ca56b07ffdc9f7e0bc6a62af61961b9128eb/roles/legacy_ebtables/tasks/main.yaml
    [2]https://review.opendev.org/c/openstack/neutron/+/785144

    Closes-Bug: #1922892
    Related-Bug: #1508155
    Closes-Bug: #1938670

    Conflicts:
        neutron/tests/unit/plugins/ml2/drivers/linuxbridge/agent/test_arp_protect.py

    Change-Id: I9463b000f6f63e65aaf91d60b30f6c92c01e3baf
    (cherry picked from commit 0a931391d8990f3e654b4bfda24ae4119c609bbf)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/804057
Committed: https://opendev.org/openstack/neutron/commit/f034de001ba224c3add364ca5d00cc15e26c2e96
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit f034de001ba224c3add364ca5d00cc15e26c2e96
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Apr 7 13:16:21 2021 +0000

    Make ARP protection commands compatible with "ebtables-nft"

    "nftables" compatible binary, "ebtables-nft", is not 100% compatible
    with the legacy API, as reported in LP#1922892.

    This patch fixes the following issues when using "ebtables-nft" (while
    keeping compatibility with legacy binary):
    - When a new chain is created, a default DROP rule is added at the end
      of the chain (append). This will prevent the error code 4 when the
      chain is listed.
    - The chain rules are added at the begining of the chain (insert),
      before the default DROP rule. This will prioritize the port rules.
    - The MAC rules are cleaned before the new ones are added. That will
      prevent the deletion of any new needed rule, now added after the
      deletion.
    - The "ebtables" command will retry on error code 4. This is the
      error returned when the chains are listed and no rule is present
      in a new created chain (reporeted in LP#1922892).

    This code is backwards compatible, that means it works with the legacy
    "ebtables" binary; this is currently installed in the Neutron CI [1].
    In order to test with the new binary, "ebtables-nft", two new CI jobs
    are added to the periodic queue [2].

    [1]https://github.com/openstack/neutron/blob/1ad9ca56b07ffdc9f7e0bc6a62af61961b9128eb/roles/legacy_ebtables/tasks/main.yaml
    [2]https://review.opendev.org/c/openstack/neutron/+/785144

    Closes-Bug: #1922892
    Related-Bug: #1508155
    Closes-Bug: #1938670

    Conflicts:
        neutron/tests/unit/plugins/ml2/drivers/linuxbridge/agent/test_arp_protect.py

    Change-Id: I9463b000f6f63e65aaf91d60b30f6c92c01e3baf
    (cherry picked from commit 0a931391d8990f3e654b4bfda24ae4119c609bbf)
    (cherry picked from commit fafa5dacd5057120562184a734e7345e7c0e9639)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/804058
Committed: https://opendev.org/openstack/neutron/commit/4245963c71280bebd94e9005230d610931a7c2df
Submitter: "Zuul (22348)"
Branch: stable/train

commit 4245963c71280bebd94e9005230d610931a7c2df
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Apr 7 13:16:21 2021 +0000

    Make ARP protection commands compatible with "ebtables-nft"

    "nftables" compatible binary, "ebtables-nft", is not 100% compatible
    with the legacy API, as reported in LP#1922892.

    This patch fixes the following issues when using "ebtables-nft" (while
    keeping compatibility with legacy binary):
    - When a new chain is created, a default DROP rule is added at the end
      of the chain (append). This will prevent the error code 4 when the
      chain is listed.
    - The chain rules are added at the begining of the chain (insert),
      before the default DROP rule. This will prioritize the port rules.
    - The MAC rules are cleaned before the new ones are added. That will
      prevent the deletion of any new needed rule, now added after the
      deletion.
    - The "ebtables" command will retry on error code 4. This is the
      error returned when the chains are listed and no rule is present
      in a new created chain (reporeted in LP#1922892).

    This code is backwards compatible, that means it works with the legacy
    "ebtables" binary; this is currently installed in the Neutron CI [1].
    In order to test with the new binary, "ebtables-nft", two new CI jobs
    are added to the periodic queue [2].

    [1]https://github.com/openstack/neutron/blob/1ad9ca56b07ffdc9f7e0bc6a62af61961b9128eb/roles/legacy_ebtables/tasks/main.yaml
    [2]https://review.opendev.org/c/openstack/neutron/+/785144

    Closes-Bug: #1922892
    Related-Bug: #1508155
    Closes-Bug: #1938670

    Conflicts:
        neutron/tests/unit/plugins/ml2/drivers/linuxbridge/agent/test_arp_protect.py

    Change-Id: I9463b000f6f63e65aaf91d60b30f6c92c01e3baf
    (cherry picked from commit 0a931391d8990f3e654b4bfda24ae4119c609bbf)
    (cherry picked from commit fafa5dacd5057120562184a734e7345e7c0e9639)

[Ian Kumlien (pomac)]

If we use nft and use inet - then ipv4 and ipv6 can be mixed, thus we could get floating ip for ipv6.