neutron-linuxbridge-agent not starting due to nf_tables rules

[Realtime]

* Description
When restarting neutron-linuxbridge-agent it fails, because it cannot remove nf_tables chains

* Pre-conditions
Openstack Ussuri on Ubuntu 20.04 installed as described in https://docs.openstack.org/install-guide on real hardware.

* reproduction steps
When you remove an instance there seem to remain some rules in neutronARP-* and neutronMAC-* tables. When restarting neutron-linuxbridge-agent then, it fails:

neutron_lib.exceptions.ProcessExecutionError: Exit code: 4; Stdin: ; Stdout: ; Stderr: ebtables v1.8.4 (nf_tables): CHAIN_USER_DEL failed (Device or resource busy): chain neutronARP-tap0a9b5e3a-21
INFO neutron.plugins.ml2.drivers.agent._common_agent [-] Stopping Linux bridge agent agent.

After flushing these chains the agent can be started.

# openstack --version
openstack 5.2.0

* severity: blocker

[Realtime (peter-icb)]

Maybe that helps: When deleting an instance, neutron-linuxbrigde-agent.log on the compute node says:

2021-02-12 10:42:42.839 380182 INFO neutron.agent.securitygroups_rpc [req-7c47485c-9777-4dfd-9266-eb986e382ff4 347286303b394a9fbf27929b147c6c73 14e1888994e044248ec6c3ccc467db92 - - -] Security group member updated ['84b7c709-67e9-46a0-8d71-1ba068a798b1']
2021-02-12 10:42:43.120 380182 INFO neutron.agent.securitygroups_rpc [req-183b5ec6-681b-4e02-a76a-7ad3753b0c85 - - - - -] Refresh firewall rules
2021-02-12 10:42:43.195 380182 ERROR neutron.agent.linux.utils [req-13205454-f497-47bb-a1a4-bd6f03f284d8 - - - - -] Exit code: 255; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: No such file or directory

[Lajos Katona (lajos-katona)]

Hi, could you please provide debug logs? I tried to reproduce the issue in devstack but without success.

[Realtime (peter-icb)]

Here is a tail from neutron-linuxbrigde-agent.log on the compute machnine.

[Realtime (peter-icb)]

Here is another log-tail.

Is there a fix already avaliable? Openstack Ussuri is UNUSABLE until this bug is fixed. Or is there a workaround to get linuxbrige-agent running without reboot on the compute machine?

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

In order to use Neutron with the older netfilter API (iptables, iptables6, ebtables, arptables), you need to force in the OS the use of the ebtables legacy API:

  /usr/bin/update-alternatives --set ebtables /usr/sbin/ebtables-legacy

This is the same procedure used in the U/S CI [1].

Currently I'm working on the migration to NFT. This is what you are using in your environment (ebtables v1.8.4 (nf_tables)) and this is why is failing. The problem is how ebtables-nft orders the rules, not in the same way as the legacy API.

I have a testing patch [2] to test the migration to the new binaries [3]. I'm still seeing some errors related to ebtables that I'm trying to fix.

Regards.

[1]https://github.com/openstack/neutron/blob/58c9912be0ce5d9bf9eb9e1c44b87cdf90aab452/roles/legacy_ebtables/tasks/main.yaml
[2]https://review.opendev.org/c/openstack/neutron/+/775413/
[3]https://review.opendev.org/c/openstack/neutron/+/775413/11/roles/nftables/tasks/main.yaml

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Patch proposed: https://review.opendev.org/c/openstack/neutron/+/785137

Because we have another two LP bugs related to the nftables migration, once this documentation patch is merged, I would close this one. Any other work related to the migration could be tracked in the other LP bugs.

Related bugs:
- https://bugs.launchpad.net/neutron/+bug/1508155: nftables firewall driver
- https://bugs.launchpad.net/neutron/+bug/1922892: ebtables-nft errors in LB mech driver

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/775413
Reason: tested patches already merged

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.