iptable rules restoring error in l3-agent and openvswitch-agent

[Bryan HWANG]

Openstack version: zed/stable
OS version: Ubuntu 22.04.2 LTS
Kernel version: 5.15.0-75-generic #82-Ubuntu
Deployment: kolla-ansible

    iptable rules restoring error in l3-agent and openvswitch-agent:

      openvswitch-agnet log:

        2023-06-23 15:54:58.616 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-4440bce1-8c07-4243-ac1b-2566b406a30a - - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['iptables-restore', '-n']; Stdin: # Generated by iptables_manager
        *filter
        :FORWARD - [0:0]
        :INPUT - [0:0]
        :OUTPUT - [0:0]
        :neutron-filter-top - [0:0]
        :neutron-openvswi-FORWARD - [0:0]
        :neutron-openvswi-INPUT - [0:0]
        :neutron-openvswi-OUTPUT - [0:0]
        :neutron-openvswi-local - [0:0]
        :neutron-openvswi-sg-chain - [0:0]
        :neutron-openvswi-sg-fallback - [0:0]
        -I FORWARD 1 -j neutron-filter-top
        -I FORWARD 2 -j neutron-openvswi-FORWARD
        -I INPUT 1 -j neutron-openvswi-INPUT
        -I OUTPUT 1 -j neutron-filter-top
        -I OUTPUT 2 -j neutron-openvswi-OUTPUT
        -I neutron-filter-top 1 -j neutron-openvswi-local
        -I neutron-openvswi-FORWARD 1 -m physdev --physdev-out tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT
        -I neutron-openvswi-FORWARD 2 -m physdev --physdev-in tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT
        -I neutron-openvswi-FORWARD 3 -m physdev --physdev-out tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT
        -I neutron-openvswi-FORWARD 4 -m physdev --physdev-in tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT
        -I neutron-openvswi-sg-chain 1 -j ACCEPT
        -I neutron-openvswi-sg-fallback 1 -m comment --comment "Default drop rule for unmatched traffic." -j DROP
        COMMIT
        # Completed by iptables_manager
        # Generated by iptables_manager
        *raw
        :OUTPUT - [0:0]
        :PREROUTING - [0:0]
        :neutron-openvswi-OUTPUT - [0:0]
        :neutron-openvswi-PREROUTING - [0:0]
        -I OUTPUT 1 -j neutron-openvswi-OUTPUT
        -I PREROUTING 1 -j neutron-openvswi-PREROUTING
        COMMIT
        # Completed by iptables_manager
        ; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables): Couldn't load match `physdev':No such file or directory

        Error occurred at line: 19
        Try `iptables-restore -h' or 'iptables-restore --help' for more information.

      l3-agent log:

        2023-06-23 16:15:49.545 33 ERROR neutron.agent.linux.iptables_manager [-] Failure applying iptables rules: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['ip', 'netns', 'exec', 'qrouter-0f0e60d0-bf51-4361-901b-4b998201b44b', 'iptables-restore', '-n']; Stdin: # Generated by iptables_manager
        *filter
        :FORWARD - [0:0]
        :INPUT - [0:0]
        :OUTPUT - [0:0]
        :neutron-filter-top - [0:0]
        :neutron-l3-agent-FORWARD - [0:0]
        :neutron-l3-agent-INPUT - [0:0]
        :neutron-l3-agent-OUTPUT - [0:0]
        :neutron-l3-agent-local - [0:0]
        :neutron-l3-agent-scope - [0:0]
        -I FORWARD 1 -j neutron-filter-top
        -I FORWARD 2 -j neutron-l3-agent-FORWARD
        -I INPUT 1 -j neutron-l3-agent-INPUT
        -I OUTPUT 1 -j neutron-filter-top
        -I OUTPUT 2 -j neutron-l3-agent-OUTPUT
        -I neutron-filter-top 1 -j neutron-l3-agent-local
        -I neutron-l3-agent-FORWARD 1 -j neutron-l3-agent-scope
        -I neutron-l3-agent-scope 1 -m mark --mark 0x1/0xffff -j DROP
        COMMIT
        # Completed by iptables_manager
        # Generated by iptables_manager
        *mangle
        :FORWARD - [0:0]
        :INPUT - [0:0]
        :OUTPUT - [0:0]
        :POSTROUTING - [0:0]
        :PREROUTING - [0:0]
        :neutron-l3-agent-FORWARD - [0:0]
        :neutron-l3-agent-INPUT - [0:0]
        :neutron-l3-agent-OUTPUT - [0:0]
        :neutron-l3-agent-POSTROUTING - [0:0]
        :neutron-l3-agent-PREROUTING - [0:0]
        :neutron-l3-agent-float-snat - [0:0]
        :neutron-l3-agent-floatingip - [0:0]
        :neutron-l3-agent-mark - [0:0]
        :neutron-l3-agent-scope - [0:0]
        -I FORWARD 1 -j neutron-l3-agent-FORWARD
        -I INPUT 1 -j neutron-l3-agent-INPUT
        -I OUTPUT 1 -j neutron-l3-agent-OUTPUT
        -I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING
        -I PREROUTING 1 -j neutron-l3-agent-PREROUTING
        -I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-mark
        -I neutron-l3-agent-PREROUTING 2 -j neutron-l3-agent-scope
        -I neutron-l3-agent-PREROUTING 3 -m connmark ! --mark 0x0/0xffff0000 -j CONNMARK --restore-mark --nfmask 0xffff0000 --ctmask 0xffff0000
        -I neutron-l3-agent-PREROUTING 4 -j neutron-l3-agent-floatingip
        -I neutron-l3-agent-PREROUTING 5 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff
        -I neutron-l3-agent-float-snat 1 -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000
        COMMIT
        # Completed by iptables_manager
        # Generated by iptables_manager
        *nat
        :OUTPUT - [0:0]
        :POSTROUTING - [0:0]
        :PREROUTING - [0:0]
        :neutron-l3-agent-OUTPUT - [0:0]
        :neutron-l3-agent-POSTROUTING - [0:0]
        :neutron-l3-agent-PREROUTING - [0:0]
        :neutron-l3-agent-float-snat - [0:0]
        :neutron-l3-agent-snat - [0:0]
        :neutron-postrouting-bottom - [0:0]
        -I OUTPUT 1 -j neutron-l3-agent-OUTPUT
        -I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING
        -I POSTROUTING 2 -j neutron-postrouting-bottom
        -I PREROUTING 1 -j neutron-l3-agent-PREROUTING
        -I neutron-l3-agent-POSTROUTING 1 ! -o rfp-0f0e60d0-b -m conntrack ! --ctstate DNAT -j ACCEPT
        -I neutron-l3-agent-PREROUTING 1 -d 137.175.31.207/32 -i rfp-0f0e60d0-b -j DNAT --to-destination 10.10.0.246
        -I neutron-l3-agent-float-snat 1 -s 10.10.0.246/32 -j SNAT --to-source 137.175.31.207 --random-fully
        -I neutron-l3-agent-snat 1 -j neutron-l3-agent-float-snat
        -I neutron-postrouting-bottom 1 -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
        COMMIT
        # Completed by iptables_manager
        # Generated by iptables_manager
        *raw
        :OUTPUT - [0:0]
        :PREROUTING - [0:0]
        :neutron-l3-agent-OUTPUT - [0:0]
        :neutron-l3-agent-PREROUTING - [0:0]
        -I OUTPUT 1 -j neutron-l3-agent-OUTPUT
        -I PREROUTING 1 -j neutron-l3-agent-PREROUTING
        COMMIT
        # Completed by iptables_manager
        ; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables): Couldn't load match `mark':No such file or directory

        Error occurred at line: 19

      And we check the system the x_tables kernel module were loaded:

        # lsmod | grep x_tablesx_tables 53248 12 xt_conntrack,nft_compat,xt_tcpudp,xt_physdev,xt_nat,xt_comment,ip6_tables,xt_connmark,xt_CT,ip_tables,xt_REDIRECT,xt_mark

        (neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*mark.so"
        ./lib/x86_64-linux-gnu/xtables/libxt_connmark.so
        ./lib/x86_64-linux-gnu/xtables/libxt_mark.so
        ./lib/x86_64-linux-gnu/xtables/libebt_mark.so
        (neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*physdev.so"
        ./lib/x86_64-linux-gnu/xtables/libxt_physdev.so

[Brian Haley (brian-haley)]

This is most likely due to a system update, as iptables is being replaced by nftables I've seen this happen.

You should be able to fix this with update-alternatives, this is my working system:

$ sudo update-alternatives --config iptables
There are 2 choices for the alternative iptables (providing /usr/sbin/iptables).

  Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/sbin/iptables-nft 20 auto mode
  1 /usr/sbin/iptables-legacy 10 manual mode
  2 /usr/sbin/iptables-nft 20 manual mode

Press <enter> to keep the current choice[*], or type selection number:

[Bryan HWANG (8ryan9)]

Thanks a lot, Brian, will try your solution

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Pawel Kubica (pawcykca)]

On DevStack Wallaby installed on CentOS Stream 8 downgrade of iptables package to version 'iptables-1.8.4-24.el8' solved this problem.