LinuxBridgeARPSpoofTestCase functional tests fails with latest jammy kernel 5.15.0-86.96

[yatin]

Tests fails while running ebtables(['-D', chain] + rule.split()) with:-
2023-10-05 12:09:19.307 41358 ERROR neutron.agent.linux.utils [None req-defd197a-c4e2-4761-a4cc-cc960a3ff71a - - - - - -] Exit code: 4; Cmd: ['ip', 'netns', 'exec', 'test-b58b5cf9-5018-4801-aacb-8b00fae3fe37', 'ebtables', '-t', 'nat', '--concurrent', '-D', 'neutronMAC-test-veth09e6dc', '-i', 'test-veth09e6dc', '--among-src', 'fa:16:3e:ac:fd:b6', '-j', 'RETURN']; Stdin: ; Stdout: ; Stderr: ebtables v1.8.7 (nf_tables): RULE_DELETE failed (Invalid argument): rule in chain neutronMAC-test-veth09e6dc

2023-10-05 12:09:29.576 41358 ERROR neutron.agent.linux.utils [None req-defd197a-c4e2-4761-a4cc-cc960a3ff71a - - - - - -] Exit code: 4; Cmd: ['ip', 'netns', 'exec', 'test-b58b5cf9-5018-4801-aacb-8b00fae3fe37', 'ebtables', '-t', 'nat', '--concurrent', '-D', 'neutronMAC-test-veth09e6dc', '-i', 'test-veth09e6dc', '--among-src', 'fa:16:3e:ac:fd:b6', '-j', 'RETURN']; Stdin: ; Stdout: ; Stderr: ebtables v1.8.7 (nf_tables): RULE_DELETE failed (Invalid argument): rule in chain neutronMAC-test-veth09e6dc

2023-10-05 12:09:50.099 41358 ERROR neutron.agent.linux.utils [None req-defd197a-c4e2-4761-a4cc-cc960a3ff71a - - - - - -] Exit code: 4; Cmd: ['ip', 'netns', 'exec', 'test-b58b5cf9-5018-4801-aacb-8b00fae3fe37', 'ebtables', '-t', 'nat', '--concurrent', '-D', 'neutronMAC-test-veth09e6dc', '-i', 'test-veth09e6dc', '--among-src', 'fa:16:3e:ac:fd:b6', '-j', 'RETURN']; Stdin: ; Stdout: ; Stderr: ebtables v1.8.7 (nf_tables): RULE_DELETE failed (Invalid argument): rule in chain neutronMAC-test-veth09e6dc

The new kernel includes below changes which have triggered this, described in https://launchpad.net/ubuntu/+source/linux/5.15.0-86.96:-
    - netfilter: nf_tables: disallow element updates of bound anonymous sets
    - netfilter: nf_tables: reject unbound anonymous set before commit phase
    - netfilter: nf_tables: reject unbound chain set before commit phase
    - netfilter: nf_tables: disallow updates of anonymous sets

Following two test fails:-
- test_arp_protection_update
- test_arp_fails_incorrect_mac_protection

[yatin (yatinkarel)]

Rodolfo has pushed https://review.opendev.org/c/openstack/neutron/+/897412 to avoid this issue.

[yatin (yatinkarel)]

Seeing in linuxbridge scenario jobs[1][2] running on jammy:-
Oct 06 00:41:16 np0035425548 neutron-linuxbridge-agent[57270]: DEBUG oslo.privsep.daemon [-] privsep: reply[ce7ef588-0462-40c3-9270-a994748678c2]: (4, ('', 'ebtables v1.8.7 (nf_tables): RULE_DELETE failed (Invalid argument): rule in chain neutronMAC-tap322ad9bd-6c\n', 4)) {{(pid=58378) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}}
Oct 06 00:41:16 np0035425548 neutron-linuxbridge-agent[57270]: ERROR neutron.agent.linux.utils [None req-15ae2c74-0fbe-4879-8ad8-a35d87b58d41 None None] Exit code: 4; Cmd: ['ebtables', '-t', 'nat', '--concurrent', '-D', 'neutronMAC-tap322ad9bd-6c', '-i', 'tap322ad9bd-6c', '--among-src', 'fa:16:3e:b3:a9:37', '-j', 'RETURN']; Stdin: ; Stdout: ; Stderr: ebtables v1.8.7 (nf_tables): RULE_DELETE failed (Invalid argument): rule in chain neutronMAC-tap322ad9bd-6c
Oct 06 01:24:57 np0035425548 neutron-linuxbridge-agent[57270]: DEBUG oslo.privsep.daemon [-] privsep: reply[eb1311e5-9142-470d-97e3-5dfee826404b]: (4, ('', 'ebtables v1.8.7 (nf_tables): RULE_DELETE failed (Invalid argument): rule in chain neutronMAC-tap322ad9bd-6c\n', 4)) {{(pid=58378) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}}
Oct 06 01:24:57 np0035425548 neutron-linuxbridge-agent[57270]: ERROR neutron.agent.linux.utils [None req-15ae2c74-0fbe-4879-8ad8-a35d87b58d41 None None] Exit code: 4; Cmd: ['ebtables', '-t', 'nat', '--concurrent', '-D', 'neutronMAC-tap322ad9bd-6c', '-i', 'tap322ad9bd-6c', '--among-src', 'fa:16:3e:b3:a9:37', '-j', 'RETURN']; Stdin: ; Stdout: ; Stderr: ebtables v1.8.7 (nf_tables): RULE_DELETE failed (Invalid argument): rule in chain neutronMAC-tap322ad9bd-6c

[1] https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-linuxbridge-2023-1
[2] https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-linuxbridge

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/897529

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/897565

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/897565
Committed: https://opendev.org/openstack/neutron/commit/27a19fb88afba2900c611741f5975d7461455a0f
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 27a19fb88afba2900c611741f5975d7461455a0f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Oct 6 00:20:14 2023 +0000

    [stable-only] Disable "neutron-tempest-plugin-jobs-2023-1" temporarily

    There is a cyclic dependency between two patches:
    * https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/897529
    * https://review.opendev.org/c/openstack/neutron/+/897440

    In order to merge the Neutron one first, it is needed to disable the
    "neutron-tempest-plugin-jobs-2023-1" jobs.

    This patch is also disabling the functional and fullstack jobs that
    will be fixed by the upper Neutron patch.

    This patch is proposed for 2023.1 only.

    Related-Bug: #2038541
    Change-Id: I8af6b1dc0552f0aba44400b41a542818b6f01fcf

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/897529
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/7f679d3cd48e1bcb8a3cbb8346ec8f374e9a9f74
Submitter: "Zuul (22348)"
Branch: master

commit 7f679d3cd48e1bcb8a3cbb8346ec8f374e9a9f74
Author: yatinkarel <email address hidden>
Date: Fri Oct 6 14:39:37 2023 +0530

    Include legacy_ebtables for LinuxBridge Jammy jobs

    With the latest kernel update in Ubuntu jammy we
    need to include this role else jobs are failing.

    Related-Bug: #2038541
    Change-Id: Ia277239cf5cc8d4534d46a4a2340ba42905923ff

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/897438
Committed: https://opendev.org/openstack/neutron/commit/4266dce979a8e168a82b0e27c338b9248bdb1c3f
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 4266dce979a8e168a82b0e27c338b9248bdb1c3f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Oct 5 10:49:23 2023 +0000

    Add "jammy" distribution release to the legacy ebtables installation

    Related-Bug: #2038541

    Change-Id: Ia8393f0e4f736dafb3ba7fb50cd2b679c24b1a01
    (cherry picked from commit acc6ff05806b414e552525a7e6d758e253d6ea23)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/898729
Committed: https://opendev.org/openstack/neutron/commit/1879d925330af5598a105a8893ab6cfda9dc37e6
Submitter: "Zuul (22348)"
Branch: master

commit 1879d925330af5598a105a8893ab6cfda9dc37e6
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Oct 16 00:09:50 2023 +0000

    "ebtables-nft" MAC rule deletion failing

    "ebtables-nft" is failing to delete the rule filtering by MAC address:
      Bridge chain: neutronMAC-test-veth024379, entries: 2, policy: DROP
      -i test-veth024379 --among-src fa:16:3e:47:87:0 -j RETURN
      -j DROP

    A workaround for this issue, that works with both "ebtables-nft" and
    "ebtables-legacy", is to flush the table and recreate the DROP rule.
    The MAC spoofing tables have two rules: the one filtering by MAC address
    and the default DROP rule. This workaround has the same effect as just
    deleting the filtering rule.

    Closes-Bug: #2038541
    Change-Id: I38bd016c35d7a76d88c6eceec797d1cea84c45d1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/898829

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/898831

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/898832

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/898833

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/898831
Committed: https://opendev.org/openstack/neutron/commit/50fb47b10c0142cebefef3357bf2f9bfadadd5dd
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 50fb47b10c0142cebefef3357bf2f9bfadadd5dd
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Oct 16 00:09:50 2023 +0000

    "ebtables-nft" MAC rule deletion failing

    "ebtables-nft" is failing to delete the rule filtering by MAC address:
      Bridge chain: neutronMAC-test-veth024379, entries: 2, policy: DROP
      -i test-veth024379 --among-src fa:16:3e:47:87:0 -j RETURN
      -j DROP

    A workaround for this issue, that works with both "ebtables-nft" and
    "ebtables-legacy", is to flush the table and recreate the DROP rule.
    The MAC spoofing tables have two rules: the one filtering by MAC address
    and the default DROP rule. This workaround has the same effect as just
    deleting the filtering rule.

    Closes-Bug: #2038541
    Change-Id: I38bd016c35d7a76d88c6eceec797d1cea84c45d1
    (cherry picked from commit 1879d925330af5598a105a8893ab6cfda9dc37e6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/898833
Committed: https://opendev.org/openstack/neutron/commit/7dbd06d66e4daebab90e4d334ae43013580e555a
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 7dbd06d66e4daebab90e4d334ae43013580e555a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Oct 16 00:09:50 2023 +0000

    "ebtables-nft" MAC rule deletion failing

    "ebtables-nft" is failing to delete the rule filtering by MAC address:
      Bridge chain: neutronMAC-test-veth024379, entries: 2, policy: DROP
      -i test-veth024379 --among-src fa:16:3e:47:87:0 -j RETURN
      -j DROP

    A workaround for this issue, that works with both "ebtables-nft" and
    "ebtables-legacy", is to flush the table and recreate the DROP rule.
    The MAC spoofing tables have two rules: the one filtering by MAC address
    and the default DROP rule. This workaround has the same effect as just
    deleting the filtering rule.

    Closes-Bug: #2038541
    Change-Id: I38bd016c35d7a76d88c6eceec797d1cea84c45d1
    (cherry picked from commit 1879d925330af5598a105a8893ab6cfda9dc37e6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/898832
Committed: https://opendev.org/openstack/neutron/commit/85b25009af394d0ee0a50534d6aa1d2240828067
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 85b25009af394d0ee0a50534d6aa1d2240828067
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Oct 16 00:09:50 2023 +0000

    "ebtables-nft" MAC rule deletion failing

    "ebtables-nft" is failing to delete the rule filtering by MAC address:
      Bridge chain: neutronMAC-test-veth024379, entries: 2, policy: DROP
      -i test-veth024379 --among-src fa:16:3e:47:87:0 -j RETURN
      -j DROP

    A workaround for this issue, that works with both "ebtables-nft" and
    "ebtables-legacy", is to flush the table and recreate the DROP rule.
    The MAC spoofing tables have two rules: the one filtering by MAC address
    and the default DROP rule. This workaround has the same effect as just
    deleting the filtering rule.

    Closes-Bug: #2038541
    Change-Id: I38bd016c35d7a76d88c6eceec797d1cea84c45d1
    (cherry picked from commit 1879d925330af5598a105a8893ab6cfda9dc37e6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/898829
Committed: https://opendev.org/openstack/neutron/commit/5485a19356a62e0b98969323c064eaa5dab970f9
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 5485a19356a62e0b98969323c064eaa5dab970f9
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Oct 16 00:09:50 2023 +0000

    "ebtables-nft" MAC rule deletion failing

    "ebtables-nft" is failing to delete the rule filtering by MAC address:
      Bridge chain: neutronMAC-test-veth024379, entries: 2, policy: DROP
      -i test-veth024379 --among-src fa:16:3e:47:87:0 -j RETURN
      -j DROP

    A workaround for this issue, that works with both "ebtables-nft" and
    "ebtables-legacy", is to flush the table and recreate the DROP rule.
    The MAC spoofing tables have two rules: the one filtering by MAC address
    and the default DROP rule. This workaround has the same effect as just
    deleting the filtering rule.

    Closes-Bug: #2038541
    Change-Id: I38bd016c35d7a76d88c6eceec797d1cea84c45d1
    (cherry picked from commit 1879d925330af5598a105a8893ab6cfda9dc37e6)

[Björn Hinz (bhinz83)]

Workaround for Ubuntu 22.04 with OpenStack Yoga:

Switch ebtables to legacy and restart linuxbridge-service.

# /usr/bin/update-alternatives --set ebtables /usr/sbin/ebtables-legacy
# systemctl restart neutron-linuxbridge-agent.service

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 23.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 24.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/905537

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/905538

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/905538
Committed: https://opendev.org/openstack/neutron/commit/73f9a31fa16cd75a6ac3eb17765c8335069156f5
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 73f9a31fa16cd75a6ac3eb17765c8335069156f5
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Oct 16 00:09:50 2023 +0000

    "ebtables-nft" MAC rule deletion failing

    "ebtables-nft" is failing to delete the rule filtering by MAC address:
      Bridge chain: neutronMAC-test-veth024379, entries: 2, policy: DROP
      -i test-veth024379 --among-src fa:16:3e:47:87:0 -j RETURN
      -j DROP

    A workaround for this issue, that works with both "ebtables-nft" and
    "ebtables-legacy", is to flush the table and recreate the DROP rule.
    The MAC spoofing tables have two rules: the one filtering by MAC address
    and the default DROP rule. This workaround has the same effect as just
    deleting the filtering rule.

    Closes-Bug: #2038541
    Change-Id: I38bd016c35d7a76d88c6eceec797d1cea84c45d1
    (cherry picked from commit 1879d925330af5598a105a8893ab6cfda9dc37e6)
    (cherry picked from commit 7dbd06d66e4daebab90e4d334ae43013580e555a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/905537
Committed: https://opendev.org/openstack/neutron/commit/732d7976d6c16ec37b4ec4908716a9d28ea64281
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 732d7976d6c16ec37b4ec4908716a9d28ea64281
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Oct 16 00:09:50 2023 +0000

    "ebtables-nft" MAC rule deletion failing

    "ebtables-nft" is failing to delete the rule filtering by MAC address:
      Bridge chain: neutronMAC-test-veth024379, entries: 2, policy: DROP
      -i test-veth024379 --among-src fa:16:3e:47:87:0 -j RETURN
      -j DROP

    A workaround for this issue, that works with both "ebtables-nft" and
    "ebtables-legacy", is to flush the table and recreate the DROP rule.
    The MAC spoofing tables have two rules: the one filtering by MAC address
    and the default DROP rule. This workaround has the same effect as just
    deleting the filtering rule.

    Closes-Bug: #2038541
    Change-Id: I38bd016c35d7a76d88c6eceec797d1cea84c45d1
    (cherry picked from commit 1879d925330af5598a105a8893ab6cfda9dc37e6)
    (cherry picked from commit 7dbd06d66e4daebab90e4d334ae43013580e555a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron xena-eom release.