new nftables 0.9.8-3 breaks firewalld 0.9.3 autopkgtest

[Christian Ehrhardt ]

The new nftables
  https://launchpad.net/ubuntu/+source/nftables/0.9.8-3
is stuck in proposed since it fails autopkgtest of firewalld
  https://autopkgtest.ubuntu.com/packages/f/firewalld/impish/amd64
  https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/amd64/f/firewalld/20210510_135128_36f9c@/log.gz
  https://autopkgtest.ubuntu.com/packages/f/firewalld/impish/s390x
  https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/s390x/f/firewalld/20210510_131115_faeb7@/log.gz

It fails the same way across architectures in:
## ------------------------ ##
## Summary of the failures. ##
## ------------------------ ##
Failed tests:
firewalld 0.9.3 test suite test groups:

 NUM: FILE-NAME:LINE TEST-GROUP-NAME
      KEYWORDS

  97: icmp_block_in_forward_chain.at:1 ICMP block present FORWARD chain
      nftables icmp
 124: rhbz1855140.at:1 rich rule icmptypes with one family
      nftables rich icmp rhbz1855140

The upstream issue tracker
  https://github.com/firewalld/firewalld/issues?q=is%3Aissue+is%3Aopen
does not list those cases, but there is a new v9.4.0 that we might try.

In Debian this isn't showing up
  https://ci.debian.net/packages/f/firewalld/
Because they are all Skipped for not having machine level isolation
  https://ci.debian.net/data/autopkgtest/testing/amd64/f/firewalld/13738304/log.gz

In detail it seems there re two cases of expected-output-mismatch
in #97:
-icmp type destination-unreachable icmp code host-prohibited reject with icmpx type admin-prohibited
+icmp code host-prohibited reject with icmpx type admin-prohibited
in #124:
-icmpv6 type parameter-problem icmpv6 code no-route mark set mark & 0x00000086 ^ 0x00000086
+icmpv6 code no-route mark set mark & 0x00000086 ^ 0x00000086

Those look like they might have the same root cause.

It seems that this is present for a while, this is nftables nftables/0.9.8-1 in Hirsute half a year ago.
  https://autopkgtest.ubuntu.com/results/autopkgtest-hirsute/hirsute/amd64/f/firewalld/20210118_230221_66bea@/log.gz

before nftables 0.9.8 it worked on 0.9.7-1:
https://autopkgtest.ubuntu.com/results/autopkgtest-hirsute/hirsute/amd64/f/firewalld/20201101_064747_2b123@/log.gz

With the right keywords I've found closed bugs in firewalld pointing to a nftables fix:
- https://github.com/firewalld/firewalld/issues/752 (thanks Costamagna/Michael for filing)
- https://marc.info/?l=netfilter-devel&m=161221629204555&w=2 <- supposed to be the fix

The issue is locally reproducible in e.g. autopkgtest VM and thereby
fixes can be tested the same way.

[Christian Ehrhardt  (paelzer)]

Fail #1 - stdout mismatch

# -*- compilation -*-
97. icmp_block_in_forward_chain.at:1: testing ICMP block present FORWARD chain ...
./icmp_block_in_forward_chain.at:1: if ! cp "${FIREWALLD_DEFAULT_CONFIG}/firewalld.conf" ./firewalld.conf; then exit 77; fi
./icmp_block_in_forward_chain.at:1: sed -i 's/^CleanupOnExit.*/CleanupOnExit=no/' ./firewalld.conf
./icmp_block_in_forward_chain.at:1: sed -i 's/^FirewallBackend.*/FirewallBackend=nftables/' ./firewalld.conf
./icmp_block_in_forward_chain.at:1: ip netns add fwd-test-${at_group_normalized}
./icmp_block_in_forward_chain.at:1: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} nft -f ./nft_rule_index.nft
./icmp_block_in_forward_chain.at:1: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} rm ./nft_rule_index.nft
./icmp_block_in_forward_chain.at:1: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} nft delete table inet firewalld_check_rule_index
not running
running
./icmp_block_in_forward_chain.at:4: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} firewall-cmd -q --zone=public --add-icmp-block=host-prohibited
./icmp_block_in_forward_chain.at:6: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} sh <<-"HERE"
    { { { { nft $NFT_NUMERIC_ARGS list chain inet firewalld filter_IN_public_deny | sed -e 's/icmp code 10/icmp code host-prohibited/'; echo $? >&3; } | sed -e 's/^[ \t]*//' -e 's/[ \t]*$//' | sed -e '/^[ \t]*$/d' | sed -e 's/[ \t]\+/ /g' | { printf "%s" "$(cat /dev/stdin)"; echo; } | sed -e 's/meta mark/mark/g' -e '/type.*hook.*priority.*policy.*/d' -e '/ct \(state\|status\)/{s/\(ct \(state\|status\)\) {/\1/g; s/ }//; s/\([a-z]*\), /\1,/g;}' >&4; } 3>&1; } | { read RC; exit $RC; } } 4>&1
HERE

--- - 2021-05-10 12:53:27.518505435 +0000
+++ /tmp/autopkgtest.b8ayAF/build.gyK/src/src/tests/testsuite.dir/at-groups/97/stdout 2021-05-10 12:53:27.513423335 +0000
@@ -1,6 +1,6 @@
 table inet firewalld {
 chain filter_IN_public_deny {
-icmp type destination-unreachable icmp code host-prohibited reject with icmpx type admin-prohibited
+icmp code host-prohibited reject with icmpx type admin-prohibited
 }
 }

97. icmp_block_in_forward_chain.at:1: 97. ICMP block present FORWARD chain (icmp_block_in_forward_chain.at:1): FAILED (icmp_block_in_forward_chain.at:6)

[Christian Ehrhardt  (paelzer)]

Fail #2 - stdout mismatch

# -*- compilation -*-
124. rhbz1855140.at:1: testing rich rule icmptypes with one family ...
./rhbz1855140.at:1: if ! cp "${FIREWALLD_DEFAULT_CONFIG}/firewalld.conf" ./firewalld.conf; then exit 77; fi
./rhbz1855140.at:1: sed -i 's/^CleanupOnExit.*/CleanupOnExit=no/' ./firewalld.conf
./rhbz1855140.at:1: sed -i 's/^FirewallBackend.*/FirewallBackend=nftables/' ./firewalld.conf
./rhbz1855140.at:1: ip netns add fwd-test-${at_group_normalized}
./rhbz1855140.at:1: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} nft -f ./nft_rule_index.nft
./rhbz1855140.at:1: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} rm ./nft_rule_index.nft
./rhbz1855140.at:1: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} nft delete table inet firewalld_check_rule_index
not running
running
./rhbz1855140.at:4: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} firewall-cmd --permanent --zone public --add-rich-rule='rule icmp-type name="echo-request" accept'
stdout:
success
./rhbz1855140.at:5: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} firewall-cmd --permanent --zone public --add-rich-rule='rule icmp-type name="neighbour-advertisement" accept'
stdout:
success
./rhbz1855140.at:6: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} firewall-cmd --permanent --zone public --add-rich-rule='rule icmp-type name="timestamp-request" accept'
stdout:
success
./rhbz1855140.at:7: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} firewall-cmd --permanent --zone public --add-rich-rule 'rule icmp-type name=bad-header mark set=0x86/0x86'
stdout:
success
./rhbz1855140.at:8: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} firewall-cmd -q --reload
./rhbz1855140.at:8: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} firewall-cmd -q --state
./rhbz1855140.at:9: env DBUS_SYSTEM_BUS_ADDRESS="unix:abstract=firewalld-testsuite-dbus-system-socket-${at_group_normalized}" ip netns exec fwd-test-${at_group_normalized} sh <<-"HERE"
    { { { { nft $NFT_NUMERIC_ARGS list chain inet firewalld mangle_PRE_public_allow; echo $? >&3; } | sed -e 's/^[ \t]*//' -e 's/[ \t]*$//' | sed -e '/^[ \t]*$/d' | sed -e 's/[ \t]\+/ /g' | { printf "%s" "$(cat /dev/stdin)"; echo; } | sed -e 's/me...

[Christian Ehrhardt  (paelzer)]

The fix is upstream accepted by now:
https://git.netfilter.org/nftables/commit/?id=533565244d88a818d8828ebabd7625e5a8a4c374
And it is released as part of v0.9.9

0.9.9 might change a bunch of other things we are not prepared for, so for the time being I'd try to resolve this regression in a backport of just this fix to then pick up 0.9.9 together with Debian after the release-freeze is lifted.

Prepping an MP and PPA ...

P.S. or if ignoring the additional risks we can try the 0.9.9-1~exp1 from Debian/experimental

[Christian Ehrhardt  (paelzer)]

After confirming the build and effect of the fix in PPA
  https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4626/+packages

I recreated the same in a VM and isolated the broken tests.
Then I updated to the PPA and with these tests I was able to confirm that they now pass.
 97: ICMP block present FORWARD chain ok
124: rich rule icmptypes with one family ok

In addition I think this is worth to consider for Debian as well, so I filed
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991309

Since we can't be sure if/when Debian takes that and on the other hand we are already behind a bit (still on 0.9.7) I think we should upload the same as delta in a 0.9.8-3ubuntu1.
Hereby I attach the debdiff to someone to have a quick look at.

We can still sync over an 0.9.8-4 if Debian accepts the suggestion.
That should unblock the case for now in Impish.

[Graham Inggs (ginggs)]

fix-lp-1936902-impish.debdiff looks good to me

[Christian Ehrhardt  (paelzer)]

I re-ran the full test - it really seems good now:
  242 tests were successful.
  2 tests were skipped.

Thanks Graham for the second pair of eyes on this.
I Uploaded it to Impish and hope to unblock nftables by that.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nftables - 0.9.8-3ubuntu1

---------------
nftables (0.9.8-3ubuntu1) impish; urgency=medium

  * d/p/lp-1936902-payload-check-icmp-dependency-before-removing-previo.patch
    Fix a regression in nftables 0.9.8 that made nftables too greedy
    in removing icmp dependencies (LP: #1936902).

 -- Christian Ehrhardt <email address hidden> Tue, 20 Jul 2021 10:01:47 +0200