Fix shell test suite

[Andreas Hasenack]

[ Impact ]

The shell test suite (iptables/tests/shell/run-tests.sh) is currently failing on the firewalld tests in focal only:

W: [FAILED] ././testcases/firewalld-restore/0001-firewalld_0: expected 0 but got 1
W: [FAILED] ././testcases/firewalld-restore/0002-firewalld-restart_0: expected 0 but got 1

After some troubleshooting, it turns out this is happening because of an unsorted order in the output of iptables-save, which was fixed[1] in later releases of iptables. The code was trying to compensate for that, but there was a small mistake[2] in a case/esac globbing:

case "$XT_MULTI" in
-*/xtables-nft-multi)
+*xtables-nft-multi)

The upstream fix includes other similar changes in other tests, but in the case of focal, the above is the minimal fix needed.

Note that this shell test suite is not being run in focal, just in later ubuntu releases. But since the fix for #1992454 is adding such a test, I decided to fix the shell test run and add it to the existing DEP8 tests for focal via this bug, so we have test parity between focal and later ubuntu releases.

1. https://git.netfilter.org/iptables/commit/?id=e28cf12cf50b9e2e0114f04331635fc122cb8aef
2. https://git.netfilter.org/iptables/commit/?id=2b2b7948c1960ba4680677664ff58477be869de6

[ Test Plan ]
Verify that the DEP8 tests now include a run-tests.sh test suite, and that it passes.

[ Where problems could occur ]
If the fix is incorrect, it would affect only the already-failing firewalld test. But in addition to fixing that test, we are now also including a full test run of all shell tests, something which wasn't being done for focal until now. While these tests are passing now, they could fail in a future iptables SRU, or turn out to be flaky. They are being run in ubuntu releases after focal, though, so that is a good sign.

[ Other Info ]
This fix is being included in the same upload as bug #1992454.

[ Original Description ]

The shell test suite (iptables/tests/shell/run-tests.sh) is currently failing on the firewalld tests:

W: [FAILED] ././testcases/firewalld-restore/0001-firewalld_0: expected 0 but got 1
W: [FAILED] ././testcases/firewalld-restore/0002-firewalld-restart_0: expected 0 but got 1

After some troubleshooting, it turns out this is happening because of an unsorted order in the output of iptables-save, which was fixed[1] in later releases of iptables. The code was trying to compensate for that, but there was a small mistake[2] in a case/esac globbing:

case "$XT_MULTI" in
-*/xtables-nft-multi)
+*xtables-nft-multi)

1. https://git.netfilter.org/iptables/commit/?id=e28cf12cf50b9e2e0114f04331635fc122cb8aef
2. https://git.netfilter.org/iptables/commit/?id=2b2b7948c1960ba4680677664ff58477be869de6

[Timo Aaltonen (tjaalton)]

Hello Andreas, or anyone else affected,

Accepted iptables into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/iptables/1.8.4-3ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted iptables (1.8.4-3ubuntu2.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

nova/2:21.2.4-0ubuntu2.3 (armhf)
systemd/245.4-4ubuntu3.21 (armhf, ppc64el, s390x)
ufw/0.36-6ubuntu1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#iptables

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Andreas Hasenack (ahasenack)]

Focal amd64 dep8 run: https://autopkgtest.ubuntu.com/results/autopkgtest-focal/focal/amd64/i/iptables/20230516_045413_de974@/log.gz

It now contains a new dep8 test, labeled command17, which runs run-tests.sh:

autopkgtest [04:53:52]: test command17: chmod +x ./iptables/tests/shell/testcases/chain/0006rename-segfault_0; cd iptables/tests/shell; ./run-tests.sh --host

Which runs the shell test suite, including the test for bug #1992454:

autopkgtest [04:53:52]: test command17: chmod +x ./iptables/tests/shell/testcases/chain/0006rename-segfault_0; cd iptables/tests/shell; ./run-tests.sh --host
autopkgtest [04:53:52]: test command17: [-----------------------

I: [EXECUTING] ././testcases/arptables/0001-arptables-save-restore_0
I: [OK] ././testcases/arptables/0001-arptables-save-restore_0
(...)
I: [EXECUTING] ././testcases/chain/0006rename-segfault_0
I: [OK] ././testcases/chain/0006rename-segfault_0
(...)

And this suite passes:
(...)
I: [EXECUTING] ././testcases/nft-only/0003delete-with-comment_0
I: [OK] ././testcases/nft-only/0003delete-with-comment_0
I: nft results: [OK] 41 [FAILED] 0 [TOTAL] 41
I: combined results: [OK] 82 [FAILED] 0 [TOTAL] 82
autopkgtest [04:54:00]: test command17: -----------------------]
autopkgtest [04:54:00]: test command17: - - - - - - - - - - results - - - - - - - - - -
command17 PASS

Focal verification succeeded.

[Launchpad Janitor (janitor)]

This bug was fixed in the package iptables - 1.8.4-3ubuntu2.1

---------------
iptables (1.8.4-3ubuntu2.1) focal; urgency=medium

  [ Louis Bouchard ]
  * d/p/0001-libiptc-Fix-for-segfault-when-renaming-a-chain.patch: apply
    upstream fix for segfault when renaming a chain. Includes a test
    case (LP: #1992454)

  [ Andreas Hasenack ]
  * Fix shell tests, and run them as DEP8 (LP: #2019023):
    - d/p/0501-fix-firewalld-shell-test.patch: fix the globbing expression
      which was used to decide whether to sort the restore output or not
      before comparing it to the good case
    - d/t/control: run all shell tests as DEP8

 -- Andreas Hasenack <email address hidden> Tue, 09 May 2023 15:39:57 -0300

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for iptables has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.