enable upstream python testsuite in autopkgtests

[Alex Murray]

Currently the upstream python-based testsuite for nftables is disabled in the autopkgtests in debian/tests/control as follows:

# Disable test until we decide what to do with the nftables python module
#Tests: internaltest-py.sh
#Depends: @, python
#Restrictions: needs-root, allow-stderr, isolation-container, build-needed

This should be enabled to ensure testing is as comprehensive as this is in the process of getting promoted to main.

[Steve Beattie (sbeattie)]

So this looks okay, there are unfortunately a bunch of errors in the tests with v1.0.2 against a 5.15 kernel because the 'egress' hook support was only added in 5.16 (https://git.kernel.org/linus/42df6e1d221dddc0f2acf2be37e68d553ad65f96). This results in the following output in a jammy VM:

  96 test files, 69 files passed, 1889 unit tests,
  717 error, 0 warning

I'd like to suggest one additional change, adding a `-f` option to the nft-test.py invocation in debian/tests/internaltest-py.sh . From the README in test/py/ that tells the testsuite to carry on testing additional families in case of error. With that change in place, I get the following total results, again in a jammy VM with a 5.15 kernel:

  96 test files, 69 files passed, 1889 unit tests,
  5776 total executed, 719 error, 0 warning

I have not run this in adt, but running the tests manually doesn't return an error value on exit(), even with 700+ errors, so I'm not sure how the adt test will detect a regression that causes more test cases to error.

[Steve Beattie (sbeattie)]

(If this were a build time testsuite, our log comparison process would pick up changes. We could *maybe* do something akin to how we try to detect new failing tests in openjdk in qrt's notes_testing/openjdk/ where we maybe compare our current adt runs of nftables against a prior run, and look for differences.)

[Alex Murray (alexmurray)]

Turns out I wasn't looking closely enough at the autopkgtest logs from my local testing - I was trying to run the tests with the pre-built binary packages and whilst it would indicate the internaltest-py.sh tests were passing, they were actually completely failing without any error indication:

autopkgtest [19:10:18]: test internaltest-py.sh: [-----------------------
The nftables library at 'src/.libs/libnftables.so.1' does not exist. You need to build the project.
autopkgtest [19:10:19]: test internaltest-py.sh: -----------------------]
autopkgtest [19:10:20]: test internaltest-py.sh: - - - - - - - - - - results - - - - - - - - - -
internaltest-py.sh PASS

So I think clearly this is a lot less straightforward than I originally thought and perhaps we shouldn't bother with it at this stage - also since as you say since it doesn't fail the autopkgtest run itself then is there any point trying to enable these tests as we won't be able to use it to easily detect regressions in that case.

[Alex Murray (alexmurray)]

Attaching the updated debdiff in case we do decide we want this (even in the broken state)