autopkgtest with needs-internet restriction fails

[Balint Reczey]

From libreswan debian/tests/control:

Tests: opportunistic
Restrictions: needs-root, isolation-machine, needs-internet, skippable
Depends:
...

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/amd64/libr/libreswan/20200824_154023_1e390@/log.gz

autopkgtest [15:33:30]: test opportunistic: [-----------------------
PING oe.libreswan.org (193.110.157.124) 56(84) bytes of data.

--- oe.libreswan.org ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4075ms

autopkgtest [15:33:45]: test opportunistic: -----------------------]
autopkgtest [15:33:45]: test opportunistic: - - - - - - - - - - results - - - - - - - - - -
opportunistic FAIL non-zero exit status 1

This test used to be skipped:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/amd64/libr/libreswan/20200810_200417_6d4ae@/log.gz

...
opportunistic SKIP unknown restriction needs-internet
...

I think if the CI infra does not provide unfiltered Internet access it should skip tests with needs-internet

[Julian Andres Klode (juliank)]

We don't have a separate needs-web restriction, and most tests do actually need http, I'm not convinced skipping tests with needs-internet is worthwhile.

[Łukasz Zemczak (sil2100)]

This should no longer be an issue, as I see this has been addressed by the latest two libreswan uploads:

https://launchpad.net/ubuntu/+source/libreswan/3.32-3ubuntu1
https://launchpad.net/ubuntu/+source/libreswan/3.32-3ubuntu2

[Balint Reczey (rbalint)]

@sil2100 Thanks! IMO this is still an issue with the test infra.

[Balint Reczey (rbalint)]

@juliank As I understand the definition of the needs-internet restriction Ubuntu's CI does not provide that and still running those small number of potentially tests is worse than loosing the result by not executing them.

https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst

needs-internet
The test needs unrestricted internet access, e.g. to download test data
that's not shipped as a package, or to test a protocol implementation
against a test server. Please also see the note about Network access later
in this document.
...
Network access

autopkgtest needs access to the network at least for downloading test dependencies and possibly dist-upgrading testbeds. In environments with restricted internet access you need to set up an apt proxy and configure the testbed to use it. (Note that the standard tools like autopkgtest-build-lxc or mk-sbuild automatically use the apt proxy from the host system.)

In general, tests should not access the internet themselves. If a test does use the internet outside of the pre-configured apt domain, the test must be marked with the needs-internet restriction. Using the internet usually makes tests less reliable, so this should be kept to a minimum. But for many packages their main purpose is to interact with remote web services and thus their testing should actually cover those too, to ensure that the distribution package keeps working with their corresponding web service.

Please note that for Debian, the ftp-master have ruled (in their REJECT-FAQ (Non-Main II) that tests must not execute code they download. In particular, tests must not use external repositories to depend on software (as opposed to data) that is not in Debian. However, currently there is nothing preventing this.

Debian's production CI infrastructure allows unrestricted network access on most workers. Tests with needs-internet can be skipped on some to avoid flaky behavior. In Ubuntu's infrastructure access to sites other than *.ubuntu.com and *.launchpad.net happens via a proxy (limited to DNS and http/https).

[Steve Langasek (vorlon)]

> @juliank As I understand the definition of the needs-internet
> restriction Ubuntu's CI does not provide that and still running
> those small number of potentially tests is worse than loosing
> the result by not executing them.

We have a large number of tests today that access the Internet, *successfully*, when running on the Ubuntu infrastructure. If all these tests were to be annotated as needs-internet, which seems to be correct from an upstream perspective, it would not be a small number of tests affected. So I also disagree with changing the infrastructure to skip tests tagged needs-internet.