Sometimes VMs requested to openstack are unreachable

[Paride Legovini]

This is well known issue, unfortunately. This shows as a timeout after 1800 at the very beginning of a test run. Example log:

https://objectstorage.prodstack5.canonical.com/swift/v1/AUTH_0f9aae918d5b4744bf7b827671c86842/autopkgtest-oracular/oracular/amd64/u/ubuntu-advantage-tools/20240524_034656_9100b@/log.gz

This requires investigation. This may be a fully infrastructure related issue; in this case we should come up with a reproducer not involving autopkgtest (script that creates/tests/destroys VMs, and which hangs after a while?) and ask IS to look into it.

I don't think we can fully rule out an issue with autopkgtest-cloud. My feeling (nothing more than that!) is that it may have something to do with security group lifecycle management.

[Bryce Harrington (bryce)]

There are a lot of tmpfail test failures; more than usual. Just spot checking but a portion seem to match the errors from this bug report's log file, e.g.:

# https://autopkgtest.ubuntu.com/results/autopkgtest-oracular/oracular/s390x/p/pythran/20240531_093453_cc6d6@/log.gz
693s No UUID given. Instance won't be deleted!
693s <VirtSubproc>: failure: setup script failed with code 1...
693s autopkgtest [09:34:52]: ERROR: testbed failure: unexpected eof from the testbed

Other tmpfails have different error messages, bit I suspect may be same root cause:

# https://autopkgtest.ubuntu.com/results/autopkgtest-oracular/oracular/armhf/r/rust-bytecheck/20240531_101049_0a27a@/log.gz
340s autopkgtest [10:10:49]: ERROR: testbed failure: timed out on command "sh -ec #!/bin/sh

In general, it seems that retriggering the tests once has been enough to get them to pass, but it's a bit labor intensive. If the tmpfails are indeed due to VM infrastructure issues it would be beneficial to get some analysis under way.