autopkgtest is flaky

[Balint Reczey]

The rtags-rc -w command is racy, see more details at:

https://github.com/Andersbakken/rtags/pull/1348

[Frank Heimes (fheimes)]

I'm currently fighting with the very last 'regression' (or better autopkgtest issue) with 'rtags' on 'amd64' that needs to be solved to unblock the 'zlib' focal-proposed migration, hence adding the tag 'update-excuse-focal'.

In the case I'm currently looking at, it seems to be safest to run the rtags autopkgtests with zlib/1:1.2.11.dfsg-2ubuntu1.4 and htslib/1.10.2-3ubuntu0.1, but even then a repeating failure occurs on amd64 in our test infrastructure:
https://autopkgtest.ubuntu.com/packages/r/rtags/focal/amd64

A timeout, but w/o much further details happens:
autopkgtest [07:42:53]: test run-test: [-----------------------
............autopkgtest [10:29:37]: ERROR: timed out on command "su -s /bin/bash ubuntu -c set -e; export USER=`id -nu`; . /etc/profile >/dev/null 2>&1 || true; . ~/.profile >/dev/null 2>&1 || true; buildtree="/tmp/autopkgtest.MokDZx/build.wN5/src"; mkdir -p -m 1777 -- "/tmp/autopkgtest.MokDZx/run-test-artifacts"; export AUTOPKGTEST_ARTIFACTS="/tmp/autopkgtest.MokDZx/run-test-artifacts"; export ADT_ARTIFACTS="$AUTOPKGTEST_ARTIFACTS"; mkdir -p -m 755 "/tmp/autopkgtest.MokDZx/autopkgtest_tmp"; export AUTOPKGTEST_TMP="/tmp/autopkgtest.MokDZx/autopkgtest_tmp"; export ADTTMP="$AUTOPKGTEST_TMP"; export DEBIAN_FRONTEND=noninteractive; export LANG=C.UTF-8; export DEB_BUILD_OPTIONS=parallel=1; unset LANGUAGE LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT LC_IDENTIFICATION LC_ALL;rm -f /tmp/autopkgtest_script_pid; set -C; echo $$ > /tmp/autopkgtest_script_pid; set +C; trap "rm -f /tmp/autopkgtest_script_pid" EXIT INT QUIT PIPE; cd "$buildtree"; export 'ADT_TEST_TRIGGERS=zlib/1:1.2.11.dfsg-2ubuntu1.4 htslib/1.10.2-3ubuntu0.1'; chmod +x /tmp/autopkgtest.MokDZx/build.wN5/src/debian/tests/run-test; touch /tmp/autopkgtest.MokDZx/run-test-stdout /tmp/autopkgtest.MokDZx/run-test-stderr; /tmp/autopkgtest.MokDZx/build.wN5/src/debian/tests/run-test 2> >(tee -a /tmp/autopkgtest.MokDZx/run-test-stderr >&2) > >(tee -a /tmp/autopkgtest.MokDZx/run-test-stdout);" (kind: test)
autopkgtest [10:29:38]: test run-test: -----------------------]
autopkgtest [10:29:38]: test run-test: - - - - - - - - - - results - - - - - - - - - -
run-test FAIL timed out
autopkgtest [10:29:38]: @@@@@@@@@@@@@@@@@@@@ summary
run-test FAIL timed out

Locally on my workstation these autopkgtests (on amd64) all succeed all the time in slightly different configurations (5x with LXD and 6x with VM), which makes it difficult to debug further.

And on my system run-test usually runs only for 2 or 3 seconds - not more - but the timeout above is after 2.5+h! (Which makes me thing about a infrastructural problem?!)

The tests run surprisingly smooth on armhf and s390x - even on our test infrastructure, but obviously caused problems in the past on arm64 (Ignored failure) and ppc64el (Not a regression), too.

It could be that the issue is caused by nose (python3-nose), since there are several reports that it can hang (esp. in multithreadding or parallel cases - which is however not used here.)

But rtags upstream was obviously unhappy with nose too and migrated from nose to pytest st...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Brian Murray (brian-murray)]

The rtags autopkgtest on amd64 for Focal ended up failing when run with a migration-reference/0 trigger. However, the fact that the test is passing for people locally and failing in the infrastructure is a bit worrisome so I think this still could use some investigation from the QA team.