TestGraphStuff.testBasicSortTests is flaky

[Free Ekanayaka]

It sometimes passes and sometimes not:

while python3 -m testtools.run testresources.tests.test_optimising_test_suite.TestGraphStuff.testBasicSortTests; do true; done

will typically spin for a few seconds and then crash.

This seems to happen only with Python 3.x.

See for example here:

https://travis-ci.org/testing-cabal/testresources/jobs/177479870

[Free Ekanayaka (free.ekanayaka)]

I tried again to run the above loop and this time I didn't get flakiness. Something across the stack might have changed in the meantime.

[Free Ekanayaka (free.ekanayaka)]

Oops, ignore the above comment, I was running the loop from within the branch containing the fix. The flakiness still occurs.

[Thomas Bechtold (toabctl)]

Same happens in the OpenBuildService. See the build results (which run the tests for python2 and python3) here: https://build.opensuse.org/package/show/devel:languages:python/python-testresources .
It fails with:

[ 45s] + PYTHONPATH=/home/abuild/rpmbuild/BUILDROOT/python-testresources-2.0.1-18.6.x86_64/usr/lib/python2.7/site-packages
[ 45s] + python2 -m testtools.run discover
[ 46s] Tests running...
[ 46s]
[ 46s] Ran 101 tests in 0.132s
[ 46s] OK
[ 46s] ++ '[' -f _current_flavor ']'
[ 46s] ++ cat _current_flavor
[ 46s] + python_flavor=python2
[ 46s] + '[' -z python2 ']'
[ 46s] + '[' python2 '!=' python3 ']'
[ 46s] + '[' -d build ']'
[ 46s] + mv build _build.python2
[ 46s] + '[' -d _build.python3 ']'
[ 46s] + mv _build.python3 build
[ 46s] + echo python3
[ 46s] + LANG=en_US.UTF-8
[ 46s] + PYTHONPATH=/home/abuild/rpmbuild/BUILDROOT/python-testresources-2.0.1-18.6.x86_64/usr/lib/python3.4/site-packages
[ 46s] + python3 -m testtools.run discover
[ 47s] Tests running...
[ 47s] ======================================================================
[ 47s] FAIL: testresources.tests.test_optimising_test_suite.TestGraphStuff.testBasicSortTests
[ 47s] ----------------------------------------------------------------------
[ 47s] Traceback (most recent call last):
[ 47s] File "/home/abuild/rpmbuild/BUILD/testresources-2.0.1/testresources/tests/test_optimising_test_suite.py", line 527, in testBasicSortTests
[ 47s] "failed with permutation %s" % (permutation,))
[ 47s] File "/usr/lib/python3.4/site-packages/testtools/testcase.py", line 417, in assertIn
[ 47s] self.assertThat(haystack, Contains(needle), message)
[ 47s] File "/usr/lib/python3.4/site-packages/testtools/testcase.py", line 498, in assertThat
[ 47s] raise mismatch_error
[ 47s] testtools.matchers._impl.MismatchError: [test_three, test_one, test_two, test_four] not in [[test_one, test_two, test_three, test_four], [test_three, test_two, test_one, test_four]]: failed with permutation [test_one, test_two, test_three, test_four]
[ 47s]
[ 47s] Ran 101 tests in 0.142s
[ 47s] FAILED (failures=1)
[ 47s] error: Bad exit status from /var/tmp/rpm-tmp.p4tsfj (%check)