The tests with random parameter cannot run

[zhongjun]

Steps to reproduce:
1. I write the following code in test.py
2. When I run "ostestr test" command, I can only run 1 test,
3. When I run 'ostestr -n test'.I can run full tests (2 tests).

I don't know what's worry about this?

########################## 1. code in test.py ########################################
import random
import ddt

from manila import test

@ddt.ddt
class RandomTestCase(test.TestCase):

    def setUp(self):
        super(self.__class__, self).setUp()

    @ddt.data('fixed', random.randint(0, 20))
    def test_with_random_parameter(self, name):
        pass

############################ 2. run "ostestr test" ######################################
# ostestr test
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \
OS_LOG_CAPTURE=${OS_LOG_CAPTURE:-1} \
${PYTHON:-python} -m subunit.run discover -t ./ ./manila/tests --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \
OS_LOG_CAPTURE=${OS_LOG_CAPTURE:-1} \
${PYTHON:-python} -m subunit.run discover -t ./ ./manila/tests --load-list /tmp/tmptQJ9aA
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \
OS_LOG_CAPTURE=${OS_LOG_CAPTURE:-1} \
${PYTHON:-python} -m subunit.run discover -t ./ ./manila/tests --load-list /tmp/tmpdyNk6v
{1} test.RandomTestCase.test_with_random_parameter_1_fixed [1.162648s] ... ok

======
Totals
======
Ran: 1 tests in 7.0000 sec.
 - Passed: 1
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 1.1626 sec.

==============
Worker Balance
==============
 - WARNING: missing Worker 0! Race in testr accounting.
 - Worker 1 (1 tests) => 0:00:01.162648

Slowest Tests:

Test id Runtime (s)
-------------------------------------------------------------------------------- -----------
test.RandomTestCase.test_with_random_parameter_1_fixed 1.163

########################### 3. run "ostestr -n test" ####################################

# ostestr -n test
{0} test.RandomTestCase.test_with_random_parameter_1_fixed [1.123144s] ... ok
{0} test.RandomTestCase.test_with_random_parameter_2_18 [0.011921s] ... ok

======
Totals
======
Ran: 2 tests in 1.0000 sec.
 - Passed: 2
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 1.1351 sec.

==============
Worker Balance
==============
 - Worker 0 (2 tests) => 0:00:01.135313

[zhongjun (jun-zhongjun)]

After discuss with clarkb, he gave some good explanation:
When we run "ostestr" command, the first step is --list (first process) then after partitioning that list into subsets each subset is given to a different process (second set of processes) so we import at least twice. The problem is importing multiple times in different processes produces different test names because rand int is different most of the time.

[Matthew Treinish (treinish)]

Right, the fundamental issue here is with the non determinism that you're adding to the test_id in that test suite. The test_id has to be a unique identifier for the test method. You'll have a lot of weird behavior when you have a generated test that has a random parameter. Python unittest expects that you can identify a test to be identified by a unique test_id for all parts of a run (which is what testr relies on)

So TL;DR the bug here is in your test suite by calling randint() in the test generator. If you want to use a random int as a test parameter I would do that inside the test method and have a static method name. Or if you really want to use ddt, use a deterministic input to the generator like range(20) instead. So you'll test all the parameters and the test ids will be consistent between runs.