Non-zero exit code (2) from test listing

[Andriy Yurchuk]

When trying to run a specific tempest test suite I get this error:

 ~ $ testr run tempest.api.compute.servers.test_create_server.ServersTestJSON
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./tempest/test_discover} --list
Non-zero exit code (2) from test listing. stdout='stdout='\xb3)\x01@<email address hidden>test_api_discovery.TestApiDiscovery.test_api_versions[gate,smoke]\xf1\xb5\x19t\xb3)\x01@<email address hidden>test_api_discovery.TestApiDiscovery.test_default_version[gate,smoke]HF\xf2T\xb3)\x01@i@^tempest.api.baremetal.test_api_discovery.TestApiDiscovery.test_version_1_resources[gate,smoke]\xa4}\x93Z\xb3)\x01@<email address hidden>test_chassis.TestChassis.test_create_chassis[gate,smoke]r\xa1\xb8\x93\xb3)\x01@<email address hidden>test_chassis.TestChassis.test_create_chassis_unicode_description[gate,smoke]\xdb\xff>\xf9\xb3)\x01@<email address hidden>test_chassis.TestChassis.test_delete_chassis[gate,smoke]oQ\xe0y\xb3)\x01@W@L ... ... ... ...
(followed by a long list of tests)

When I run the whole set of tempest tests (i.e. 'testr run') it works fine. The above error is thrown by 'testr list-tests' command so if you try to run 'testr list-tests' you'll get the same error.

[David Kranz (david-kranz)]

I don't see this with a fresh tempest checkout:

[dkranz@localhost tempest]$ testr list-tests tempest.api.compute.servers.test_create_server.ServersTestJSON
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./tempest/test_discover} --list
tempest.api.compute.servers.test_create_server.ServersTestJSON.test_create_server_response[gate,smoke]
tempest.api.compute.servers.test_create_server.ServersTestJSON.test_host_name_is_same_as_server_name[gate]
tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers[gate,smoke]
tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers_with_detail[gate,smoke]
tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_created_server_vcpus[gate]
tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_server_details[gate,smoke]

[Andriy Yurchuk (ch00k)]

This is very strange. I could reproduce this on two different systems (OS X and Gentoo Linux) with a fresh virtualenv and freshly cloned tempest. Can we perhaps compare our pip list? This is mine:

anyjson (0.3.3)
Babel (1.3)
boto (2.24.0)
cliff (1.5.2)
cmd2 (0.6.7)
coverage (3.7.1)
docutils (0.9.1)
ecdsa (0.10)
extras (0.0.3)
fixtures (0.3.14)
flake8 (2.0)
hacking (0.8.0)
httplib2 (0.8)
iso8601 (0.1.8)
Jinja2 (2.7.2)
jsonpatch (1.3)
jsonpointer (1.1)
jsonschema (2.3.0)
keyring (3.3)
lxml (3.3.0)
MarkupSafe (0.18)
mccabe (0.2.1)
mock (1.0.1)
mox (0.5.3)
netaddr (0.7.10)
nose (1.3.0)
oslo.config (1.2.1)
oslo.sphinx (1.1)
paramiko (1.12.1)
pbr (0.6)
pep8 (1.4.5)
pip (1.4.1)
prettytable (0.7.2)
pycrypto (2.6.1)
pyflakes (0.7.3)
Pygments (1.6)
pyOpenSSL (0.13.1)
pyparsing (2.0.1)
python-cinderclient (1.0.8)
python-glanceclient (0.12.0)
python-heatclient (0.2.6)
python-keystoneclient (0.5.1)
python-mimeparse (0.1.4)
python-neutronclient (2.3.3)
python-novaclient (2.15.0)
python-subunit (0.0.18)
python-swiftclient (1.8.0)
pytz (2013.9)
PyYAML (3.10)
requests (2.2.1)
setuptools (0.9.8)
simplejson (3.3.2)
six (1.5.2)
Sphinx (1.1.3)
stevedore (0.14.1)
testrepository (0.0.18)
testresources (0.2.7)
testscenarios (0.4)
testtools (0.9.35)
warlock (1.1.0)
wsgiref (0.1.2)

[David Kranz (david-kranz)]

anyjson (0.3.3)
Babel (1.3)
boto (2.13.3)
cliff (1.4.5)
cmd2 (0.6.7)
d2to1 (0.2.11)
distribute (0.6.28)
ecdsa (0.8)
eventlet (0.14.0)
extras (0.0.3)
fixtures (0.3.14)
gerritlib (0.2.7)
greenlet (0.4.1)
httplib2 (0.8)
iso8601 (0.1.4)
jsonpatch (1.1)
jsonpointer (1.0)
jsonschema (2.1.0)
keyring (3.0.5)
lxml (3.2.3)
netaddr (0.7.10)
nose (1.3.0)
oslo.config (1.2.1)
paramiko (1.12.0)
pbr (0.5.21)
pep8 (1.4.6)
prettytable (0.7.2)
py (1.4.17)
pycrypto (2.6)
pyOpenSSL (0.13.1)
pyparsing (2.0.1)
python-cinderclient (1.0.5)
python-glanceclient (0.11.0)
python-heatclient (0.2.5)
python-keystoneclient (0.3.2)
python-mimeparse (0.1.4)
python-neutronclient (2.3.1)
python-novaclient (2.15.0)
python-subunit (0.0.15)
pytz (2013d)
PyYAML (3.10)
requests (2.0.0)
simplejson (3.3.0)
six (1.4.1)
testrepository (0.0.17)
testresources (0.2.7)
testscenarios (0.4)
testtools (0.9.32)
tox (1.6.1)
virtualenv (1.10.1)
warlock (1.0.1)
wsgiref (0.1.2)

[Andriy Yurchuk (ch00k)]

I'm suspecting that this happens because some package in my environment is of newer version and is broken. Can you perhaps try to create a new virtualenv and install all the latest package versions (just run pip install -r requirements.txt && pip install -r test-requirements.txt)?

[Andriy Yurchuk (ch00k)]

I have just reproduced this on the third machine (Ubuntu). These are the steps I take:
1. git clone https://github.com/openstack/tempest.git
2. cd tempest
3. pip install -r requirements.txt
4. testr init
5. testr list-tests

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/72460

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/72568

[Andriy Yurchuk (ch00k)]

Upstream bug https://bugs.launchpad.net/subunit/+bug/1278539

[Joseph Lanoux (joseph-lanoux)]

I came into the same error and it's tricky. However, it's worth noticing that tempest.conf was updated in the lastest versions. Some parameters don't exist anymore for instance. Updating my tempest.conf to fit the latest tempest.conf.sample fixed this issue for me.

[Darragh O'Reilly (darragh-oreilly)]

I hit this today with neutron on a ubuntu 12.04 vm I build a few months ago. Updating the packages fixed it.

$ sudo apt-get update && sudo apt-get --yes dist-upgrade

reboot

$ cd /opt/stack/neutron
$ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

[Travis McPeak (travis-mcpeak)]

I'm seeing this even after updating all packages. I've also tried downgrading testtools and subunit, as described in the parent bug comments, and nothing fixes it.

[Travis McPeak (travis-mcpeak)]

Update: installing python-subunit==0.0.15 and testtools==0.9.34 in the venv fixed this for me.

[Chris St. Pierre (stpierre)]

Travis's solution worked for me, too.

[daya kamath (daya-k)]

i hit this one on a fresh fedora 20 VM. test-requirements.txt has python-subunit version at 0.0.18
travis's solution works for me too.

[Ryan Hsu (rhsu)]

I hit this on fresh Ubuntu 12.04 when running ./run_tests.sh -N -P on neutron. Travis' solution worked for me.

[Kanagaraj Manickam (kanagaraj-manickam)]

Awesome !! I also get rid this problem with Travis solution. Thanks a lot.

[Yaniv Kaul (yaniv-kaul)]

Did not work for me. CentOS 6.5.
Looks like https://bugs.launchpad.net/subunit/+bug/1278539 is a dup of this bug.
https://bugs.launchpad.net/subunit/+bug/1278539 was marked as dup of https://bugs.launchpad.net/testrepository/+bug/1271133 which is claimed to be fixed. I could not get it to work.
Downgraded to above mentioned versions - no go.
Upgraded Testrepository to 0.0.21, testtools to 1.0.0 and python-subunit to 0.0.20 - still, no go:

[root@lgdrm404 Tempest]# ./run_tempest.sh -N --serial tempest.api.volume*
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec. You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec. You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning)
Non-zero exit code (2) from test listing.

----------------------------------------------------------------------
Ran 0 tests in 4.496s

[Robert Collins (lifeless)]

@Yaniv it looks like you've lost stdout there - test loading has failed. What happens if you do
testr list-tests tempest.api.volume.*

?

[Robert Collins (lifeless)]

So I'm going to mark this invalid, its confusing to people as it stands.

Here is the summary:
If a test can't be imported, the error reported as the subject of this bug will be shown.
It should always list the test(s) that can't be imported. If it doesn't, *that* is a bug.

In future, when http://bugs.python.org/issue19746 is committed, we'll be able to show the actual import error for each of the tests as well, which will be a lot nicer for folk trying to debug things remotely.

[Robert Collins (lifeless)]

To debug things today, just get the module name from the testr output (or run testr list-tests to get that output locally) and then do
python -c 'import $namehere' to get the error.