Any tempest plugin import failure leads other plugin tests fails

[Ghanshyam Mann]

There is failure for all projects runs tempest plugin tests. That was due to fwaas tempest plugin having import error.

When testr try to list all tests and it also list the loaded plugin tests, if there is any import error in any plugin, it will leads failure to other plugin also.

I think plugin should work in isolated way, mean if Tempest detect any error from any plugin, it should just log the error and ignore that plugin tests and run others. Tempest does while registering conf options or loading plugin.

But we should have same way for import error also.

error- http://logs.openstack.org/50/289650/4/check/gate-congress-dsvm-api/6a27be7/console.html

2016-03-10 03:42:34.018 | all-plugin runtests: commands[1] | bash tools/pretty_tox.sh --concurrency=4 congress_tempest_tests
2016-03-10 03:42:36.139 | running testr
2016-03-10 03:42:39.664 | /usr/local/lib/python2.7/dist-packages/tempest_lib/__init__.py:28: DeprecationWarning: tempest-lib is deprecated for future bug-fixes and code changes in favor of tempest. Please change your imports from tempest_lib to tempest.lib
2016-03-10 03:42:39.664 | DeprecationWarning)
2016-03-10 03:42:39.916 | running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
2016-03-10 03:42:39.916 | OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
2016-03-10 03:42:39.916 | OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
2016-03-10 03:42:39.916 | OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
2016-03-10 03:42:39.917 | ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
2016-03-10 03:42:39.917 | --- imNon-zero exit code (2) from test listing.
2016-03-10 03:42:39.949 | perror: testr failed (3)o
2016-03-10 03:42:39.950 | rt errors ---
2016-03-10 03:42:39.983 | Failed to import test module: neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions
2016-03-10 03:42:39.985 | Traceback (most recent call last):
2016-03-10 03:42:39.989 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 456, in _find_test_path
2016-03-10 03:42:39.989 | module = self._get_module_from_name(name)
2016-03-10 03:42:39.989 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 395, in _get_module_from_name
2016-03-10 03:42:39.989 | __import__(name)
2016-03-10 03:42:39.989 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/api/test_fwaas_extensions.py", line 23, in <module>
2016-03-10 03:42:39.989 | from neutron_fwaas.tests.tempest_plugin.tests.api import base
2016-03-10 03:42:39.989 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/api/base.py", line 18, in <module>
2016-03-10 03:42:39.989 | from neutron_fwaas.tests.tempest_plugin.tests import fwaas_client
2016-03-10 03:42:39.989 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/fwaas_client.py", line 25, in <module>
2016-03-10 03:42:39.990 | from neutron_fwaas.tests.tempest_plugin.services import client
2016-03-10 03:42:39.990 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/services/client.py", line 18, in <module>
2016-03-10 03:42:39.990 | from tempest.services.network.json import base
2016-03-10 03:42:39.990 | ImportError: cannot import name base
2016-03-10 03:42:39.990 |
2016-03-10 03:42:39.990 | Failed to import test module: neutron_fwaas.tests.tempest_plugin.tests.scenario.test_fwaas
2016-03-10 03:42:39.990 | Traceback (most recent call last):
2016-03-10 03:42:39.990 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 456, in _find_test_path
2016-03-10 03:42:39.990 | module = self._get_module_from_name(name)
2016-03-10 03:42:39.990 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 395, in _get_module_from_name
2016-03-10 03:42:39.990 | __import__(name)
2016-03-10 03:42:39.990 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/scenario/test_fwaas.py", line 21, in <module>
2016-03-10 03:42:39.991 | from neutron_fwaas.tests.tempest_plugin.tests.scenario import base
2016-03-10 03:42:39.991 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/scenario/base.py", line 21, in <module>
2016-03-10 03:42:39.991 | from neutron_fwaas.tests.tempest_plugin.tests import fwaas_client
2016-03-10 03:42:39.991 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/fwaas_client.py", line 25, in <module>
2016-03-10 03:42:39.991 | from neutron_fwaas.tests.tempest_plugin.services import client
2016-03-10 03:42:39.991 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/services/client.py", line 18, in <module>
2016-03-10 03:42:39.991 | from tempest.services.network.json import base
2016-03-10 03:42:39.991 | ImportError: cannot import name base
2016-03-10 03:42:39.991 | The test run didn't actually run any tests

[Ghanshyam Mann (ghanshyammann)]

Actual issue is with fwaas plugin tests and fixed in https://review.openstack.org/#/c/290983/

after merging that, all tests should run fine.

[hongbin (hongbin034)]

This bug broke Magnum gate, so I mark it as Critical.

[Ghanshyam Mann (ghanshyammann)]

Putting this for fwaas and fixed. Invalid for other projects.

Tempest we will keep and let's see if we can have nice way to handle such issues.

[Joshua White (joshua-l-white)]

Set to In Progress, if you are no longer working on it please unassign and set to status accordingly.

[chandan kumar (chkumar246)]

Unassigning this bug as there is no activity from last 6 months. If you are still want to work on this bug, feel free to assign yourself.

[Martin Kopec (mkopec)]

This isn't for sure InProgress due to a long inactivity here. I'll mark it as Confirmed as we need to at least check if something like this can happen again - we should make sure that a failure in any of the installed plugins won't affect other installed plugins.

[Pragadeeswaran Sathyanarayanan (psathyan)]

In unittest, a test module that throws ImportError is marked as a _FailedTest with the prefix unittest.loader._FailedTest. In other words this is gracefully handled here.

By design, stestr throws an exception when it encounters an error in the loader or identifies test modules having ImportError.

[Pragadeeswaran Sathyanarayanan (psathyan)]

Marking this as invalid - please refer earlier comment