Any tempest plugin import failure leads other plugin tests fails

Bug #1555524 reported by Ghanshyam Mann
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ironic
Invalid
Undecided
Unassigned
Magnum
Invalid
Critical
Unassigned
neutron
Fix Released
Undecided
Unassigned
tempest
Invalid
Undecided
Pragadeeswaran Sathyanarayanan

Bug Description

There is failure for all projects runs tempest plugin tests. That was due to fwaas tempest plugin having import error.

When testr try to list all tests and it also list the loaded plugin tests, if there is any import error in any plugin, it will leads failure to other plugin also.

I think plugin should work in isolated way, mean if Tempest detect any error from any plugin, it should just log the error and ignore that plugin tests and run others. Tempest does while registering conf options or loading plugin.

But we should have same way for import error also.

error- http://logs.openstack.org/50/289650/4/check/gate-congress-dsvm-api/6a27be7/console.html

2016-03-10 03:42:34.018 | all-plugin runtests: commands[1] | bash tools/pretty_tox.sh --concurrency=4 congress_tempest_tests
2016-03-10 03:42:36.139 | running testr
2016-03-10 03:42:39.664 | /usr/local/lib/python2.7/dist-packages/tempest_lib/__init__.py:28: DeprecationWarning: tempest-lib is deprecated for future bug-fixes and code changes in favor of tempest. Please change your imports from tempest_lib to tempest.lib
2016-03-10 03:42:39.664 | DeprecationWarning)
2016-03-10 03:42:39.916 | running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
2016-03-10 03:42:39.916 | OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
2016-03-10 03:42:39.916 | OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
2016-03-10 03:42:39.916 | OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
2016-03-10 03:42:39.917 | ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
2016-03-10 03:42:39.917 | --- imNon-zero exit code (2) from test listing.
2016-03-10 03:42:39.949 | perror: testr failed (3)o
2016-03-10 03:42:39.950 | rt errors ---
2016-03-10 03:42:39.983 | Failed to import test module: neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions
2016-03-10 03:42:39.985 | Traceback (most recent call last):
2016-03-10 03:42:39.989 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 456, in _find_test_path
2016-03-10 03:42:39.989 | module = self._get_module_from_name(name)
2016-03-10 03:42:39.989 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 395, in _get_module_from_name
2016-03-10 03:42:39.989 | __import__(name)
2016-03-10 03:42:39.989 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/api/test_fwaas_extensions.py", line 23, in <module>
2016-03-10 03:42:39.989 | from neutron_fwaas.tests.tempest_plugin.tests.api import base
2016-03-10 03:42:39.989 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/api/base.py", line 18, in <module>
2016-03-10 03:42:39.989 | from neutron_fwaas.tests.tempest_plugin.tests import fwaas_client
2016-03-10 03:42:39.989 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/fwaas_client.py", line 25, in <module>
2016-03-10 03:42:39.990 | from neutron_fwaas.tests.tempest_plugin.services import client
2016-03-10 03:42:39.990 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/services/client.py", line 18, in <module>
2016-03-10 03:42:39.990 | from tempest.services.network.json import base
2016-03-10 03:42:39.990 | ImportError: cannot import name base
2016-03-10 03:42:39.990 |
2016-03-10 03:42:39.990 | Failed to import test module: neutron_fwaas.tests.tempest_plugin.tests.scenario.test_fwaas
2016-03-10 03:42:39.990 | Traceback (most recent call last):
2016-03-10 03:42:39.990 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 456, in _find_test_path
2016-03-10 03:42:39.990 | module = self._get_module_from_name(name)
2016-03-10 03:42:39.990 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 395, in _get_module_from_name
2016-03-10 03:42:39.990 | __import__(name)
2016-03-10 03:42:39.990 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/scenario/test_fwaas.py", line 21, in <module>
2016-03-10 03:42:39.991 | from neutron_fwaas.tests.tempest_plugin.tests.scenario import base
2016-03-10 03:42:39.991 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/scenario/base.py", line 21, in <module>
2016-03-10 03:42:39.991 | from neutron_fwaas.tests.tempest_plugin.tests import fwaas_client
2016-03-10 03:42:39.991 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/fwaas_client.py", line 25, in <module>
2016-03-10 03:42:39.991 | from neutron_fwaas.tests.tempest_plugin.services import client
2016-03-10 03:42:39.991 | File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/services/client.py", line 18, in <module>
2016-03-10 03:42:39.991 | from tempest.services.network.json import base
2016-03-10 03:42:39.991 | ImportError: cannot import name base
2016-03-10 03:42:39.991 | The test run didn't actually run any tests

Changed in tempest:
assignee: nobody → Ghanshyam Mann (ghanshyammann)
Revision history for this message
Ghanshyam Mann (ghanshyammann) wrote :

Actual issue is with fwaas plugin tests and fixed in https://review.openstack.org/#/c/290983/

after merging that, all tests should run fine.

Revision history for this message
hongbin (hongbin034) wrote :

This bug broke Magnum gate, so I mark it as Critical.

Changed in magnum:
importance: Undecided → Critical
hongbin (hongbin034)
Changed in magnum:
status: New → Fix Committed
Revision history for this message
Ghanshyam Mann (ghanshyammann) wrote :

Putting this for fwaas and fixed. Invalid for other projects.

Tempest we will keep and let's see if we can have nice way to handle such issues.

Changed in ironic:
status: New → Invalid
Changed in magnum:
status: Fix Committed → Invalid
Changed in neutron:
status: New → Fix Committed
Revision history for this message
Joshua White (joshua-l-white) wrote :

Set to In Progress, if you are no longer working on it please unassign and set to status accordingly.

Changed in tempest:
status: New → In Progress
Revision history for this message
chandan kumar (chkumar246) wrote :

Unassigning this bug as there is no activity from last 6 months. If you are still want to work on this bug, feel free to assign yourself.

Changed in tempest:
assignee: Ghanshyam Mann (ghanshyammann) → nobody
Revision history for this message
Martin Kopec (mkopec) wrote :

This isn't for sure InProgress due to a long inactivity here. I'll mark it as Confirmed as we need to at least check if something like this can happen again - we should make sure that a failure in any of the installed plugins won't affect other installed plugins.

Changed in tempest:
status: In Progress → Confirmed
Changed in neutron:
status: Fix Committed → Fix Released
Revision history for this message
Pragadeeswaran Sathyanarayanan (psathyan) wrote :

In unittest, a test module that throws ImportError is marked as a _FailedTest with the prefix unittest.loader._FailedTest. In other words this is gracefully handled here.

By design, stestr throws an exception when it encounters an error in the loader or identifies test modules having ImportError.

Revision history for this message
Pragadeeswaran Sathyanarayanan (psathyan) wrote :

Marking this as invalid - please refer earlier comment

Changed in tempest:
assignee: nobody → Pragadeeswaran Sathyanarayanan (psathyan)
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.