Manila jobs with neutron-dynamic-routing enabled are failing

[Goutham Pacha Ravi]

openstack/neutron-dynamic-routing allows manila gate jobs (ex: manila-tempest-minimal-dsvm-lvm-centos-7) to set up dynamic routing with BGP for testing IPv6 connectivity to storage backends and IPv6 access control to clients.

These jobs have been failing with the following errors:

2018-05-18 10:53:42.956780 | 2018-05-18 10:53:42.956 | ++ /opt/stack/new/manila/contrib/ci/post_test_hook.sh:source:360 : sudo -H -u jenkins tempest run -r manila_tempest_tests.tests.scenario --concurrency=1
2018-05-18 10:53:52.495927 | 2018-05-18 10:53:52.495 |
2018-05-18 10:53:52.497727 | 2018-05-18 10:53:52.497 | =========================
2018-05-18 10:53:52.499316 | 2018-05-18 10:53:52.498 | Failures during discovery
2018-05-18 10:53:52.501041 | 2018-05-18 10:53:52.500 | =========================
2018-05-18 10:53:52.503444 | 2018-05-18 10:53:52.502 | --- import errors ---
2018-05-18 10:53:52.505581 | 2018-05-18 10:53:52.505 | Failed to import test module: neutron_dynamic_routing.tests.tempest.api.test_bgp_speaker_extensions
2018-05-18 10:53:52.507203 | 2018-05-18 10:53:52.506 | Traceback (most recent call last):
2018-05-18 10:53:52.508986 | 2018-05-18 10:53:52.508 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 456, in _find_test_path
2018-05-18 10:53:52.510602 | 2018-05-18 10:53:52.509 | module = self._get_module_from_name(name)
2018-05-18 10:53:52.512199 | 2018-05-18 10:53:52.511 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 395, in _get_module_from_name
2018-05-18 10:53:52.513805 | 2018-05-18 10:53:52.513 | __import__(name)
2018-05-18 10:53:52.515422 | 2018-05-18 10:53:52.514 | File "/opt/stack/new/neutron-dynamic-routing/neutron_dynamic_routing/tests/tempest/api/test_bgp_speaker_extensions.py", line 21, in <module>
2018-05-18 10:53:52.517391 | 2018-05-18 10:53:52.516 | from neutron_tempest_plugin.api import base
2018-05-18 10:53:52.519247 | 2018-05-18 10:53:52.518 | ImportError: No module named neutron_tempest_plugin.api
2018-05-18 10:53:52.523434 | 2018-05-18 10:53:52.521 |
2018-05-18 10:53:52.525248 | 2018-05-18 10:53:52.524 | Failed to import test module: neutron_dynamic_routing.tests.tempest.api.test_bgp_speaker_extensions_negative
2018-05-18 10:53:52.527395 | 2018-05-18 10:53:52.526 | Traceback (most recent call last):
2018-05-18 10:53:52.528828 | 2018-05-18 10:53:52.528 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 456, in _find_test_path
2018-05-18 10:53:52.530971 | 2018-05-18 10:53:52.530 | module = self._get_module_from_name(name)
2018-05-18 10:53:52.537862 | 2018-05-18 10:53:52.534 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 395, in _get_module_from_name
2018-05-18 10:53:52.539381 | 2018-05-18 10:53:52.538 | __import__(name)
2018-05-18 10:53:52.541241 | 2018-05-18 10:53:52.540 | File "/opt/stack/new/neutron-dynamic-routing/neutron_dynamic_routing/tests/tempest/api/test_bgp_speaker_extensions_negative.py", line 21, in <module>
2018-05-18 10:53:52.542803 | 2018-05-18 10:53:52.542 | from neutron_dynamic_routing.tests.tempest.api import test_bgp_speaker_extensions as test_base # noqa
2018-05-18 10:53:52.544197 | 2018-05-18 10:53:52.543 | File "/opt/stack/new/neutron-dynamic-routing/neutron_dynamic_routing/tests/tempest/api/test_bgp_speaker_extensions.py", line 21, in <module>
2018-05-18 10:53:52.545869 | 2018-05-18 10:53:52.545 | from neutron_tempest_plugin.api import base
2018-05-18 10:53:52.547747 | 2018-05-18 10:53:52.546 | ImportError: No module named neutron_tempest_plugin.api
2018-05-18 10:53:52.549421 | 2018-05-18 10:53:52.549 |
2018-05-18 10:53:52.550938 | 2018-05-18 10:53:52.550 | Failed to import test module: neutron_dynamic_routing.tests.tempest.scenario.4byte_asn.test_4byte_asn
2018-05-18 10:53:52.553017 | 2018-05-18 10:53:52.552 | Traceback (most recent call last):
2018-05-18 10:53:52.555572 | 2018-05-18 10:53:52.554 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 456, in _find_test_path
2018-05-18 10:53:52.556969 | 2018-05-18 10:53:52.556 | module = self._get_module_from_name(name)
2018-05-18 10:53:52.559333 | 2018-05-18 10:53:52.558 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 395, in _get_module_from_name
2018-05-18 10:53:52.561099 | 2018-05-18 10:53:52.560 | __import__(name)
2018-05-18 10:53:52.562870 | 2018-05-18 10:53:52.562 | File "/opt/stack/new/neutron-dynamic-routing/neutron_dynamic_routing/tests/tempest/scenario/4byte_asn/test_4byte_asn.py", line 17, in <module>
2018-05-18 10:53:52.564153 | 2018-05-18 10:53:52.563 | from neutron_dynamic_routing.tests.tempest.scenario import base
2018-05-18 10:53:52.565673 | 2018-05-18 10:53:52.565 | File "/opt/stack/new/neutron-dynamic-routing/neutron_dynamic_routing/tests/tempest/scenario/base.py", line 26, in <module>
2018-05-18 10:53:52.566980 | 2018-05-18 10:53:52.566 | from neutron_tempest_plugin.api import base
2018-05-18 10:53:52.568808 | 2018-05-18 10:53:52.568 | ImportError: No module named neutron_tempest_plugin.api
2018-05-18 10:53:52.571786 | 2018-05-18 10:53:52.571 |
2018-05-18 10:53:52.574061 | 2018-05-18 10:53:52.573 | Failed to import test module: neutron_dynamic_routing.tests.tempest.scenario.basic.test_basic
2018-05-18 10:53:52.575514 | 2018-05-18 10:53:52.575 | Traceback (most recent call last):
2018-05-18 10:53:52.577013 | 2018-05-18 10:53:52.576 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 456, in _find_test_path
2018-05-18 10:53:52.579796 | 2018-05-18 10:53:52.578 | module = self._get_module_from_name(name)
2018-05-18 10:53:52.580611 | 2018-05-18 10:53:52.580 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 395, in _get_module_from_name
2018-05-18 10:53:52.582381 | 2018-05-18 10:53:52.581 | __import__(name)
2018-05-18 10:53:52.584346 | 2018-05-18 10:53:52.583 | File "/opt/stack/new/neutron-dynamic-routing/neutron_dynamic_routing/tests/tempest/scenario/basic/test_basic.py", line 20, in <module>
2018-05-18 10:53:52.585645 | 2018-05-18 10:53:52.585 | from neutron_dynamic_routing.tests.tempest.scenario import base as s_base
2018-05-18 10:53:52.588130 | 2018-05-18 10:53:52.587 | File "/opt/stack/new/neutron-dynamic-routing/neutron_dynamic_routing/tests/tempest/scenario/base.py", line 26, in <module>
2018-05-18 10:53:52.591194 | 2018-05-18 10:53:52.589 | from neutron_tempest_plugin.api import base
2018-05-18 10:53:52.593514 | 2018-05-18 10:53:52.593 | ImportError: No module named neutron_tempest_plugin.api
2018-05-18 10:53:52.596743 | 2018-05-18 10:53:52.595 |
2018-05-18 10:53:52.599415 | 2018-05-18 10:53:52.597 | Failed to import test module: neutron_dynamic_routing.tests.tempest.scenario.ipv4.test_ipv4
2018-05-18 10:53:52.601552 | 2018-05-18 10:53:52.600 | Traceback (most recent call last):
2018-05-18 10:53:52.603094 | 2018-05-18 10:53:52.602 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 456, in _find_test_path
2018-05-18 10:53:52.605971 | 2018-05-18 10:53:52.605 | module = self._get_module_from_name(name)
2018-05-18 10:53:52.607633 | 2018-05-18 10:53:52.607 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 395, in _get_module_from_name
2018-05-18 10:53:52.608955 | 2018-05-18 10:53:52.608 | __import__(name)
2018-05-18 10:53:52.610845 | 2018-05-18 10:53:52.610 | File "/opt/stack/new/neutron-dynamic-routing/neutron_dynamic_routing/tests/tempest/scenario/ipv4/test_ipv4.py", line 20, in <module>
2018-05-18 10:53:52.613081 | 2018-05-18 10:53:52.611 | from neutron_dynamic_routing.tests.tempest.scenario import base
2018-05-18 10:53:52.614561 | 2018-05-18 10:53:52.614 | File "/opt/stack/new/neutron-dynamic-routing/neutron_dynamic_routing/tests/tempest/scenario/base.py", line 26, in <module>
2018-05-18 10:53:52.615860 | 2018-05-18 10:53:52.615 | from neutron_tempest_plugin.api import base
2018-05-18 10:53:52.617487 | 2018-05-18 10:53:52.617 | ImportError: No module named neutron_tempest_plugin.api
2018-05-18 10:53:52.618920 | 2018-05-18 10:53:52.618 |
2018-05-18 10:53:52.621790 | 2018-05-18 10:53:52.620 | Failed to import test module: neutron_dynamic_routing.tests.tempest.scenario.ipv6.test_ipv6
2018-05-18 10:53:52.622436 | 2018-05-18 10:53:52.622 | Traceback (most recent call last):
2018-05-18 10:53:52.623745 | 2018-05-18 10:53:52.623 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 456, in _find_test_path
2018-05-18 10:53:52.625084 | 2018-05-18 10:53:52.624 | module = self._get_module_from_name(name)
2018-05-18 10:53:52.626710 | 2018-05-18 10:53:52.626 | File "/usr/local/lib/python2.7/dist-packages/unittest2/loader.py", line 395, in _get_module_from_name
2018-05-18 10:53:52.628076 | 2018-05-18 10:53:52.627 | __import__(name)
2018-05-18 10:53:52.630307 | 2018-05-18 10:53:52.629 | File "/opt/stack/new/neutron-dynamic-routing/neutron_dynamic_routing/tests/tempest/scenario/ipv6/test_ipv6.py", line 20, in <module>
2018-05-18 10:53:52.633455 | 2018-05-18 10:53:52.631 | from neutron_dynamic_routing.tests.tempest.scenario import base
2018-05-18 10:53:52.635028 | 2018-05-18 10:53:52.634 | File "/opt/stack/new/neutron-dynamic-routing/neutron_dynamic_routing/tests/tempest/scenario/base.py", line 26, in <module>
2018-05-18 10:53:52.636172 | 2018-05-18 10:53:52.635 | from neutron_tempest_plugin.api import base
2018-05-18 10:53:52.638726 | 2018-05-18 10:53:52.638 | ImportError: No module named neutron_tempest_plugin.api
2018-05-18 10:53:52.640998 | 2018-05-18 10:53:52.639 |
2018-05-18 10:53:52.642329 | 2018-05-18 10:53:52.641 | ================================================================================
2018-05-18 10:53:52.643699 | 2018-05-18 10:53:52.643 | The above traceback was encountered during test discovery which imports all the found test modules in the specified test_path.
2018-05-18 10:53:52.644996 | 2018-05-18 10:53:52.644 | ++ /opt/stack/new/manila/contrib/ci/post_test_hook.sh:source:361 : cd -
2018-05-18 10:53:52.646306 | 2018-05-18 10:53:52.646 | /opt/stack/new/tempest
2018-05-18 10:53:52.648812 | 2018-05-18 10:53:52.647 | ++ /opt/stack/new/manila/contrib/ci/post_test_hook.sh:source:362 : RETVAL=0

Gerrit Review: https://review.openstack.org/#/c/569360/
Logs: http://logs.openstack.org/60/569360/2/check/manila-tempest-minimal-dsvm-lvm-centos-7/3c689d2/
(Also attached to this bug report)

RCA
===
neutron's in-tree tempest plugin has moved out of tree, but the in-tree tempest plugin in neutron-dynamic-routing has not. We don't need this tempest plugin, but by the virtue of installing neutron-dynamic-routing, these tests will become discoverable. We need a workaround until neutron-dynamic-routing folks decide to excise their tempest plugin from in-tree and house it elsewhere so we don't have in when we install neutron-dynamic-routing.

Relevant links:
[1] https://review.openstack.org/#/c/519310/ (Porting tempest plugin out-of-tree)
[2] https://review.openstack.org/#/c/536931/ (Removal of tempest plugin from neutron tree)
[3] https://review.openstack.org/#/c/521346/ (neutron-dynamic-routing relying on the new out-of-tree neutron tempest plugin)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/569481

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/569481
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=f983b91181388bdcd296b7bedad60c3217371a8a
Submitter: Zuul
Branch: master

commit f983b91181388bdcd296b7bedad60c3217371a8a
Author: Goutham Pacha Ravi <email address hidden>
Date: Fri May 18 11:03:55 2018 -0700

    Fix test plugin issues in dsvm-lvm-centos job

    Require running the job with neutron-tempest-plugin
    since neutron-dynamic-routing relies on it,
    and does not specify it in its requirements.

    Change-Id: Ic8e8033b77f82c3264270ba042a44177babaff9b
    Closes-Bug: #1772077

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 7.0.0.0b2 development milestone.