enable package test suites: dependency on generated egg from git.openstack.org

[James Page]

The split out drivers for neutron (lbaas, vpnaas, fwaas) all rely on a source snapshot of neutron, which we don't currently have an equivalent for in Ubuntu.

We need to resolve this situation and re-enable the upstream test suites as part of the package build.

[James Page (james-page)]

Traceback (most recent call last):
  File "/«PKGBUILDDIR»/neutron_fwaas/tests/unit/services/firewall/agents/varmour/test_varmour_router.py", line 153, in test_agent_add_internal_network
    router = self._create_router()
  File "/«PKGBUILDDIR»/neutron_fwaas/tests/unit/services/firewall/agents/varmour/test_varmour_router.py", line 89, in _create_router
    router = varmour_router.vArmourL3NATAgent(HOSTNAME, self.conf)
  File "/«PKGBUILDDIR»/neutron_fwaas/services/firewall/agents/varmour/varmour_router.py", line 54, in __init__
    super(vArmourL3NATAgent, self).__init__(host, conf)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 168, in __init__
    self.context = n_context.get_admin_context_without_session()
  File "/usr/lib/python2.7/dist-packages/neutron/context.py", line 176, in get_admin_context_without_session
    read_deleted=read_deleted)
  File "/usr/lib/python2.7/dist-packages/neutron/context.py", line 68, in __init__
    self.is_advsvc = policy.check_is_advsvc(self)
  File "/usr/lib/python2.7/dist-packages/neutron/policy.py", line 448, in check_is_advsvc
    return _ENFORCER.enforce(advsvc_policy, target, credentials)
  File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/policy.py", line 314, in enforce
    self.load_rules()
  File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/policy.py", line 241, in load_rules
    self.policy_path = self._get_policy_path(self.policy_file)
  File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/policy.py", line 287, in _get_policy_path
    raise cfg.ConfigFilesNotFoundError((path,))
ConfigFilesNotFoundError: Failed to find some config files: /usr/lib/python2.7/dist-packages/etc/policy.json

[James Page (james-page)]

======================================================================
FAIL: neutron_fwaas.tests.unit.db.firewall.test_db_firewall.TestFirewallDBPlugin.test_update_firewall_policy_with_non_existing_rule
neutron_fwaas.tests.unit.db.firewall.test_db_firewall.TestFirewallDBPlugin.test_update_firewall_policy_with_non_existing_rule
----------------------------------------------------------------------
_StringException: pythonlogging:'': {{{
2015-02-16 13:02:18,727 INFO [neutron.manager] Loading core plugin: neutron.db.db_base_plugin_v2.NeutronDbPluginV2
2015-02-16 13:02:18,727 INFO [neutron.manager] Loading Plugin: neutron_fwaas.db.firewall.firewall_db.Firewall_db_mixin
2015-02-16 13:02:18,734 ERROR [neutron.api.extensions] Extension path 'neutron/tests/unit/extensions' doesn't exist!
}}}

pythonlogging:'neutron.api.extensions': {{{2015-02-16 13:02:18,734 ERROR [neutron.api.extensions] Extension path 'neutron/tests/unit/extensions' doesn't exist!}}}

Traceback (most recent call last):
  File "/«PKGBUILDDIR»/neutron_fwaas/tests/unit/db/firewall/test_db_firewall.py", line 96, in setUp
    {constants.FIREWALL: self.plugin}
  File "/usr/lib/python2.7/dist-packages/neutron/api/extensions.py", line 576, in __init__
    self.check_if_plugin_extensions_loaded()
  File "/usr/lib/python2.7/dist-packages/neutron/api/extensions.py", line 624, in check_if_plugin_extensions_loaded
    extensions=list(missing_aliases))
ExtensionsNotFound: Extensions not found: ['fwaas']

[Ihar Hrachyshka (ihar-hrachyshka)]

Don't you ship egg files with python-neutron package? Isn't it enough to get access to entry points?

[Alan Pevec (apevec)]

AFAICT egg metadata IS included in Ubuntu packages:

http://packages.ubuntu.com/trusty/all/python-neutron/filelist

[Armando Migliaccio (armando-migliaccio)]

Perhaps there are some preliminary steps that need to be executed ahead of the testsuite in the package build process? As far as I can tell I see nothing that could be missing from a source codebase standpoint.

[James Page (james-page)]

As Alan points out, egg metadata is included in the Ubuntu packages; however right now that does not appear to be sufficient. The python-neutron package is just the python code - its not going to contain configuration files or suchlike.

[James Page (james-page)]

The neutron-*aas package all build-depend on python-neutron, but still have these problems.

[James Page (james-page)]

I think the difference between the upstream egg snapshot (which contains all source) and the Ubuntu packaged source (which does not container everything) is the problem here - looking at sys.path in a fresh venv:

>>> print sys.path
['', '/home/jamespage/src/upstream/neutron-fwaas/.tox/py27/src/neutron', '/home/jamespage/src/upstream/neutron-fwaas', '/home/jamespage/src/upstream/neutron-fwaas/.tox/py27/lib/python2.7', '/home/jamespage/src/upstream/neutron-fwaas/.tox/py27/lib/python2.7/plat-x86_64-linux-gnu', '/home/jamespage/src/upstream/neutron-fwaas/.tox/py27/lib/python2.7/lib-tk', '/home/jamespage/src/upstream/neutron-fwaas/.tox/py27/lib/python2.7/lib-old', '/home/jamespage/src/upstream/neutron-fwaas/.tox/py27/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/home/jamespage/src/upstream/neutron-fwaas/.tox/py27/local/lib/python2.7/site-packages', '/home/jamespage/src/upstream/neutron-fwaas/.tox/py27/lib/python2.7/site-packages']

You can see that "/home/jamespage/src/upstream/neutron-fwaas/.tox/py27/src/neutron" is listed on the path - which includes all the neutron goodness required to execute the tests.

We don't have that in the Ubuntu distribution and would account for at least:

ConfigFilesNotFoundError: Failed to find some config files: /usr/lib/python2.7/dist-packages/etc/policy.json

as this is present in the generated egg, but not python-neutron.

[Armando Migliaccio (armando-migliaccio)]

It sounds to me then that (and please bear with me if I am completely off here) file [1] would need to include more just than simply the python code? Or have *-aas services depend on neutron-common?

Having said that, I think we'd need to have a more comprehensive discussion also on how the decomp efforts in Kilo impact Ubuntu packaging. I am wondering if you'd like to track it here, somewhere else, or what your thoughts about this are in general.

[1] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/vivid/neutron/vivid-proposed/view/head:/debian/python-neutron.install
[2] http://specs.openstack.org/openstack/neutron-specs/specs/kilo/core-vendor-decomposition.html

[James Page (james-page)]

Armando

Lets track the discussion here; right now the *aas bits are functional once deployed; the challenge is currently unit test execution during package build, which we use to gate updates to packages much the same as happens upstream in openstack.

I'm toying with creating a neutron-source binary package from the neutron build which we could use to provide a complete source tree for out-of-tree drivers to use for this purpose - its a little ugly but may do the trick - I'll try to get to testing that approach next week.

[Ihar Hrachyshka (ihar-hrachyshka)]

Can't you make python-neutron package a build dependency for your *aas packages?

[James Page (james-page)]

All packaging unit test suites are now enabled - marking 'Fix Released'