q-svc fails to start in kilo due to "ImportError: No module named neutron_vpnaas.services.vpn.service_drivers.ipsec"

Bug #1483266 reported by Matt Riedemann
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Critical
Doug Wiegley
Kilo
Fix Released
Undecided
Unassigned

Bug Description

http://logs.openstack.org/70/210870/1/check/gate-grenade-dsvm-neutron/20f794e/logs/new/screen-q-svc.txt.gz?level=TRACE

Looks like this is blocking kilo jobs that use neutron:

2015-08-10 00:37:30.529 8402 ERROR neutron.common.config [-] Unable to load neutron from configuration file /etc/neutron/api-paste.ini.
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config Traceback (most recent call last):
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/common/config.py", line 227, in load_paste_app
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config app = deploy.loadapp("config:%s" % config_path, name=app_name)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config return loadobj(APP, uri, name=name, **kw)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config return context.create()
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config return self.object_type.invoke(self)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config **context.local_conf)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config val = callable(*args, **kw)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 28, in urlmap_factory
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config app = loader.get_app(app_name, global_conf=global_conf)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config name=name, global_conf=global_conf).create()
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config return self.object_type.invoke(self)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config **context.local_conf)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config val = callable(*args, **kw)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/auth.py", line 71, in pipeline_factory
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config app = loader.get_app(pipeline[-1])
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config name=name, global_conf=global_conf).create()
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config return self.object_type.invoke(self)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config return fix_call(context.object, context.global_conf, **context.local_conf)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config val = callable(*args, **kw)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/api/v2/router.py", line 71, in factory
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config return cls(**local_config)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/api/v2/router.py", line 75, in __init__
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config plugin = manager.NeutronManager.get_plugin()
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/manager.py", line 222, in get_plugin
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config return weakref.proxy(cls.get_instance().plugin)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/manager.py", line 216, in get_instance
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config cls._create_instance()
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config return f(*args, **kwargs)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/manager.py", line 202, in _create_instance
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config cls._instance = cls()
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/manager.py", line 128, in __init__
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config self._load_service_plugins()
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/manager.py", line 175, in _load_service_plugins
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config provider)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/manager.py", line 143, in _get_plugin_instance
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config return plugin_class()
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/plugin.py", line 78, in __init__
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config self.service_type_manager = st_db.ServiceTypeManager.get_instance()
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/db/servicetype_db.py", line 41, in get_instance
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config cls._instance = cls()
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/db/servicetype_db.py", line 45, in __init__
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config self._load_conf()
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/db/servicetype_db.py", line 49, in _load_conf
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config pconf.parse_service_provider_opt())
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/services/provider_configuration.py", line 121, in parse_service_provider_opt
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config driver = get_provider_driver_class(driver)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/opt/stack/new/neutron/neutron/services/provider_configuration.py", line 53, in get_provider_driver_class
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config namespace, driver).driver
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/local/lib/python2.7/dist-packages/stevedore/driver.py", line 45, in __init__
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config verify_requirements=verify_requirements,
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/local/lib/python2.7/dist-packages/stevedore/named.py", line 55, in __init__
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config verify_requirements)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 170, in _load_plugins
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config self._on_load_failure_callback(self, ep, err)
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config File "/usr/local/lib/python2.7/dist-packages/stevedore/driver.py", line 50, in _default_on_load_failure
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config raise err
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config ImportError: No module named neutron_vpnaas.services.vpn.service_drivers.ipsec
2015-08-10 00:37:30.529 8402 TRACE neutron.common.config

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSW1wb3J0RXJyb3I6IE5vIG1vZHVsZSBuYW1lZCBuZXV0cm9uX3ZwbmFhcy5zZXJ2aWNlcy52cG4uc2VydmljZV9kcml2ZXJzLmlwc2VjXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MzkxNzAwNTUzODJ9

Kyle Mestery (mestery)
Changed in neutron:
assignee: nobody → Doug Wiegley (dougwig)
importance: Undecided → High
Revision history for this message
Kyle Mestery (mestery) wrote :

Doug thinks this one will fix it by removing the advanced services tests from grenade for now:

https://review.openstack.org/#/c/201279/

Kyle Mestery (mestery)
Changed in neutron:
status: New → Confirmed
Revision history for this message
Doug Wiegley (dougwig) wrote :

As discussed last week, this commit is meant to address this: https://review.openstack.org/#/c/201279/

Alternately, we need to revert the changes to remove vpn from devstack proper, as there is stuff that needs to be fixed before that happens.

Revision history for this message
Doug Wiegley (dougwig) wrote :

This will be broken for a bit.

Reverts of the changes that broke stable/kilo have been refused, and instead there is an insistence that the root problem be fixed, though it's scope is unknown and could've been assessed while the branch was not broken. That process is ongoing here https://review.openstack.org/#/c/211760/

I strongly object to this approach, but will continue to pursue the fix.

Revision history for this message
Paul Michali (pcm) wrote :

Will weigh in now, after spending considerable time looking at the problem.

There are two problems currently, IMHO. First, the VPN devstack plugin and devstack repo were not correctly set up to allow BOTH q-vpn service and the plugin to be enabled. This is what led to removing q-vpn service in the first place (an addition to the thought that other jobs shouldn't need q-vpn enabled). I didn't realize this broke kilo.

Second, I just found out that there is a bug in the devstack plugin/devstack setup for VPN. If the plugin is enabled, we will incorrectly get two q-vpn processes created. There should have been logic to prevent this, in the devstack script (part of that being to use a different service name in the plugin).

Current breakage, AFAICT is with A) stable/kilo not having the plugin support and requiring q-vpn to be set, and B) neutron-client not using the devstack plugin, and C) the bug in the plugin/devstack script starting two processes.

My proposal is to do the following:
1) modify devstack repo to only do q-vpn service setup if plugin is not in use.
2) modify plugin to use a different service name (proposed neutron-vpnaas)

Once these are applied, and verified that existing working jobs are still OK, we can do...
3) Backport both to stable/kilo
4) Have neutron-client use devstack plugin (upstream 209887 and 210021).

This fixes stable/kilo and neutron-client. Then, cleanup can occur to remove VPN from jobs that don't need VPN, and finally VPN setup can be removed from devstack scripts.

For details on proposals I have thought through, see https://etherpad.openstack.org/p/vpn-test-changes L46+

I have patches ready (and locally tested for stacking with different service combinations) to be pushed up for review, if this is desired.

Assaf Muller (amuller)
Changed in neutron:
importance: High → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/juno)

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/211706
Reason: As per Doug's comment.

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/211760
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=25420bb1bbccb037866872efdf1e5d3458d286a6
Submitter: Jenkins
Branch: master

commit 25420bb1bbccb037866872efdf1e5d3458d286a6
Author: Doug Wiegley <email address hidden>
Date: Tue Aug 11 15:46:31 2015 -0600

    Setup reference service providers for API test runs

    Partial-bug: #1483266
    Change-Id: Ia979fa2fdf25f3c1dbe9e7160d6dde2df51c305e

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/213256

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-vpnaas (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/213257

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/213256
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6298a90c45c5737a838ad4c4e332fe0deb8312d6
Submitter: Jenkins
Branch: stable/kilo

commit 6298a90c45c5737a838ad4c4e332fe0deb8312d6
Author: Doug Wiegley <email address hidden>
Date: Tue Aug 11 15:46:31 2015 -0600

    Setup reference service providers for API test runs

    Partial-Bug: #1483266
    Change-Id: Ia979fa2fdf25f3c1dbe9e7160d6dde2df51c305e
    (cherry picked from commit 25420bb1bbccb037866872efdf1e5d3458d286a6)

tags: added: in-stable-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-vpnaas (master)

Reviewed: https://review.openstack.org/211758
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=f9d5c0505752b5720beab589f9d76573740a695e
Submitter: Jenkins
Branch: master

commit f9d5c0505752b5720beab589f9d76573740a695e
Author: Doug Wiegley <email address hidden>
Date: Tue Aug 11 15:45:10 2015 -0600

    Remove default service provider from conf file

    Partial-bug: #1483266
    Depends-On: Ia979fa2fdf25f3c1dbe9e7160d6dde2df51c305e
    Change-Id: If04871e4461ad5838a6fd7a3993d4eda1566d145

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/213750
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=805496dd5ce85c9280342f1702b12dd30b201bbb
Submitter: Jenkins
Branch: master

commit 805496dd5ce85c9280342f1702b12dd30b201bbb
Author: Doug Wiegley <email address hidden>
Date: Mon Aug 17 09:17:46 2015 -0600

    Don't fatal error during initialization for missing service providers

    Sometime during the split, code was added to fixup driver paths,
    which imports service providers even for plugins which are not
    in use. That, combined with neutron including default service
    providers for VPN and LOADBALANCER, resulted in a really messy
    mess in terms of removing VPN from the main neutron test suites.

    This change stops the imports, so that if one of the services is
    missing, neutron server can still start. It likely breaks the driver
    path fixup, which can be fixed outside of this gate blockage.

    Closes-Bug: #1483266
    Change-Id: I23f9007357d8cbbae599997c244561a4e2f32ce1

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/213751
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2a0001647ed1ecb35e9dc9f61d4b22c5170ae45f
Submitter: Jenkins
Branch: stable/kilo

commit 2a0001647ed1ecb35e9dc9f61d4b22c5170ae45f
Author: Doug Wiegley <email address hidden>
Date: Mon Aug 17 09:17:46 2015 -0600

    Don't fatal error during initialization for missing service providers

    Sometime during the split, code was added to fixup driver paths,
    which imports service providers even for plugins which are not
    in use. That, combined with neutron including default service
    providers for VPN and LOADBALANCER, resulted in a really messy
    mess in terms of removing VPN from the main neutron test suites.

    This change stops the imports, so that if one of the services is
    missing, neutron server can still start. It likely breaks the driver
    path fixup, which can be fixed outside of this gate blockage.

    Closes-Bug: #1483266
    Change-Id: I23f9007357d8cbbae599997c244561a4e2f32ce1

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron-vpnaas (stable/kilo)

Change abandoned by Doug Wiegley (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/213257

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/pecan)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/218710

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (feature/pecan)
Download full text (155.6 KiB)

Reviewed: https://review.openstack.org/218710
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2c5f44e1b3bd4ed8a0b7232fd293b576cc8c1c87
Submitter: Jenkins
Branch: feature/pecan

commit f35d1c5c50dccbef1a2e079f967b82f0df0e22e9
Author: Adelina Tuvenie <email address hidden>
Date: Thu Aug 27 02:27:28 2015 -0700

    Fixes wrong neutron Hyper-V Agent name in constants

    Change Id03fb147e11541be309c1cd22ce27e70fadc28b5 moved the
    AGENT_TYPE_HYPERV constant from common.constants to
    plugins.ml2.drivers.hyperv.constants but change the value of the
    constant from 'HyperV agent' to 'hyperv'. This patch changes
    the name back to 'HyperV agent'

    Change-Id: If74b4b2a84811e266c8b12e70bf6bfe74ed4ea21
    Partial-Bug: #1487598

commit de604de334854e2eb6b4312ff57920564cbd4459
Author: OpenStack Proposal Bot <email address hidden>
Date: Sun Aug 30 01:39:06 2015 +0000

    Updated from global requirements

    Change-Id: Ie52aa3b59784722806726e4046bd07f4a4d97328

commit f0415ac20eaf5ab4abb9bd4839bf6d04ceee85d0
Author: armando-migliaccio <email address hidden>
Date: Fri Aug 28 13:53:04 2015 -0700

    Revert "Add support for unaddressed port"

    This implementation may expose a vulnerability where a malicious
    user can sieze the opportunity of a time window where a port
    may land unaddressed on a shared network, thus allowing him/her
    to suck up all the tenant traffic he/she wants....oh the shivers.

    This reverts commit d4c52b7f5a36a103a92bf9dcda7f371959112292.

    Change-Id: I7ebdaa8d3defa80eab90e460fde541a5bdd8864c

commit 013fdcd2a6d45dbe4de5d6e7077e5e9b60985ef9
Author: Assaf Muller <email address hidden>
Date: Fri Aug 28 16:41:07 2015 -0400

    Improve logging upon failure in iptables functional tests

    This will help us nail down a more accurate and efficient logstash
    query.

    Change-Id: Iee4238e358f7b056e373c7be8d6aa3202117a680
    Related-Bug: #1478847

commit 622dea818d851224a43d5276a81d5ce8a6eebb76
Author: Ivar Lazzaro <email address hidden>
Date: Mon Aug 17 17:17:42 2015 -0700

    handle gw_info outside of the db transaction on router creation

    Move the gateway interface creation outside the DB transaction
    to avoid lock timeout.

    Change-Id: I5a78d7f32e8ca912016978105221d5f34618af19
    Closes-bug: 1485809

commit 5b27d290a0a95f6247fc5a0fe6da1e7d905e6b2d
Author: Assaf Muller <email address hidden>
Date: Wed Aug 26 10:07:03 2015 -0400

    Remove ml2 resource extension success logging

    This is the cause of a tremendous amount of logs, for no
    perceivable gain. A normal dvr run in the gate shows this debug
    message around 120K times, which is way too much.

    Closes-Bug: #1489952

    Change-Id: I26fca8515d866a7cc1638d07fa33bc04479ae221

commit 8d3faf549cba2f58c872ef4121b2481e73464010
Author: huangpengtao <email address hidden>
Date: Fri Aug 28 23:20:46 2015 +0800

    Replace "prt" variable by "port"

    the local variable prt is meaningless,
    and port is used popular.

    Change-Id: I20849102cf5b4d84433c46791b4b1e2a22dc4739

commit ee374e7a5f4dea538fcd942f5...

tags: added: in-feature-pecan
Thierry Carrez (ttx)
Changed in neutron:
milestone: none → liberty-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: liberty-3 → 7.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.