neutron server failed with 'NoForeignKeysError'

Bug #1766267 reported by Hongbin Lu
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Unassigned

Bug Description

The failure was found from Heat gate:

  http://logs.openstack.org/03/559803/2/gate/heat-functional-convg-mysql-lbaasv2/825236e/logs/screen-q-svc.txt.gz#_Apr_23_10_02_01_252456

The error message and traceback is as following:

Apr 23 10:02:01.252456 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service [None req-494752bd-c98f-45d2-9a16-8e6c9dd02b8e None None] Unrecoverable error: please check log for details.: NoForeignKeysError: Could not determine join condition between parent/child tables on relationship ProviderResourceAssociation._provider_backref - there are no foreign keys linking these tables. Ensure that referencing columns are associated with a ForeignKey or ForeignKeyConstraint, or specify a 'primaryjoin' expression.
Apr 23 10:02:01.253089 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service Traceback (most recent call last):
Apr 23 10:02:01.253204 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/service.py", line 86, in serve_wsgi
Apr 23 10:02:01.253315 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service service.start()
Apr 23 10:02:01.253425 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/service.py", line 62, in start
Apr 23 10:02:01.253541 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service self.wsgi_app = _run_wsgi(self.app_name)
Apr 23 10:02:01.253661 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/service.py", line 290, in _run_wsgi
Apr 23 10:02:01.253787 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service app = config.load_paste_app(app_name)
Apr 23 10:02:01.253918 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/common/config.py", line 122, in load_paste_app
Apr 23 10:02:01.254024 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service app = loader.load_app(app_name)
Apr 23 10:02:01.254128 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/oslo_service/wsgi.py", line 353, in load_app
Apr 23 10:02:01.254231 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service return deploy.loadapp("config:%s" % self.config_path, name=name)
Apr 23 10:02:01.254351 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
Apr 23 10:02:01.254456 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service return loadobj(APP, uri, name=name, **kw)
Apr 23 10:02:01.254573 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
Apr 23 10:02:01.254700 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service return context.create()
Apr 23 10:02:01.254831 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
Apr 23 10:02:01.254954 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service return self.object_type.invoke(self)
Apr 23 10:02:01.255108 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
Apr 23 10:02:01.255392 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service **context.local_conf)
Apr 23 10:02:01.255616 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
Apr 23 10:02:01.255851 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service val = callable(*args, **kw)
Apr 23 10:02:01.256092 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 31, in urlmap_factory
Apr 23 10:02:01.256312 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service app = loader.get_app(app_name, global_conf=global_conf)
Apr 23 10:02:01.256582 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
Apr 23 10:02:01.256821 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service name=name, global_conf=global_conf).create()
Apr 23 10:02:01.257047 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
Apr 23 10:02:01.257267 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service return self.object_type.invoke(self)
Apr 23 10:02:01.257489 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
Apr 23 10:02:01.257753 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service **context.local_conf)
Apr 23 10:02:01.257976 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
Apr 23 10:02:01.258194 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service val = callable(*args, **kw)
Apr 23 10:02:01.258411 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/auth.py", line 47, in pipeline_factory
Apr 23 10:02:01.258693 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service app = loader.get_app(pipeline[-1])
Apr 23 10:02:01.258925 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
Apr 23 10:02:01.259187 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service name=name, global_conf=global_conf).create()
Apr 23 10:02:01.259431 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
Apr 23 10:02:01.259659 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service return self.object_type.invoke(self)
Apr 23 10:02:01.259989 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
Apr 23 10:02:01.260102 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service return fix_call(context.object, context.global_conf, **context.local_conf)
Apr 23 10:02:01.260209 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
Apr 23 10:02:01.260336 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service val = callable(*args, **kw)
Apr 23 10:02:01.260453 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/api/v2/router.py", line 25, in _factory
Apr 23 10:02:01.260588 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service return pecan_app.v2_factory(global_config, **local_config)
Apr 23 10:02:01.260696 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/pecan_wsgi/app.py", line 47, in v2_factory
Apr 23 10:02:01.260886 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service startup.initialize_all()
Apr 23 10:02:01.261074 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/pecan_wsgi/startup.py", line 39, in initialize_all
Apr 23 10:02:01.261262 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service manager.init()
Apr 23 10:02:01.261389 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/manager.py", line 296, in init
Apr 23 10:02:01.261498 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service NeutronManager.get_instance()
Apr 23 10:02:01.261638 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/manager.py", line 247, in get_instance
Apr 23 10:02:01.261747 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service cls._create_instance()
Apr 23 10:02:01.261934 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
Apr 23 10:02:01.262212 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service return f(*args, **kwargs)
Apr 23 10:02:01.262328 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/manager.py", line 233, in _create_instance
Apr 23 10:02:01.262437 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service cls._instance = cls()
Apr 23 10:02:01.262536 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/manager.py", line 141, in __init__
Apr 23 10:02:01.262692 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service self._load_service_plugins()
Apr 23 10:02:01.262857 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/manager.py", line 203, in _load_service_plugins
Apr 23 10:02:01.262969 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service provider)
Apr 23 10:02:01.263140 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/manager.py", line 166, in _get_plugin_instance
Apr 23 10:02:01.263253 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service return plugin_class()
Apr 23 10:02:01.263351 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/plugin.py", line 90, in __init__
Apr 23 10:02:01.263465 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service self._load_drivers()
Apr 23 10:02:01.263574 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/plugin.py", line 118, in _load_drivers
Apr 23 10:02:01.263729 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service self._check_orphan_loadbalancer_associations(ctx, self.drivers.keys())
Apr 23 10:02:01.263870 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/plugin.py", line 127, in _check_orphan_loadbalancer_associations
Apr 23 10:02:01.264053 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service loadbalancers = self.db.get_loadbalancers(context)
Apr 23 10:02:01.264206 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron-lbaas/neutron_lbaas/db/loadbalancer/loadbalancer_dbv2.py", line 396, in get_loadbalancers
Apr 23 10:02:01.264401 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service filters=filters)
Apr 23 10:02:01.264582 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron-lbaas/neutron_lbaas/db/loadbalancer/loadbalancer_dbv2.py", line 98, in _get_resources
Apr 23 10:02:01.264809 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service filters=filters)
Apr 23 10:02:01.264967 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/db/common_db_mixin.py", line 82, in _get_collection_query
Apr 23 10:02:01.265143 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service page_reverse)
Apr 23 10:02:01.265302 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/db/_model_query.py", line 256, in get_collection_query
Apr 23 10:02:01.265470 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service collection = query_with_hooks(context, model)
Apr 23 10:02:01.265630 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/db/_model_query.py", line 137, in query_with_hooks
Apr 23 10:02:01.265797 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service query = context.session.query(model)
Apr 23 10:02:01.265909 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1399, in query
Apr 23 10:02:01.266021 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service return self._query_cls(entities, self, **kwargs)
Apr 23 10:02:01.266157 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 141, in __init__
Apr 23 10:02:01.266295 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service self._set_entities(entities)
Apr 23 10:02:01.266423 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 152, in _set_entities
Apr 23 10:02:01.266553 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service self._set_entity_selectables(self._entities)
Apr 23 10:02:01.266663 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 182, in _set_entity_selectables
Apr 23 10:02:01.266910 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service ent.setup_entity(*d[entity])
Apr 23 10:02:01.267097 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3630, in setup_entity
Apr 23 10:02:01.267231 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service self._with_polymorphic = ext_info.with_polymorphic_mappers
Apr 23 10:02:01.267363 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 767, in __get__
Apr 23 10:02:01.267535 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service obj.__dict__[self.__name__] = result = self.fget(obj)
Apr 23 10:02:01.267719 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/mapper.py", line 2011, in _with_polymorphic_mappers
Apr 23 10:02:01.267896 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service configure_mappers()
Apr 23 10:02:01.268010 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/mapper.py", line 3029, in configure_mappers
Apr 23 10:02:01.268113 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service mapper._post_configure_properties()
Apr 23 10:02:01.268225 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/mapper.py", line 1831, in _post_configure_properties
Apr 23 10:02:01.268334 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service prop.post_instrument_class(self)
Apr 23 10:02:01.268437 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/interfaces.py", line 544, in post_instrument_class
Apr 23 10:02:01.268557 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service self.strategy.init_class_attribute(mapper)
Apr 23 10:02:01.268670 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 507, in init_class_attribute
Apr 23 10:02:01.268782 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service active_history=active_history
Apr 23 10:02:01.268903 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 106, in _register_attribute
Apr 23 10:02:01.269005 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service **kw
Apr 23 10:02:01.269124 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 1524, in register_attribute_impl
Apr 23 10:02:01.269241 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service manager.post_configure_attribute(key)
Apr 23 10:02:01.269354 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/instrumentation.py", line 278, in post_configure_attribute
Apr 23 10:02:01.269448 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service attribute_instrument(self.class_, key, self[key])
Apr 23 10:02:01.269540 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/event/attr.py", line 284, in __call__
Apr 23 10:02:01.269653 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service fn(*args, **kw)
Apr 23 10:02:01.269765 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/events.py", line 71, in listen
Apr 23 10:02:01.269872 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service return fn(target_cls, *arg)
Apr 23 10:02:01.269978 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/opt/stack/new/neutron/neutron/db/api.py", line 403, in _listen_for_changes
Apr 23 10:02:01.270085 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service remote_mapper.add_property(name, backref_prop)
Apr 23 10:02:01.270217 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/mapper.py", line 1854, in add_property
Apr 23 10:02:01.270321 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service self._configure_property(key, prop, init=self.configured)
Apr 23 10:02:01.270425 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/mapper.py", line 1731, in _configure_property
Apr 23 10:02:01.270528 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service prop.init()
Apr 23 10:02:01.270648 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/interfaces.py", line 184, in init
Apr 23 10:02:01.270788 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service self.do_init()
Apr 23 10:02:01.270897 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/relationships.py", line 1656, in do_init
Apr 23 10:02:01.271003 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service self._setup_join_conditions()
Apr 23 10:02:01.271168 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/relationships.py", line 1731, in _setup_join_conditions
Apr 23 10:02:01.271292 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service can_be_synced_fn=self._columns_are_mapped
Apr 23 10:02:01.271396 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/relationships.py", line 1998, in __init__
Apr 23 10:02:01.271510 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service self._determine_joins()
Apr 23 10:02:01.271630 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/relationships.py", line 2103, in _determine_joins
Apr 23 10:02:01.271736 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service "specify a 'primaryjoin' expression." % self.prop)
Apr 23 10:02:01.271853 ubuntu-xenial-ovh-gra1-0003676293 neutron-server[10199]: ERROR neutron.service NoForeignKeysError: Could not determine join condition between parent/child tables on relationship ProviderResourceAssociation._provider_backref - there are no foreign keys linking these tables. Ensure that referencing columns are associated with a ForeignKey or ForeignKeyConstraint, or specify a 'primaryjoin' expression.

Changed in neutron:
assignee: nobody → Hongbin Lu (hongbin.lu)
status: New → In Progress
Revision history for this message
Brian Haley (brian-haley) wrote :

Not sure if this is also triggering this failure:

sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <NetworkSegment at 0x7f0f9606ae90> is not bound to a Session; lazy load operation of attribute 'network' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)

From:

http://logs.openstack.org/39/561039/3/gate/openstack-tox-cover/933029b/testr_results.html.gz

Just seemed suspicious since this change was in the gate right after the one that just merged.

Revision history for this message
Rabi Mishra (rabi) wrote :

Neutron fails to start in heat jobs presumably when neutron-lbaas is enabled.

Revision history for this message
Pawel Suder (pasuder) wrote :

As per comment in https://review.openstack.org/#/c/559803/ that bug might be related to change https://review.openstack.org/#/c/556667/

Revision history for this message
Pawel Suder (pasuder) wrote :
Revision history for this message
Boden R (boden) wrote :
Download full text (16.5 KiB)

Seems related to an error we hit on many vmware-nsx UTs as of very recent. For example [1].
This impacts a huge amount of tests for us [2].

The exception message looks like::

---------------
2018-04-23 15:57:49.528065 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2018-04-23 15:57:49.528408 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/git.openstack.org/openstack/vmware-nsx/.tox/py27/local/lib/python2.7/site-packages/pecan/core.py", line 682, in __call__
2018-04-23 15:57:49.528624 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation controller, args, kwargs = self.find_controller(state)
2018-04-23 15:57:49.530693 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/git.openstack.org/openstack/vmware-nsx/.tox/py27/local/lib/python2.7/site-packages/pecan/core.py", line 858, in find_controller
2018-04-23 15:57:49.530913 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation controller, args, kw = super(Pecan, self).find_controller(_state)
2018-04-23 15:57:49.531228 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/git.openstack.org/openstack/vmware-nsx/.tox/py27/local/lib/python2.7/site-packages/pecan/core.py", line 550, in find_controller
2018-04-23 15:57:49.531443 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation self.handle_hooks(self.determine_hooks(controller), 'before', state)
2018-04-23 15:57:49.531751 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/git.openstack.org/openstack/vmware-nsx/.tox/py27/local/lib/python2.7/site-packages/pecan/core.py", line 865, in handle_hooks
2018-04-23 15:57:49.531946 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation return super(Pecan, self).handle_hooks(hooks, *args, **kw)
2018-04-23 15:57:49.532275 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/git.openstack.org/openstack/vmware-nsx/.tox/py27/local/lib/python2.7/site-packages/pecan/core.py", line 342, in handle_hooks
2018-04-23 15:57:49.532456 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation result = getattr(hook, hook_type)(*args)
2018-04-23 15:57:49.532803 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/git.openstack.org/openstack/vmware-nsx/.tox/py27/local/lib/python2.7/site-packages/neutron/pecan_wsgi/hooks/quota_enforcement.py", line 52, in before
2018-04-23 15:57:49.532933 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation plugin)
2018-04-23 15:57:49.533296 | ubuntu-xenial | 2018-04-23 15:57:49.133 7780 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/git.openstack.org/openstack/vmware-nsx/.tox/py27/local/lib/python2.7/site...

Hongbin Lu (hongbin.lu)
Changed in neutron:
assignee: Hongbin Lu (hongbin.lu) → nobody
Revision history for this message
Pawel Suder (pasuder) wrote :

Marking as High based on other bugs importance.

Changed in neutron:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/563601
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a75014792aeda0851510531919d040090e1aec15
Submitter: Zuul
Branch: master

commit a75014792aeda0851510531919d040090e1aec15
Author: Rabi Mishra <email address hidden>
Date: Mon Apr 23 12:14:05 2018 +0000

    Revert "Automatically expire obsolete relationships"

    This reverts commit 90ede813b06eb46472824a7090354c31cf43791f.

    Closes-Bug: #1766267
    Change-Id: Ia491a2a404bcbd9e108d20b2d9393aff1bb48c8e

Changed in neutron:
status: In Progress → Fix Released
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 13.0.0.0b2

This issue was fixed in the openstack/neutron 13.0.0.0b2 development milestone.

tags: removed: neutron-proactive-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.