neutron

MechanismDriverError/sqlite3.OperationalError: no such table: ml2_geneve_allocations

Bug #1600733 reported by Darren Hoyland
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Invalid
Undecided
Unassigned

Bug Description

The Openstack Interoperability Lab hit this error: "OperationalError: (sqlite3.OperationalError) no such table: ml2_geneve_allocations [SQL: u'SELECT ml2_geneve_allocations.geneve_vni AS ml2_geneve_allocations_geneve_vni, ml2_geneve_allocations.allocated AS ml2_geneve_allocations_allocated \nFROM ml2_geneve_allocations']"

This manifests itself in the console as "NeutronClientException: 500-{u'NeutronError': {u'message': u'create_subnet_postcommit failed.', u'type': u'MechanismDriverError', u'detail': u''}}"

We hit this whilst using the following components:

Block Storage cinder-iscsi
Compute nova-kvm
Database mysql
Image Storage glance-swift
Openstack Version mitaka
SDN neutron-calico
Ubuntu Version xenial

The following traceback was found in /neutron-api_*/var/log/neutron/neutron-server.log

2016-07-10 00:45:36.418 15401 ERROR neutron.service [-] Unrecoverable error: please check log for details.
2016-07-10 00:45:36.418 15401 ERROR neutron.service Traceback (most recent call last):
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 107, in serve_wsgi
2016-07-10 00:45:36.418 15401 ERROR neutron.service service.start()
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 80, in start
2016-07-10 00:45:36.418 15401 ERROR neutron.service self.wsgi_app = _run_wsgi(self.app_name)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 234, in _run_wsgi
2016-07-10 00:45:36.418 15401 ERROR neutron.service app = config.load_paste_app(app_name)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/common/config.py", line 290, in load_paste_app
2016-07-10 00:45:36.418 15401 ERROR neutron.service app = loader.load_app(app_name)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/oslo_service/wsgi.py", line 353, in load_app
2016-07-10 00:45:36.418 15401 ERROR neutron.service return deploy.loadapp("config:%s" % self.config_path, name=name)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2016-07-10 00:45:36.418 15401 ERROR neutron.service return loadobj(APP, uri, name=name, **kw)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2016-07-10 00:45:36.418 15401 ERROR neutron.service return context.create()
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-07-10 00:45:36.418 15401 ERROR neutron.service return self.object_type.invoke(self)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-07-10 00:45:36.418 15401 ERROR neutron.service **context.local_conf)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-07-10 00:45:36.418 15401 ERROR neutron.service val = callable(*args, **kw)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 28, in urlmap_factory
2016-07-10 00:45:36.418 15401 ERROR neutron.service app = loader.get_app(app_name, global_conf=global_conf)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-07-10 00:45:36.418 15401 ERROR neutron.service name=name, global_conf=global_conf).create()
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-07-10 00:45:36.418 15401 ERROR neutron.service return self.object_type.invoke(self)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-07-10 00:45:36.418 15401 ERROR neutron.service **context.local_conf)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-07-10 00:45:36.418 15401 ERROR neutron.service val = callable(*args, **kw)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/auth.py", line 71, in pipeline_factory
2016-07-10 00:45:36.418 15401 ERROR neutron.service app = loader.get_app(pipeline[-1])
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-07-10 00:45:36.418 15401 ERROR neutron.service name=name, global_conf=global_conf).create()
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-07-10 00:45:36.418 15401 ERROR neutron.service return self.object_type.invoke(self)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-07-10 00:45:36.418 15401 ERROR neutron.service return fix_call(context.object, context.global_conf, **context.local_conf)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-07-10 00:45:36.418 15401 ERROR neutron.service val = callable(*args, **kw)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/api/v2/router.py", line 71, in factory
2016-07-10 00:45:36.418 15401 ERROR neutron.service return cls(**local_config)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/api/v2/router.py", line 75, in __init__
2016-07-10 00:45:36.418 15401 ERROR neutron.service plugin = manager.NeutronManager.get_plugin()
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 233, in get_plugin
2016-07-10 00:45:36.418 15401 ERROR neutron.service return weakref.proxy(cls.get_instance().plugin)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 227, in get_instance
2016-07-10 00:45:36.418 15401 ERROR neutron.service cls._create_instance()
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
2016-07-10 00:45:36.418 15401 ERROR neutron.service return f(*args, **kwargs)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 213, in _create_instance
2016-07-10 00:45:36.418 15401 ERROR neutron.service cls._instance = cls()
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 116, in __init__
2016-07-10 00:45:36.418 15401 ERROR neutron.service plugin_provider)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 149, in _get_plugin_instance
2016-07-10 00:45:36.418 15401 ERROR neutron.service return plugin_class()
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/quota/resource_registry.py", line 121, in wrapper
2016-07-10 00:45:36.418 15401 ERROR neutron.service return f(*args, **kwargs)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 157, in __init__
2016-07-10 00:45:36.418 15401 ERROR neutron.service self.type_manager.initialize()
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 182, in initialize
2016-07-10 00:45:36.418 15401 ERROR neutron.service driver.obj.initialize()
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/type_geneve.py", line 88, in initialize
2016-07-10 00:45:36.418 15401 ERROR neutron.service self._initialize(cfg.CONF.ml2_type_geneve.vni_ranges)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/type_tunnel.py", line 121, in _initialize
2016-07-10 00:45:36.418 15401 ERROR neutron.service self.sync_allocations()
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
2016-07-10 00:45:36.418 15401 ERROR neutron.service ectxt.value = e.inner_exc
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-10 00:45:36.418 15401 ERROR neutron.service self.force_reraise()
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-10 00:45:36.418 15401 ERROR neutron.service six.reraise(self.type_, self.value, self.tb)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2016-07-10 00:45:36.418 15401 ERROR neutron.service return f(*args, **kwargs)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/type_tunnel.py", line 155, in sync_allocations
2016-07-10 00:45:36.418 15401 ERROR neutron.service with_lockmode("update").all())
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
2016-07-10 00:45:36.418 15401 ERROR neutron.service return list(self)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
2016-07-10 00:45:36.418 15401 ERROR neutron.service return self._execute_and_instances(context)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
2016-07-10 00:45:36.418 15401 ERROR neutron.service result = conn.execute(querycontext.statement, self._params)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-07-10 00:45:36.418 15401 ERROR neutron.service return meth(self, multiparams, params)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-07-10 00:45:36.418 15401 ERROR neutron.service return connection._execute_clauseelement(self, multiparams, params)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-07-10 00:45:36.418 15401 ERROR neutron.service compiled_sql, distilled_params
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-07-10 00:45:36.418 15401 ERROR neutron.service context)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-07-10 00:45:36.418 15401 ERROR neutron.service util.raise_from_cause(newraise, exc_info)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2016-07-10 00:45:36.418 15401 ERROR neutron.service reraise(type(exception), exception, tb=exc_tb)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-07-10 00:45:36.418 15401 ERROR neutron.service context)
2016-07-10 00:45:36.418 15401 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-07-10 00:45:36.418 15401 ERROR neutron.service cursor.execute(statement, parameters)
2016-07-10 00:45:36.418 15401 ERROR neutron.service OperationalError: (sqlite3.OperationalError) no such table: ml2_geneve_allocations [SQL: u'SELECT ml2_geneve_allocations.geneve_vni AS ml2_geneve_allocations_geneve_vni, ml2_geneve_allocations.allocated AS ml2_geneve_allocations_allocated \nFROM ml2_geneve_allocations']
2016-07-10 00:45:36.418 15401 ERROR neutron.service

TestRun #1432b2b9-9885-472e-96e5-8cffeba64685

Tags: oil
Darren Hoyland (autonomouse)
tags: added: oil
Revision history for this message
Darren Hoyland (autonomouse) wrote :
Revision history for this message
lixiaolong (kong62) wrote :
Download full text (31.2 KiB)

I also encountered this problem:

Ubuntu 16.04
openstack queens

2018-09-01 14:48:55.968 38585 INFO neutron.common.config [-] Logging enabled!
2018-09-01 14:48:55.969 38585 INFO neutron.common.config [-] /usr/bin/neutron-server version 12.0.2
2018-09-01 14:48:55.969 38585 INFO neutron.common.config [-] Logging enabled!
2018-09-01 14:48:55.970 38585 INFO neutron.common.config [-] /usr/bin/neutron-server version 12.0.2
2018-09-01 14:48:56.123 38585 INFO neutron.manager [-] Loading core plugin: ml2
2018-09-01 14:48:56.443 38585 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat', 'vlan', 'gre', 'vxlan', 'geneve']
2018-09-01 14:48:56.446 38585 INFO neutron.plugins.ml2.drivers.type_flat [-] Arbitrary flat physical_network names allowed
2018-09-01 14:48:56.454 38585 INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: OrderedDict()
2018-09-01 14:48:56.458 38585 INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete
2018-09-01 14:48:56.463 38585 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve', 'vlan', 'gre', 'local', 'vxlan']
2018-09-01 14:48:56.463 38585 INFO neutron.plugins.ml2.managers [-] Registered types: ['flat', 'geneve', 'vlan', 'gre', 'local', 'vxlan']
2018-09-01 14:48:56.463 38585 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['local']
2018-09-01 14:48:56.464 38585 INFO neutron.plugins.ml2.managers [-] Configured extension driver names: []
2018-09-01 14:48:56.464 38585 INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: []
2018-09-01 14:48:56.464 38585 INFO neutron.plugins.ml2.managers [-] Registered extension drivers: []
2018-09-01 14:48:56.465 38585 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: []
2018-09-01 14:48:56.465 38585 INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: []
2018-09-01 14:48:56.465 38585 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: []
2018-09-01 14:48:56.556 38585 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat'
2018-09-01 14:48:56.556 38585 INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete
2018-09-01 14:48:56.556 38585 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'geneve'
2018-09-01 14:48:56.557 38585 INFO neutron.plugins.ml2.drivers.type_tunnel [-] geneve ID ranges: []
2018-09-01 14:48:56.788 38585 ERROR neutron.service [req-95ebec71-1093-450c-add3-1fc84630cec9 - - - - -] Unrecoverable error: please check log for details.: DBNonExistentTa
ble: (sqlite3.OperationalError) no such table: ml2_geneve_allocations [SQL: u'SELECT ml2_geneve_allocations.geneve_vni AS ml2_geneve_allocations_geneve_vni, ml2_geneve_allo
cations.allocated AS ml2_geneve_allocations_allocated \nFROM ml2_geneve_allocations']
2018-09-01 14:48:56.788 38585 ERROR neutron.service Traceback (most recent call last):
2018-09-01 14:48:56.788 38585 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 86, in serve_wsgi
2018-09-01 14:48:56.788 38585 ERROR neutron.service service.start()
2018-09-01 14:48:56.788 38585 ...

Revision history for this message
LIU Yulong (dragon889) wrote :

Meets the same issue in our jenkins CI ut job.
Something similar to this bug:
"random sqlite related failure during unit testing"
https://bugs.launchpad.net/ubuntu/+source/vmware-nsx/+bug/1676712

ENV: neutron stable/queens
OS: centos7.5

Rodolfo Alonso (rodolfo-alonso-hernandez)
Changed in neutron:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.