neutron

DB race when starting multiple neutron-servers with GRE

Bug #1417560 reported by Jakub Libosvar on 2015-02-03

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	Undecided	Jakub Libosvar	neutron 2015.1.0 "kilo"
	Juno	Fix Released	Undecided	Jakub Libosvar	neutron 2014.2.3

Bug Description

If more neutron-servers are started at once, all of them are trying to make the same entries in ml2_gre_allocations. Before entries are created, neutron looks whether allocation is already created and then creates missing allocations. It can happen that allocation is created by second neutron server causing first neutron server to crash:

2015-01-09 14:28:25.768 24538 INFO neutron.plugins.ml2.drivers.type_tunnel [-] gre ID ranges: [(10, 10000)]
2015-01-09 14:28:39.072 24538 ERROR neutron.service [-] Unrecoverable error: please check log for details.
2015-01-09 14:28:39.072 24538 TRACE neutron.service Traceback (most recent call last):
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/service.py", line 102, in serve_wsgi
2015-01-09 14:28:39.072 24538 TRACE neutron.service service.start()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/service.py", line 73, in start
2015-01-09 14:28:39.072 24538 TRACE neutron.service self.wsgi_app = _run_wsgi(self.app_name)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/service.py", line 168, in _run_wsgi
2015-01-09 14:28:39.072 24538 TRACE neutron.service app = config.load_paste_app(app_name)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 183, in load_paste_app
2015-01-09 14:28:39.072 24538 TRACE neutron.service app = deploy.loadapp("config:%s" % config_path, name=app_name)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2015-01-09 14:28:39.072 24538 TRACE neutron.service return loadobj(APP, uri, name=name, **kw)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2015-01-09 14:28:39.072 24538 TRACE neutron.service return context.create()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2015-01-09 14:28:39.072 24538 TRACE neutron.service return self.object_type.invoke(self)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2015-01-09 14:28:39.072 24538 TRACE neutron.service **context.local_conf)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
2015-01-09 14:28:39.072 24538 TRACE neutron.service val = callable(*args, **kw)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
2015-01-09 14:28:39.072 24538 TRACE neutron.service app = loader.get_app(app_name, global_conf=global_conf)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2015-01-09 14:28:39.072 24538 TRACE neutron.service name=name, global_conf=global_conf).create()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2015-01-09 14:28:39.072 24538 TRACE neutron.service return self.object_type.invoke(self)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2015-01-09 14:28:39.072 24538 TRACE neutron.service **context.local_conf)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
2015-01-09 14:28:39.072 24538 TRACE neutron.service val = callable(*args, **kw)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
2015-01-09 14:28:39.072 24538 TRACE neutron.service app = loader.get_app(pipeline[-1])
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2015-01-09 14:28:39.072 24538 TRACE neutron.service name=name, global_conf=global_conf).create()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2015-01-09 14:28:39.072 24538 TRACE neutron.service return self.object_type.invoke(self)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2015-01-09 14:28:39.072 24538 TRACE neutron.service return fix_call(context.object, context.global_conf, **context.local_conf)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
2015-01-09 14:28:39.072 24538 TRACE neutron.service val = callable(*args, **kw)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 71, in factory
2015-01-09 14:28:39.072 24538 TRACE neutron.service return cls(**local_config)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 75, in __init__
2015-01-09 14:28:39.072 24538 TRACE neutron.service plugin = manager.NeutronManager.get_plugin()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 219, in get_plugin
2015-01-09 14:28:39.072 24538 TRACE neutron.service return weakref.proxy(cls.get_instance().plugin)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 213, in get_instance
2015-01-09 14:28:39.072 24538 TRACE neutron.service cls._create_instance()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py", line 249, in inner
2015-01-09 14:28:39.072 24538 TRACE neutron.service return f(*args, **kwargs)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 199, in _create_instance
2015-01-09 14:28:39.072 24538 TRACE neutron.service cls._instance = cls()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 114, in __init__
2015-01-09 14:28:39.072 24538 TRACE neutron.service plugin_provider)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 140, in _get_plugin_instance
2015-01-09 14:28:39.072 24538 TRACE neutron.service return plugin_class()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 124, in __init__
2015-01-09 14:28:39.072 24538 TRACE neutron.service self.type_manager.initialize()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 146, in initialize
2015-01-09 14:28:39.072 24538 TRACE neutron.service driver.obj.initialize()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_gre.py", line 73, in initialize
2015-01-09 14:28:39.072 24538 TRACE neutron.service self._initialize(cfg.CONF.ml2_type_gre.tunnel_id_ranges)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_tunnel.py", line 65, in _initialize
2015-01-09 14:28:39.072 24538 TRACE neutron.service self.sync_allocations()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_gre.py", line 110, in sync_allocations
2015-01-09 14:28:39.072 24538 TRACE neutron.service session.add(alloc)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__
2015-01-09 14:28:39.072 24538 TRACE neutron.service self.rollback()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2015-01-09 14:28:39.072 24538 TRACE neutron.service compat.reraise(exc_type, exc_value, exc_tb)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__
2015-01-09 14:28:39.072 24538 TRACE neutron.service self.commit()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 377, in commit
2015-01-09 14:28:39.072 24538 TRACE neutron.service self._prepare_impl()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 357, in _prepare_impl
2015-01-09 14:28:39.072 24538 TRACE neutron.service self.session.flush()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1919, in flush
2015-01-09 14:28:39.072 24538 TRACE neutron.service self._flush(objects)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2037, in _flush
2015-01-09 14:28:39.072 24538 TRACE neutron.service transaction.rollback(_capture_exception=True)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2015-01-09 14:28:39.072 24538 TRACE neutron.service compat.reraise(exc_type, exc_value, exc_tb)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2001, in _flush
2015-01-09 14:28:39.072 24538 TRACE neutron.service flush_context.execute()
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
2015-01-09 14:28:39.072 24538 TRACE neutron.service rec.execute(self)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 526, in execute
2015-01-09 14:28:39.072 24538 TRACE neutron.service uow
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 65, in save_obj
2015-01-09 14:28:39.072 24538 TRACE neutron.service mapper, table, insert)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 570, in _emit_insert_statements
2015-01-09 14:28:39.072 24538 TRACE neutron.service execute(statement, multiparams)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
2015-01-09 14:28:39.072 24538 TRACE neutron.service return meth(self, multiparams, params)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
2015-01-09 14:28:39.072 24538 TRACE neutron.service return connection._execute_clauseelement(self, multiparams, params)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
2015-01-09 14:28:39.072 24538 TRACE neutron.service compiled_sql, distilled_params
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
2015-01-09 14:28:39.072 24538 TRACE neutron.service context)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
2015-01-09 14:28:39.072 24538 TRACE neutron.service util.raise_from_cause(newraise, exc_info)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2015-01-09 14:28:39.072 24538 TRACE neutron.service reraise(type(exception), exception, tb=exc_tb)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 928, in _execute_context
2015-01-09 14:28:39.072 24538 TRACE neutron.service context)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/connectors/mysqldb.py", line 68, in do_executemany
2015-01-09 14:28:39.072 24538 TRACE neutron.service rowcount = cursor.executemany(statement, parameters)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 206, in executemany
2015-01-09 14:28:39.072 24538 TRACE neutron.service r = r + self.execute(query, a)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2015-01-09 14:28:39.072 24538 TRACE neutron.service self.errorhandler(self, exc, value)
2015-01-09 14:28:39.072 24538 TRACE neutron.service File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-01-09 14:28:39.072 24538 TRACE neutron.service raise errorclass, errorvalue
2015-01-09 14:28:39.072 24538 TRACE neutron.service DBDuplicateEntry: (IntegrityError) (1062, "Duplicate entry '10' for key 'PRIMARY'") 'INSERT INTO ml2_gre_allocations (gre_id, allocated) VALUES (%s, %s)' ((10, 0), (11, 0), (12, 0), (13, 0), (14, 0), (15, 0), (16, 0), (17, 0) ... displaying 10 of 9991 total bound parameter sets ... (9999, 0), (10000, 0))

Tags:

Jakub Libosvar (libosvar) on 2015-02-03

Changed in neutron:
assignee:	nobody → Jakub Libosvar (libosvar)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-03: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/152598

Changed in neutron:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-10: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/152598
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=50b469fae2dbd1517d856509f37328394e0697e4
Submitter: Jenkins
Branch: master

commit 50b469fae2dbd1517d856509f37328394e0697e4
Author: Jakub Libosvar <email address hidden>
Date: Tue Feb 3 14:33:39 2015 +0100

Don't crash when adding duplicate gre allocation

    This patch catches DBDuplicateError when initializing ML2 GRE type
    driver and allocation already exists in DB. Because current allocations
    are queried and then only those that doesn't exist in database are
    added, DBDuplicateError should never occur.

But the race can happen when running multiple neutron-servers and one
of servers adds allocations between allocations are queried and added.

Change-Id: I427b7020d61b0d2c06292ff2804ba1f4483696c6
Closes-bug: 1417560

Changed in neutron:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-10: Fix proposed to neutron (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/154381

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-12: Fix merged to neutron (stable/juno)

Reviewed: https://review.openstack.org/154381
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9bb7eb5e1baed71e95f772474df9c6376563225b
Submitter: Jenkins
Branch: stable/juno

commit 9bb7eb5e1baed71e95f772474df9c6376563225b
Author: Jakub Libosvar <email address hidden>
Date: Tue Feb 3 14:33:39 2015 +0100

Don't crash when adding duplicate gre allocation

But the race can happen when running multiple neutron-servers and one
of servers adds allocations between allocations are queried and added.

Closes-bug: 1417560

Conflicts:
neutron/tests/unit/ml2/test_type_gre.py

Change-Id: I427b7020d61b0d2c06292ff2804ba1f4483696c6
(cherry picked from commit 50b469fae2dbd1517d856509f37328394e0697e4)

tags:

added: in-stable-juno

Thierry Carrez (ttx) on 2015-03-19

Changed in neutron:
milestone:	none → kilo-3
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2015-04-30

Changed in neutron:
milestone:	kilo-3 → 2015.1.0

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.