networking-cisco

DBDuplicateEntry cisco_ml2_with nexus_host_interface_mapping when multiple controllers

Bug #1743573 reported by Carol Bouchard on 2018-01-16

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	networking-cisco	Fix Released	Undecided	Carol Bouchard

Bug Description

Earlier fixed issue 1735540 where DBDuplicateEntry was seen on a the same controller. Now there are 3 controllers of which only one is failing with DBDuplicateEntry.

2018-01-12 22:53:19.179 96650 ERROR neutron.service [-] Unrecoverable error: please check log for details.
2018-01-12 22:53:19.179 96650 ERROR neutron.service Traceback (most recent call last):
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/service.py", line 87, in serve_wsgi
2018-01-12 22:53:19.179 96650 ERROR neutron.service service.start()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/service.py", line 63, in start
2018-01-12 22:53:19.179 96650 ERROR neutron.service self.wsgi_app = _run_wsgi(self.app_name)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/service.py", line 289, in _run_wsgi
2018-01-12 22:53:19.179 96650 ERROR neutron.service app = config.load_paste_app(app_name)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 125, in load_paste_app
2018-01-12 22:53:19.179 96650 ERROR neutron.service app = loader.load_app(app_name)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/oslo_service/wsgi.py", line 353, in load_app
2018-01-12 22:53:19.179 96650 ERROR neutron.service return deploy.loadapp("config:%s" % self.config_path, name=name)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2018-01-12 22:53:19.179 96650 ERROR neutron.service return loadobj(APP, uri, name=name, **kw)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2018-01-12 22:53:19.179 96650 ERROR neutron.service return context.create()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2018-01-12 22:53:19.179 96650 ERROR neutron.service return self.object_type.invoke(self)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2018-01-12 22:53:19.179 96650 ERROR neutron.service **context.local_conf)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2018-01-12 22:53:19.179 96650 ERROR neutron.service val = callable(*args, **kw)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
2018-01-12 22:53:19.179 96650 ERROR neutron.service app = loader.get_app(app_name, global_conf=global_conf)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2018-01-12 22:53:19.179 96650 ERROR neutron.service name=name, global_conf=global_conf).create()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2018-01-12 22:53:19.179 96650 ERROR neutron.service return self.object_type.invoke(self)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2018-01-12 22:53:19.179 96650 ERROR neutron.service **context.local_conf)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2018-01-12 22:53:19.179 96650 ERROR neutron.service val = callable(*args, **kw)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
2018-01-12 22:53:19.179 96650 ERROR neutron.service app = loader.get_app(pipeline[-1])
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2018-01-12 22:53:19.179 96650 ERROR neutron.service name=name, global_conf=global_conf).create()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2018-01-12 22:53:19.179 96650 ERROR neutron.service return self.object_type.invoke(self)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2018-01-12 22:53:19.179 96650 ERROR neutron.service return fix_call(context.object, context.global_conf, **context.local_conf)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2018-01-12 22:53:19.179 96650 ERROR neutron.service val = callable(*args, **kw)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 72, in factory
2018-01-12 22:53:19.179 96650 ERROR neutron.service return cls(**local_config)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 76, in __init__
2018-01-12 22:53:19.179 96650 ERROR neutron.service plugin = manager.NeutronManager.get_plugin()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 244, in get_plugin
2018-01-12 22:53:19.179 96650 ERROR neutron.service return weakref.proxy(cls.get_instance().plugin)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 238, in get_instance
2018-01-12 22:53:19.179 96650 ERROR neutron.service cls._create_instance()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2018-01-12 22:53:19.179 96650 ERROR neutron.service return f(*args, **kwargs)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 224, in _create_instance
2018-01-12 22:53:19.179 96650 ERROR neutron.service cls._instance = cls()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 126, in __init__
2018-01-12 22:53:19.179 96650 ERROR neutron.service plugin_provider)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 160, in _get_plugin_instance
2018-01-12 22:53:19.179 96650 ERROR neutron.service return plugin_class()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/quota/resource_registry.py", line 127, in wrapper
2018-01-12 22:53:19.179 96650 ERROR neutron.service return f(*args, **kwargs)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 162, in __init__
2018-01-12 22:53:19.179 96650 ERROR neutron.service self.mechanism_manager.initialize()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 399, in initialize
2018-01-12 22:53:19.179 96650 ERROR neutron.service driver.obj.initialize()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py", line 322, in initialize
2018-01-12 22:53:19.179 96650 ERROR neutron.service conf.ML2MechCiscoConfig()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/config.py", line 84, in __init__
2018-01-12 22:53:19.179 96650 ERROR neutron.service self._create_ml2_mech_device_cisco_dictionary()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/config.py", line 116, in _create_ml2_mech_device_cisco_dictionary
2018-01-12 22:53:19.179 96650 ERROR neutron.service dev_key, dev_ip, interface, 0, True)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/nexus_db_v2.py", line 489, in add_host_mapping
2018-01-12 22:53:19.179 96650 ERROR neutron.service session.flush()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2027, in flush
2018-01-12 22:53:19.179 96650 ERROR neutron.service self._flush(objects)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2145, in _flush
2018-01-12 22:53:19.179 96650 ERROR neutron.service transaction.rollback(_capture_exception=True)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2018-01-12 22:53:19.179 96650 ERROR neutron.service compat.reraise(exc_type, exc_value, exc_tb)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2109, in _flush
2018-01-12 22:53:19.179 96650 ERROR neutron.service flush_context.execute()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2018-01-12 22:53:19.179 96650 ERROR neutron.service rec.execute(self)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2018-01-12 22:53:19.179 96650 ERROR neutron.service uow
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
2018-01-12 22:53:19.179 96650 ERROR neutron.service mapper, table, insert)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 767, in _emit_insert_statements
2018-01-12 22:53:19.179 96650 ERROR neutron.service execute(statement, multiparams)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
2018-01-12 22:53:19.179 96650 ERROR neutron.service return meth(self, multiparams, params)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2018-01-12 22:53:19.179 96650 ERROR neutron.service return connection._execute_clauseelement(self, multiparams, params)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2018-01-12 22:53:19.179 96650 ERROR neutron.service compiled_sql, distilled_params
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2018-01-12 22:53:19.179 96650 ERROR neutron.service context)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2018-01-12 22:53:19.179 96650 ERROR neutron.service util.raise_from_cause(newraise, exc_info)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2018-01-12 22:53:19.179 96650 ERROR neutron.service reraise(type(exception), exception, tb=exc_tb)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2018-01-12 22:53:19.179 96650 ERROR neutron.service context)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2018-01-12 22:53:19.179 96650 ERROR neutron.service cursor.execute(statement, parameters)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2018-01-12 22:53:19.179 96650 ERROR neutron.service result = self._query(query)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2018-01-12 22:53:19.179 96650 ERROR neutron.service conn.query(q)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 841, in query
2018-01-12 22:53:19.179 96650 ERROR neutron.service self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1029, in _read_query_result
2018-01-12 22:53:19.179 96650 ERROR neutron.service result.read()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1312, in read
2018-01-12 22:53:19.179 96650 ERROR neutron.service first_packet = self.connection._read_packet()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
2018-01-12 22:53:19.179 96650 ERROR neutron.service packet.check_error()
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2018-01-12 22:53:19.179 96650 ERROR neutron.service err.raise_mysql_exception(self._data)
2018-01-12 22:53:19.179 96650 ERROR neutron.service File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2018-01-12 22:53:19.179 96650 ERROR neutron.service raise errorclass(errno, errval)
2018-01-12 22:53:19.179 96650 ERROR neutron.service DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry 'overcloud-controller-2.localdomain-10.86.1.34-port-channel:301' for key 'PRIMARY'") [SQL: u'INSERT INTO cisco_ml2_nexus_host_interface_mapping (host_id, switch_ip, if_id, ch_grp, is_static) VALUES (%(host_id)s, %(switch_ip)s, %(if_id)s, %(ch_grp)s, %(is_static)s)'] [parameters: {'is_static': 1, 'switch_ip': '10.86.1.34', 'if_id': 'port-channel:301', 'ch_grp': 0, 'host_id': 'overcloud-controller-2.localdomain'}]
2018-01-12 22:53:19.179 96650 ERROR neutron.service
2018-01-12 22:53:20.001 96650 CRITICAL neutron [-] DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry 'overcloud-controller-2.localdomain-10.86.1.34-port-channel:301' for key 'PRIMARY'") [SQL: u'INSERT INTO cisco_ml2_nexus_host_interface_mapping (host_id, switch_ip, if_id, ch_grp, is_static) VALUES (%(host_id)s, %(switch_ip)s, %(if_id)s, %(ch_grp)s, %(is_static)s)'] [parameters: {'is_static': 1, 'switch_ip': '10.86.1.34', 'if_id': 'port-channel:301', 'ch_grp': 0, 'host_id': 'overcloud-controller-2.localdomain'}]
2018-01-12 22:53:20.001 96650 ERROR neutron Traceback (most recent call last):
2018-01-12 22:53:20.001 96650 ERROR neutron File "/usr/bin/neutron-server", line 10, in <module>

Tags:

Carol Bouchard (caboucha) on 2018-01-16

Changed in networking-cisco:
assignee:	nobody → Carol Bouchard (caboucha)
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-16: Fix proposed to networking-cisco (master)

Fix proposed to branch: master
Review: https://review.openstack.org/534411

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-17: Fix merged to networking-cisco (master)

Reviewed: https://review.openstack.org/534411
Committed: https://git.openstack.org/cgit/openstack/networking-cisco/commit/?id=0ccd0f0af6d60640f6fa6e9c1262fe5f3947a673
Submitter: Zuul
Branch: master

commit 0ccd0f0af6d60640f6fa6e9c1262fe5f3947a673
Author: Carol Bouchard <email address hidden>
Date: Tue Jan 16 14:33:33 2018 -0500

DBDuplicateEntry in map table, multi-controllers

    When there are multiple controllers trying to add static host mappings
    to cisco_ml2_nexus_host_interface_mapping db table, the exception
    DBDuplicateEntry can occur due to timing. For static configs,
    this changeset makes this exception valid.

Change-Id: I6148eaba6f4abe65afc6537f9c20fb5aca7075bd
Closes-bug: #1743573

Changed in networking-cisco:
status:	In Progress → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.