DBDuplicateEntry: (IntegrityError) in neutron server for openvswitch-agent

Bug #1391766 reported by Wei Hu
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Low
Wei Hu

Bug Description

After deployed openstack by chef, there is an error in neutron server.log about openvswitch-agent.

2014-11-10 23:14:36.050 31989 ERROR oslo.messaging.rpc.dispatcher [req-e07a7ed7-d22f-40e9-b299-aba6e06f9470 ] Exception during message handling: (IntegrityError) ibm_db_dbi::IntegrityError: Statement Execute Failed: [IBM][CLI Driver][DB2/LINUXX8664] SQL0803N One or more values in the INSERT statement, UPDATE statement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint or unique index identified by "2" constrains table "NEUTRON.AGENTS" from having duplicate values for the index key. SQLSTATE=23505 SQLCODE=-803 'INSERT INTO agents (id, agent_type, "binary", topic, host, admin_state_up, created_at, started_at, heartbeat_timestamp, description, configurations) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' ('5028b520-1a90-46c2-9e97-b1cb80d6ebe0', 'Open vSwitch agent', 'neutron-openvswitch-agent', 'N/A', 'testhl12.sce.ibm.com', '1', datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), None, '{"arp_responder_enabled": false, "tunneling_ip": "10.11.1.12", "devices": 0, "l2_population": false, "tunnel_types": ["gre", "vxlan"], "enable_distributed_routing": false, "bridge_mappings": {"default": "br-eth1"}}')
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 237, in report_state
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher self.plugin.create_or_update_agent(context, agent_state)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 214, in create_or_update_agent
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher return self._create_or_update_agent(context, agent)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/openstack/common/excutils.py", line 82, in __exit__
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 197, in create_or_update_agent
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher return self._create_or_update_agent(context, agent)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 191, in _create_or_update_agent
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher greenthread.sleep(0)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 447, in __exit__
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher self.rollback()
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 444, in __exit__
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher self.commit()
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 354, in commit
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher self._prepare_impl()
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 334, in _prepare_impl
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher self.session.flush()
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1818, in flush
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher self._flush(objects)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1936, in _flush
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher transaction.rollback(_capture_exception=True)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1900, in _flush
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher flush_context.execute()
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher rec.execute(self)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 525, in execute
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher uow
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher table, insert)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 569, in _emit_insert_statements
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher execute(statement, params)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 662, in execute
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher params)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher compiled_sql, distilled_params
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher context)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 125, in _handle_dbapi_exception
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher six.reraise(type(newraise), newraise, sys.exc_info()[2])
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 102, in _handle_dbapi_exception
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher per_fn = fn(ctx)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 323, in handler
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher context.is_disconnect)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 223, in _db2_dupe_key_error
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher raise exception.DBDuplicateEntry([], integrity_error)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher DBDuplicateEntry: (IntegrityError) ibm_db_dbi::IntegrityError: Statement Execute Failed: [IBM][CLI Driver][DB2/LINUXX8664] SQL0803N One or more values in the INSERT statement, UPDATE statement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint or unique index identified by "2" constrains table "NEUTRON.AGENTS" from having duplicate values for the index key. SQLSTATE=23505 SQLCODE=-803 'INSERT INTO agents (id, agent_type, "binary", topic, host, admin_state_up, created_at, started_at, heartbeat_timestamp, description, configurations) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' ('5028b520-1a90-46c2-9e97-b1cb80d6ebe0', 'Open vSwitch agent', 'neutron-openvswitch-agent', 'N/A', 'testhl12.sce.ibm.com', '1', datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), None, '{"arp_responder_enabled": false, "tunneling_ip": "10.11.1.12", "devices": 0, "l2_population": false, "tunnel_types": ["gre", "vxlan"], "enable_distributed_routing": false, "bridge_mappings": {"default": "br-eth1"}}')
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
2014-11-10 23:14:36.064 31989 ERROR oslo.messaging._drivers.common [req-e07a7ed7-d22f-40e9-b299-aba6e06f9470 ] Returning exception (IntegrityError) ibm_db_dbi::IntegrityError: Statement Execute Failed: [IBM][CLI Driver][DB2/LINUXX8664] SQL0803N One or more values in the INSERT statement, UPDATE statement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint or unique index identified by "2" constrains table "NEUTRON.AGENTS" from having duplicate values for the index key. SQLSTATE=23505 SQLCODE=-803 'INSERT INTO agents (id, agent_type, "binary", topic, host, admin_state_up, created_at, started_at, heartbeat_timestamp, description, configurations) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' ('5028b520-1a90-46c2-9e97-b1cb80d6ebe0', 'Open vSwitch agent', 'neutron-openvswitch-agent', 'N/A', 'testhl12.sce.ibm.com', '1', datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), None, '{"arp_responder_enabled": false, "tunneling_ip": "10.11.1.12", "devices": 0, "l2_population": false, "tunnel_types": ["gre", "vxlan"], "enable_distributed_routing": false, "bridge_mappings": {"default": "br-eth1"}}') to caller
2014-11-10 23:14:36.064 31989 ERROR oslo.messaging._drivers.common [req-e07a7ed7-d22f-40e9-b299-aba6e06f9470 ] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 237, in report_state\n self.plugin.create_or_update_agent(context, agent_state)\n', ' File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 214, in create_or_update_agent\n return self._create_or_update_agent(context, agent)\n', ' File "/usr/lib/python2.6/site-packages/neutron/openstack/common/excutils.py", line 82, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 197, in create_or_update_agent\n return self._create_or_update_agent(context, agent)\n', ' File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 191, in _create_or_update_agent\n greenthread.sleep(0)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 447, in __exit__\n self.rollback()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 444, in __exit__\n self.commit()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 354, in commit\n self._prepare_impl()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 334, in _prepare_impl\n self.session.flush()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1818, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1936, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1900, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 525, in execute\n uow\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj\n table, insert)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 569, in _emit_insert_statements\n execute(statement, params)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 662, in execute\n params)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement\n compiled_sql, distilled_params\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context\n context)\n', ' File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 125, in _handle_dbapi_exception\n six.reraise(type(newraise), newraise, sys.exc_info()[2])\n', ' File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 102, in _handle_dbapi_exception\n per_fn = fn(ctx)\n', ' File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 323, in handler\n context.is_disconnect)\n', ' File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 223, in _db2_dupe_key_error\n raise exception.DBDuplicateEntry([], integrity_error)\n', 'DBDuplicateEntry: (IntegrityError) ibm_db_dbi::IntegrityError: Statement Execute Failed: [IBM][CLI Driver][DB2/LINUXX8664] SQL0803N One or more values in the INSERT statement, UPDATE statement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint or unique index identified by "2" constrains table "NEUTRON.AGENTS" from having duplicate values for the index key. SQLSTATE=23505 SQLCODE=-803 \'INSERT INTO agents (id, agent_type, "binary", topic, host, admin_state_up, created_at, started_at, heartbeat_timestamp, description, configurations) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\' (\'5028b520-1a90-46c2-9e97-b1cb80d6ebe0\', \'Open vSwitch agent\', \'neutron-openvswitch-agent\', \'N/A\', \'testhl12.sce.ibm.com\', \'1\', datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), None, \'{"arp_responder_enabled": false, "tunneling_ip": "10.11.1.12", "devices": 0, "l2_population": false, "tunnel_types": ["gre", "vxlan"], "enable_distributed_routing": false, "bridge_mappings": {"default": "br-eth1"}}\')\n']

Tags: db
Revision history for this message
Wei Hu (huwei-xtu) wrote :

Reference to https://github.com/openstack/neutron/blob/master/neutron/db/agents_db.py#L196, seems the retry logic code still not avoid this concurrency issue.

Revision history for this message
Wei Hu (huwei-xtu) wrote :

exception.DBDuplicateEntry for db2 does not provide columns. See the comments in https://github.com/openstack/oslo.db/blob/master/oslo/db/sqlalchemy/exc_filters.py#L221.

Changed in neutron:
assignee: nobody → Wei Hu (huwei-xtu)
Changed in neutron:
importance: Undecided → Low
status: New → Confirmed
tags: added: db
removed: neutron
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/134176

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/134176
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a7eead2f1fa71466c5734413165752dc9386ca89
Submitter: Jenkins
Branch: master

commit a7eead2f1fa71466c5734413165752dc9386ca89
Author: Wei Hu <email address hidden>
Date: Thu Nov 13 18:19:39 2014 +0800

    Remove DBDuplicateEntry columns check

    DBDuplicateEntry exception does not provide columns for db2.
    we have to remove this exception columns check in neutron/
    db/agents_db.py. It is safe to remove columns check here, since
    only the duplication of host and agent_type can raise this
    exception.

    Change-Id: I00d7b7c64de2912dda9fa57c08d90b1ef1c3aed7
    Closes-bug: #1391766

Changed in neutron:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in neutron:
milestone: none → kilo-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: kilo-2 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.