vmware-nsx

[ stable/newton] Port creation failed with error "DBDuplicateEntry: (pymysql.err.IntegrityError)" on 100 concurrency

Bug #1642203 reported by Sujai on 2016-11-16

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	vmware-nsx	Triaged	Medium	Unassigned

Bug Description

[stable/newton] Port creation failed with error "DBDuplicateEntry: (pymysql.err.IntegrityError)" on 100 concurrency

With stable/newton,executed below scenario with 100 concurrency and 500 iterations:

-Create network, subnet
-Create port (device_owner->compute:None)

For about 40 iterations, the scenario is failing with below error

"DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry '70.1.1.4-7005b2ac-9431-41f3-a59a-f6d19589b03c' for key 'PRIMARY'") [SQL: u'INSERT INTO ipamallocations (ip_address, status, ipam_subnet_id) VALUES (%(ip_address)s, %(status)s, %(ipam_subnet_id)s)'] [parameters: {'status': 'ALLOCATED', 'ip_address': '70.1.1.4', 'ipam_subnet_id': u'7005b2ac-9431-41f3-a59a-f6d19589b03c'}]"

I have attached the neutron.conf for your reference.

Neutron logs:

2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource [req-9cd01b4e-a770-4309-bd8c-987062cf4dde c_rally_94c739d9_QlP59G01 -] create failed: No details.
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource Traceback (most recent call last):
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 79, in resource
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource result = method(request=request, **args)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 430, in create
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 88, in wrapped
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource self.force_reraise()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 84, in wrapped
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource self.force_reraise()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 124, in wrapped
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource traceback.format_exc())
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource self.force_reraise()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 119, in wrapped
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 543, in _create
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource obj = do_create(body)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 525, in do_create
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource request.context, reservation.reservation_id)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource self.force_reraise()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 518, in do_create
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource return obj_creator(request.context, **kwargs)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 1769, in create_port
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource neutron_db = super(NsxV3Plugin, self).create_port(context, port)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 159, in wrapped
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource return method(*args, **kwargs)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1138, in create_port
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource db_port = self.create_port_db(context, port)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1173, in create_port_db
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource context, port, port_id)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 191, in allocate_ips_for_port_and_store
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource revert_on_fail=False)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource self.force_reraise()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 172, in allocate_ips_for_port_and_store
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource ips = self._allocate_ips_for_port(context, port_copy)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 235, in _allocate_ips_for_port
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource return self._ipam_allocate_ips(context, ipam_driver, p, ips)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 147, in _ipam_allocate_ips
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource "external system for %s"), addresses)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource self.force_reraise()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 121, in _ipam_allocate_ips
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource ip_address, subnet_id = ipam_allocator.allocate(ip_request)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/ipam/subnet_alloc.py", line 232, in allocate
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource return ipam_subnet.allocate(address_request), subnet_id
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 210, in allocate
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource self.subnet_manager.create_allocation(session, ip_address)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 490, in __exit__
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource self.rollback()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource compat.reraise(exc_type, exc_value, exc_tb)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 487, in __exit__
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource self.commit()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 392, in commit
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource self._prepare_impl()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource self.session.flush()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2019, in flush
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource self._flush(objects)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2137, in _flush
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource transaction.rollback(_capture_exception=True)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource compat.reraise(exc_type, exc_value, exc_tb)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2101, in _flush
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource flush_context.execute()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource rec.execute(self)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource uow
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource mapper, table, insert)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 767, in _emit_insert_statements
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource execute(statement, multiparams)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource return meth(self, multiparams, params)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource return connection._execute_clauseelement(self, multiparams, params)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource compiled_sql, distilled_params
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource context)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource util.raise_from_cause(newraise, exc_info)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource reraise(type(exception), exception, tb=exc_tb, cause=cause)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource context)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource cursor.execute(statement, parameters)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 167, in execute
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource result = self._query(query)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 323, in _query
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource conn.query(q)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 836, in query
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1020, in _read_query_result
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource result.read()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1303, in read
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource first_packet = self.connection._read_packet()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 982, in _read_packet
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource packet.check_error()
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 394, in check_error
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource err.raise_mysql_exception(self._data)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource _check_mysql_exception(errinfo)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource raise errorclass(errno, errorvalue)
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry '70.1.1.4-7005b2ac-9431-41f3-a59a-f6d19589b03c' for key 'PRIMARY'") [SQL: u'INSERT INTO ipamallocations (ip_address, status, ipam_subnet_id) VALUES (%(ip_address)s, %(status)s, %(ipam_subnet_id)s)'] [parameters: {'status': 'ALLOCATED', 'ip_address': '70.1.1.4', 'ipam_subnet_id': u'7005b2ac-9431-41f3-a59a-f6d19589b03c'}]
2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource

See original description

Revision history for this message

Sujai (sujai) wrote on 2016-11-16:

neutron.conf Edit (63.2 KiB, text/plain)

affects:	bash (Ubuntu) → neutron (Ubuntu)
no longer affects:	neutron (Ubuntu)
description:	updated

Revision history for this message

Victor Morales (electrocucaracha) wrote on 2016-11-16:

Hey Sujai,

In order to reproduce this I'll need the benchmark task configuration file(json or yaml format). I should be also valuable to know some details about the hardware that you used for this execution.

Changed in neutron:
status:	New → Incomplete

Revision history for this message

Sujai (sujai) wrote on 2016-11-17:

ports.json.txt Edit (1.3 KiB, text/plain)

Hi Victor,

Please find the benchmark task configuration file in the attachment.

And I have used Dell servers running ESXi 6.0. Openstack controller machine is having 12 CPUs(1816 MHz)and
89GB memory.

Kevin Benton (kevinbenton) on 2016-11-22

Changed in neutron:
status:	Incomplete → Triaged
importance:	Undecided → Medium
affects:	neutron → vmware-nsx

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.