neutron

Security groups: exception under load

Bug #1599086 reported by Gary Kotton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Won't Fix
High
Gary Kotton
oslo.db
Fix Released
Medium
Kevin Benton

Bug Description

For one of the iteration, adding router interface failed showing the below DB error.

2016-07-04 17:12:59.057 ERROR neutron.api.v2.resource [req-33bb4fd7-25a5-4460-82d0-ab5e5b8d574c ctx_rally_8204b9df57e44bcf9804a278c35bf2a4_user_0 8204b9df57e44bcf9804a278c35bf2a4] add_router_interface failed
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource Traceback (most recent call last):
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 84, in resource
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource result = method(request=request, **args)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource ectxt.value = e.inner_exc
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource self.force_reraise()
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource return f(*args, **kwargs)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 217, in _handle_action
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource ret_value = getattr(self._plugin, name)(*arg_list, **kwargs)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 1509, in add_router_interface
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource interface=info)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/opt/stack/vmware-nsx/vmware_nsx/dhcp_meta/rpc.py", line 121, in handle_router_metadata_access
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource plugin, ctx_elevated, router_id)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/opt/stack/vmware-nsx/vmware_nsx/dhcp_meta/rpc.py", line 171, in _create_metadata_access_network
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource {'network': net_data})
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 452, in create_network
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource self._ensure_default_security_group(context, tenant_id)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 710, in _ensure_default_security_group
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource return self._create_default_security_group(context, tenant_id)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 721, in _create_default_security_group
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource context, security_group, default_sg=True)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 1759, in create_security_group
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource firewall.delete_section(section_id)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource self.force_reraise()
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 1736, in create_security_group
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource context, security_group, default_sg))
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 189, in create_security_group
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource **kwargs)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource self.gen.next()
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 89, in autonested_transaction
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource yield tx
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 490, in __exit__
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource self.rollback()
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource compat.reraise(exc_type, exc_value, exc_tb)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 487, in __exit__
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource self.commit()
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 392, in commit
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource self._prepare_impl()
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource self.session.flush()
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2019, in flush
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource self._flush(objects)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2137, in _flush
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource transaction.rollback(_capture_exception=True)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource compat.reraise(exc_type, exc_value, exc_tb)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2101, in _flush
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource flush_context.execute()
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource rec.execute(self)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource uow
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource mapper, table, insert)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 767, in _emit_insert_statements
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource execute(statement, multiparams)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource return meth(self, multiparams, params)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource return connection._execute_clauseelement(self, multiparams, params)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource compiled_sql, distilled_params
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource context)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource util.raise_from_cause(newraise, exc_info)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource reraise(type(exception), exception, tb=exc_tb, cause=cause)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource context)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource cursor.execute(statement, parameters)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 158, in execute
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource result = self._query(query)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 308, in _query
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource conn.query(q)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 820, in query
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1002, in _read_query_result
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource result.read()
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1285, in read
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource first_packet = self.connection._read_packet()
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 966, in _read_packet
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource packet.check_error()
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 394, in check_error
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource err.raise_mysql_exception(self._data)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource _check_mysql_exception(errinfo)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource raise errorclass(errno, errorvalue)
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource DBError: (pymysql.err.IntegrityError) (1062, u"Duplicate entry '' for key 'PRIMARY'") [SQL: u'INSERT INTO default_security_group (tenant_id, security_group_id) VALUES (%(tenant_id)s, %(security_group_id)s)'] [parameters: {'tenant_id': '', 'security_group_id': '7e39244d-922c-4e3a-b93e-db6125bcb7e6'}]
2016-07-04 17:12:59.057 TRACE neutron.api.v2.resource

Tags: sg-fw
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/337567

Changed in neutron:
assignee: nobody → Gary Kotton (garyk)
status: New → In Progress
Gary Kotton (garyk)
Changed in neutron:
importance: Undecided → Critical
tags: added: mitaka-backport-potential sg-fw
Revision history for this message
Kevin Benton (kevinbenton) wrote :

Olso patch is here: https://review.openstack.org/#/c/338043/

Changed in oslo.db:
assignee: nobody → Kevin Benton (kevinbenton)
Changed in neutron:
importance: Critical → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by garyk (<email address hidden>) on branch: master
Review: https://review.openstack.org/337567
Reason: Thanks to Kevin - https://review.openstack.org/#/c/338043/1
ncftpg

Revision history for this message
Kevin Benton (kevinbenton) wrote :

Downgrading severity of Neutron bug because from what I can tell this will only affect the blank tenant so it will only happen once per deployment unless the default security group for the blank tenant is frequently deleted.

Roman Podoliaka (rpodolyaka)
Changed in oslo.db:
status: New → Confirmed
importance: Undecided → Medium
status: Confirmed → In Progress
Revision history for this message
Kevin Benton (kevinbenton) wrote :

This will basically just need a requirements bump on the neutron side now.

Kevin Benton (kevinbenton)
Changed in oslo.db:
status: In Progress → Fix Committed
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/oslo.db 4.8.0

This issue was fixed in the openstack/oslo.db 4.8.0 release.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/oslo.db 4.7.1

This issue was fixed in the openstack/oslo.db 4.7.1 release.

Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

We can't bump the oslo.db version in Mitaka. As for Newton, it's already >=4.10.0. I guess we can close the bug.

Changed in neutron:
status: In Progress → Won't Fix
tags: removed: mitaka-backport-potential
Roman Podoliaka (rpodolyaka)
Changed in oslo.db:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

This issue was fixed in the openstack/oslo.db 4.7.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.