neutron

Failed to create a duplicate DefaultSecurityGroup (with Calico plugin)

Bug #1883730 reported by Nell Jerram on 2020-06-16

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	Medium	Slawek Kaplonski

Bug Description

With Ussuri I'm hitting this in the neutron server:

2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers [req-6160aa90-b69e-451b-9774-07b6d840f41f 088d04a2122548c5acb628348db93c40 11447be9beda4bf78dab27cdb75058e2 - default default] Mechanism driver 'calico' failed in update_port_postcommit: neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create a duplicate DefaultSecurityGroup: for attribute(s) ['PRIMARY'] with value(s) 11447be9beda4bf78dab27cdb75058e2
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers cursor, statement, parameters, context
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 581, in do_execute
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers cursor.execute(statement, parameters)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 165, in execute
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers result = self._query(query)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 321, in _query
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers conn.query(q)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 860, in query
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1061, in _read_query_result
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers result.read()
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1349, in read
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers first_packet = self.connection._read_packet()
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1018, in _read_packet
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers packet.check_error()
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 384, in check_error
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers err.raise_mysql_exception(self._data)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers raise errorclass(errno, errval)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers pymysql.err.IntegrityError: (1062, "Duplicate entry '11447be9beda4bf78dab27cdb75058e2' for key 'PRIMARY'")
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers The above exception was the direct cause of the following exception:
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/base.py", line 867, in create
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self, self.obj_context, self.modify_fields_to_db(fields))
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/db/api.py", line 69, in create_object
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers context.session.add(db_obj)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers next(self.gen)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1064, in _transaction_scope
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers yield resource
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers next(self.gen)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 680, in _session
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self.session.flush()
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/session.py", line 2479, in flush
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self._flush(objects)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/session.py", line 2617, in _flush
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers transaction.rollback(_capture_exception=True)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers compat.reraise(exc_type, exc_value, exc_tb)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers raise value
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/session.py", line 2577, in _flush
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers flush_context.execute()
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers rec.execute(self)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers uow,
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers insert,
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/persistence.py", line 1084, in _emit_insert_statements
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers c = cached_connections[connection].execute(statement, multiparams)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return meth(self, multiparams, params)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return connection._execute_clauseelement(self, multiparams, params)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers distilled_params,
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers e, statement, parameters, cursor, context
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1474, in _handle_dbapi_exception
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers util.raise_from_cause(newraise, exc_info)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers reraise(type(exception), exception, tb=exc_tb, cause=cause)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers raise value.with_traceback(tb)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers cursor, statement, parameters, context
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 581, in do_execute
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers cursor.execute(statement, parameters)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 165, in execute
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers result = self._query(query)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 321, in _query
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers conn.query(q)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 860, in query
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1061, in _read_query_result
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers result.read()
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1349, in read
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers first_packet = self.connection._read_packet()
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1018, in _read_packet
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers packet.check_error()
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 384, in check_error
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers err.raise_mysql_exception(self._data)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers raise errorclass(errno, errval)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '11447be9beda4bf78dab27cdb75058e2' for key 'PRIMARY'")
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers [SQL: INSERT INTO default_security_group (project_id, security_group_id) VALUES (%(project_id)s, %(security_group_id)s)]
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers [parameters: {'project_id': '11447be9beda4bf78dab27cdb75058e2', 'security_group_id': '9f3a473c-b08a-4cf2-8327-10ecc8b87301'}]
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers (Background on this error at: http://sqlalche.me/e/gkpj)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers During handling of the above exception, another exception occurred:
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/managers.py", line 477, in _call_on_drivers
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/dist-packages/networking_calico/plugins/ml2/drivers/calico/mech_calico.py", line 173, in wrapper
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return f(self, *args, **kwargs)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/dist-packages/networking_calico/plugins/ml2/drivers/calico/mech_calico.py", line 789, in update_port_postcommit
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self._icehouse_migration_step(context, port, original)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/dist-packages/networking_calico/plugins/ml2/drivers/calico/mech_calico.py", line 913, in _icehouse_migration_step
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self._port_bound_update(context, port)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/dist-packages/networking_calico/plugins/ml2/drivers/calico/mech_calico.py", line 896, in _port_bound_update
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self.endpoint_syncer.write_endpoint(port, context._plugin_context)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/dist-packages/networking_calico/plugins/ml2/drivers/calico/endpoints.py", line 153, in write_endpoint
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self.policy_syncer.write_sgs_to_etcd(port['security_groups'], context)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/dist-packages/networking_calico/plugins/ml2/drivers/calico/policy.py", line 99, in write_sgs_to_etcd
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers context, filters={'security_group_id': sgids}
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 233, in wrapped
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return method(*args, **kwargs)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/db/securitygroups_db.py", line 749, in get_security_group_rules
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self._ensure_default_security_group(context, project_id)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/db/securitygroups_db.py", line 871, in _ensure_default_security_group
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers default_sg=True)['id']
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 233, in wrapped
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return method(*args, **kwargs)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/db/securitygroups_db.py", line 114, in create_security_group
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers sg.create()
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/base.py", line 337, in decorator
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return func(self, *args, **kwargs)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/rbac_db.py", line 365, in func
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return new_method(self, orig_method)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/rbac_db.py", line 301, in _create_hook
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers orig_create(self)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/securitygroup.py", line 79, in create
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers default_group.create()
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/base.py", line 337, in decorator
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return func(self, *args, **kwargs)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/base.py", line 870, in create
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers object_class=self.__class__, db_exception=db_exc)
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create a duplicate DefaultSecurityGroup: for attribute(s) ['PRIMARY'] with value(s) 11447be9beda4bf78dab27cdb75058e2
2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers
2020-06-15 14:41:44.227 21931 ERROR neutron.plugins.ml2.plugin [req-6160aa90-b69e-451b-9774-07b6d840f41f 088d04a2122548c5acb628348db93c40 11447be9beda4bf78dab27cdb75058e2 - default default] mechanism_manager.update_port_postcommit failed for port 7dcd8de4-c74a-4b4a-b68b-5701cabfefc5: neutron.plugins.ml2.common.exceptions.MechanismDriverError

IIUC, this is triggered by my Neutron driver calling

        rules = self.db.get_security_group_rules(
            context, filters={'security_group_id': sgids}
        )

where the context has project_id 11447be9beda4bf78dab27cdb75058e2. Deep down inside that call, Neutron tries to ensure that there is a default security group for that project, and somehow that hits the reported exception.

Here's the code in securitygroups_db.py:

def _ensure_default_security_group(self, context, tenant_id):
"""Create a default security group if one doesn't exist.

        :returns: the default security group id for given tenant.
        """
        default_group_id = self._get_default_sg_id(context, tenant_id)
        if default_group_id:
            return default_group_id

        security_group = {
            'security_group':
                {'name': 'default',
                 'tenant_id': tenant_id,
                 'description': _('Default security group')}
        }
        return self.create_security_group(context, security_group,
                                          default_sg=True)['id']

Obviously it checks first if the default SG already exists for the project, before creating it if not. So why would that code hit the duplicate exception as shown above?

If it helps, the Calico driver code can be found at https://github.com/projectcalico/networking-calico/. The specific get_security_group_rules call, in the traceback above, is here: https://github.com/projectcalico/networking-calico/blob/master/networking_calico/plugins/ml2/drivers/calico/policy.py#L98

Tags:

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2020-06-17:

After looking into code it seems for me that it is race in calling "_ensure_default_security_group" method from "get_security_group_rules" and also from "_ensure_default_security_group_handler" which is subscribed e.g. to port's BEFORE_UPDATE even.

This is probably caused by https://github.com/openstack/neutron/commit/4739a4febb8c77e5a0ca48bc77ae4dccd9f8a81e which added calling this method from get_security_group_rules().

I will try to propose fix for that this week.

tags:	added: db
Changed in neutron:
status:	New → Confirmed
importance:	Undecided → Medium
assignee:	nobody → Slawek Kaplonski (slaweq)

Revision history for this message

Nell Jerram (neil-jerram) wrote on 2020-06-17:

Thanks Slawek for your analysis. If you get time to work on a fix, I would very much appreciate that.

I'm still curious, however, if there is something different or unusual about the calling pattern from my driver code. If my code's usage pattern is mainline, I would expect many more people to have hit and reported this problem. On the other code, if my code's usage pattern is unusual, perhaps I could also change it to be more aligned with the mainline.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-18: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/736842

Changed in neutron:
status:	Confirmed → In Progress

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2020-06-18:

I think that Your code isn't unusual. It seems that _ensure_default_security_group method wasn't really atomic. And it was working for You before as this method wasn't called in get_security_group_rules() method.
But in commit [1] there was added call to _ensure_default_security_group method during get_security_group_rules(). It was done to avoid the case when user calls list security group rules for newly created project and there was no those default rules listed there unless first list security groups request was done.

[1] https://github.com/openstack/neutron/commit/4739a4febb8c77e5a0ca48bc77ae4dccd9f8a81e

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-24: Fix merged to neutron (master)

Reviewed: https://review.opendev.org/736842
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7019c5cf50bc4ad97b302220664a26efaf81a7fd
Submitter: Zuul
Branch: master

commit 7019c5cf50bc4ad97b302220664a26efaf81a7fd
Author: Slawek Kaplonski <email address hidden>
Date: Thu Jun 18 23:18:07 2020 +0200

Make _ensure_default_security_group method atomic

Method _ensure_default_security_group wasn't atomic as it first tries to get
default SG and if that not exists in DB, it tries to create it.

    It may happend, like e.g. in Calico plugin that between
    get_default_sg_id method and create_security_group method, this default
    SG will be created by other neutron worker. And in such case there will
    be Duplicate entry exception raised.

So this patch is adding handling of such exception.

Change-Id: I515c310f221e7d9ae3be59a26260538d1bc591c2
Closes-Bug: #1883730

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-24: Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/737699

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-24: Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/737700

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-24: Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/737702

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-24: Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/737703

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-24: Fix proposed to neutron (stable/queens)

#10

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/737704

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-26: Fix merged to neutron (stable/ussuri)

#11

Reviewed: https://review.opendev.org/737699
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bdbeb673572a7979772cd423dd7af73d3898648f
Submitter: Zuul
Branch: stable/ussuri

commit bdbeb673572a7979772cd423dd7af73d3898648f
Author: Slawek Kaplonski <email address hidden>
Date: Thu Jun 18 23:18:07 2020 +0200

Make _ensure_default_security_group method atomic

Method _ensure_default_security_group wasn't atomic as it first tries to get
default SG and if that not exists in DB, it tries to create it.

So this patch is adding handling of such exception.

    Change-Id: I515c310f221e7d9ae3be59a26260538d1bc591c2
    Closes-Bug: #1883730
    (cherry picked from commit 7019c5cf50bc4ad97b302220664a26efaf81a7fd)

tags:	added: in-stable-ussuri
tags:	added: in-stable-train

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-26: Fix merged to neutron (stable/train)

#12

Reviewed: https://review.opendev.org/737700
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=eed9bcbd7be62aa94ff1c3a3543835295be06f04
Submitter: Zuul
Branch: stable/train

commit eed9bcbd7be62aa94ff1c3a3543835295be06f04
Author: Slawek Kaplonski <email address hidden>
Date: Thu Jun 18 23:18:07 2020 +0200

Make _ensure_default_security_group method atomic

Method _ensure_default_security_group wasn't atomic as it first tries to get
default SG and if that not exists in DB, it tries to create it.

So this patch is adding handling of such exception.

    Change-Id: I515c310f221e7d9ae3be59a26260538d1bc591c2
    Closes-Bug: #1883730
    (cherry picked from commit 7019c5cf50bc4ad97b302220664a26efaf81a7fd)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-07-17: Fix merged to neutron (stable/stein)

#13

Reviewed: https://review.opendev.org/737702
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=13f8296d5c1bf0e2bc90ee689a4825ec9acf8881
Submitter: Zuul
Branch: stable/stein

commit 13f8296d5c1bf0e2bc90ee689a4825ec9acf8881
Author: Slawek Kaplonski <email address hidden>
Date: Thu Jun 18 23:18:07 2020 +0200

Make _ensure_default_security_group method atomic

Method _ensure_default_security_group wasn't atomic as it first tries to get
default SG and if that not exists in DB, it tries to create it.

So this patch is adding handling of such exception.

Conflicts:
neutron/db/securitygroups_db.py

    Change-Id: I515c310f221e7d9ae3be59a26260538d1bc591c2
    Closes-Bug: #1883730
    (cherry picked from commit 7019c5cf50bc4ad97b302220664a26efaf81a7fd)

tags:	added: in-stable-stein
tags:	added: in-stable-queens

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-07-17: Fix merged to neutron (stable/queens)

#14

Reviewed: https://review.opendev.org/737704
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0cc3032edc7d1cd533789453d19b2a1b2d193d38
Submitter: Zuul
Branch: stable/queens

commit 0cc3032edc7d1cd533789453d19b2a1b2d193d38
Author: Slawek Kaplonski <email address hidden>
Date: Thu Jun 18 23:18:07 2020 +0200

Make _ensure_default_security_group method atomic

Method _ensure_default_security_group wasn't atomic as it first tries to get
default SG and if that not exists in DB, it tries to create it.

So this patch is adding handling of such exception.

Conflicts:
neutron/db/securitygroups_db.py

    Change-Id: I515c310f221e7d9ae3be59a26260538d1bc591c2
    Closes-Bug: #1883730
    (cherry picked from commit 7019c5cf50bc4ad97b302220664a26efaf81a7fd)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-07-18: Fix merged to neutron (stable/rocky)

#15

Reviewed: https://review.opendev.org/737703
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3216c1efe904ce8be944bc7b45e6f0b6fd501913
Submitter: Zuul
Branch: stable/rocky

commit 3216c1efe904ce8be944bc7b45e6f0b6fd501913
Author: Slawek Kaplonski <email address hidden>
Date: Thu Jun 18 23:18:07 2020 +0200

Make _ensure_default_security_group method atomic

Method _ensure_default_security_group wasn't atomic as it first tries to get
default SG and if that not exists in DB, it tries to create it.

So this patch is adding handling of such exception.

Conflicts:
neutron/db/securitygroups_db.py

    Change-Id: I515c310f221e7d9ae3be59a26260538d1bc591c2
    Closes-Bug: #1883730
    (cherry picked from commit 7019c5cf50bc4ad97b302220664a26efaf81a7fd)

tags:

added: in-stable-rocky

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-11-18: Fix included in openstack/neutron queens-eol

#16

This issue was fixed in the openstack/neutron queens-eol release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-11-18: Fix included in openstack/neutron rocky-eol

#17

This issue was fixed in the openstack/neutron rocky-eol release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.