DBDuplicateEntry when creating secgroup

[Sergey Skripnick]

Error when attempted to create secutity groups in more then one thread.

full log: http://logs.openstack.org/33/144633/45/check/gate-rally-dsvm-neutron-rally/0ed9492/logs/

ERROR neutron.api.v2.resource [req-f211be5c-0c2e-42d7-86e4-773a299f3e03 ctx_rally_08bdcaef529f4ebc8b24263106e2e2ff_user_1 ctx_rally_1e92a94e-1a15-47f3-84ab-e656556d2567_tenant_1] create failed
Traceback (most recent call last):
  File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource
    result = method(request=request, **args)
  File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 452, in create
    obj = obj_creator(request.context, **kwargs)
  File "/opt/stack/new/neutron/neutron/db/securitygroups_db.py", line 127, in create_security_group
    self._ensure_default_security_group(context, tenant_id)
  File "/opt/stack/new/neutron/neutron/db/securitygroups_db.py", line 536, in _ensure_default_security_group
    default_sg=True)
  File "/opt/stack/new/neutron/neutron/db/securitygroups_db.py", line 156, in create_security_group
    context.session.add(egress_rule)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 470, in __exit__
    self.rollback()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 467, in __exit__
    self.commit()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 377, in commit
    self._prepare_impl()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 357, in _prepare_impl
    self.session.flush()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1919, in flush
    self._flush(objects)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2037, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2001, in _flush
    flush_context.execute()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
    rec.execute(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 526, in execute
    uow
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 65, in save_obj
    mapper, table, insert)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 570, in _emit_insert_statements
    execute(statement, multiparams)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception
    util.raise_from_cause(newraise, exc_info)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 436, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
DBDuplicateEntry: (IntegrityError) (1062, "Duplicate entry '08bdcaef529f4ebc8b24263106e2e2ff' for key 'PRIMARY'") 'INSERT INTO default_security_group (tenant_id, security_group_id) VALUES (%s, %s)' ('08bdcaef529f4ebc8b24263106e2e2ff', '23327aad-6c5b-40c0-8069-2401e09893d1')

[Ann Taraday (akamyshnikova)]

This error points out that more than one default security group is trying to be created in one tenant. Having more than one default security group in one tenant is not allowed, for this goal was created default_security_group table. Please, give some more information how do you get this error.

[Ann Taraday (akamyshnikova)]

My previous comment could be understood incorrectly. What I meant is that if you have such error the problem could be in test itself, when it is running in more than one thread, try to create more than one default security group in one tenant, that is not allowed. There is https://bugs.launchpad.net/tempest/+bug/1416294 bug for check-grenade-dsvm-neutron.

[Ann Taraday (akamyshnikova)]

I was asked by Boris Pavlovic to fix this issue quickly.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/155789
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ee2d52b0eb9a75df36c656462918c6f08955ad51
Submitter: Jenkins
Branch: master

commit ee2d52b0eb9a75df36c656462918c6f08955ad51
Author: Ann Kamyshnikova <email address hidden>
Date: Fri Feb 13 18:41:21 2015 +0300

    Skip DBDuplicateEntry exception in security group creation

    An DBDuplicateEntry expection appeared if it is attempted to
    create secutity groups in more then one thread. In this situation
    at one moment it is trying to be created 2 default security groups
    which is not allowed. In this case exception should skipped with
    a message about failed attempt.

    Closes-bug: #1419723

    Change-Id: Id6fe997ad3b72f875160e1e31a3dc17c3cc53b75

[Eugene Nikanorov (enikanorov)]

The issue was not fixed completely

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/164463

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/164463
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8dee8df12b402943b3f7f2e34d3ccc2c00068619
Submitter: Jenkins
Branch: master

commit 8dee8df12b402943b3f7f2e34d3ccc2c00068619
Author: Eugene Nikanorov <email address hidden>
Date: Sat Mar 14 23:35:08 2015 +0300

    Handle DBDuplicateError exception properly when creating default sg

    Previously, an exception was not caught in one of invocations
    (create_network) of _ensure_default_security_group.
    Move exception handling inside that method so it never fails
    with such exception.

    Change-Id: Ibb0597d4db187c856f9ac1d9700701e0165c3c73
    Closes-Bug: #1419723

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/165290

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/164428
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6c61cf7af71a3661a14900b7baa2e7985ba6673f
Submitter: Jenkins
Branch: master

commit 6c61cf7af71a3661a14900b7baa2e7985ba6673f
Author: Boris Pavlovic <email address hidden>
Date: Sat Mar 14 16:15:42 2015 +0300

    Run more Rally benchmark on every patch

    * Since 24 Nov 2014 we added a lot of Neutron benchmarks

    Running more Neutron related benchmarks in Neutron gate allows
    to avoid performance regressions and races.

    * Neutron benchmarks are described here:
    https://github.com/stackforge/rally/blob/master/rally/benchmark/scenarios/neutron/network.py
    It's quite simple code be free to take a look.

    * All changes in concurrency and times are related to optimization
    of duration/usefulness

    * To get description of benchmarks use:

       rally info find NeutronNetworks.create_and_update_networks

    New benchmarks:

    - NeutronNetworks.create_and_update_networks

    - NeutronNetworks.create_and_delete_networks

    - NeutronNetworks.create_and_update_subnets

    - NeutronNetworks.create_and_delete_subnets

    - NeutronNetworks.create_and_update_routers

    - NeutronNetworks.create_and_delete_routers

    - NeutronNetworks.create_and_list_routers

    - NeutronNetworks.create_and_update_ports

    - NeutronNetworks.create_and_delete_ports

    - NeutronNetworks.create_and_list_ports

    - Quotas.neutron_update

    related bug: #bug 1419723
    Change-Id: Ie3c84e057fc96c0f35ad77b7297c564442ebcf10

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/165290
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5dccff1cb3367f88b7a7851988b19caad313b036
Submitter: Jenkins
Branch: master

commit 5dccff1cb3367f88b7a7851988b19caad313b036
Author: YAMAMOTO Takashi <email address hidden>
Date: Wed Mar 18 11:07:09 2015 +0900

    Fix DBDuplicateError handling in _ensure_default_security_group

    The coding in change-id Ibb0597d4db187c856f9ac1d9700701e0165c3c73
    catches and ignores DBDuplicateError in a nested transaction.
    It would cause another exception, InvalidRequestError, on the
    next operation. ("This Session's transaction has been rolled back")
    This commit fixes it.

    Also, tweak a test case to expose the error.

    Closes-Bug: #1433418
    Related-Bug: #1419723
    Change-Id: Ie4de271c0512fb2ecc6ed6842ad20386e3785a9c

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: neutron-pecan
Review: https://review.openstack.org/185072